Image encoding device and method, and imaging device

Abstract

To generate an encoding stream of a key frame within a prescribed time from the reception of an insertion request of the key frame and have the processing time met.SOLUTION: An image encoding device for prediction-encoding a moving image comprises: in-screen encoding means which performs in-screen encoding to an input frame image; inter-screen encoding means which performs inter-screen encoding; holding means which holds a copy picture; and control means which performs control of suspending the encoding and re-encoding a P frame image as a key frame by the in-screen encoding means when the inter-screen encoding means encodes as a P picture the P frame image to be encoded as the P picture in a case of receiving a key frame insertion request, and replacing one B frame image to be encoded as a B picture within a predetermined time from the reception of the key frame insertion request with the copy picture held by holding means without performing the encoding.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image encoding device and method, and an imaging device, and more particularly to an encoding technique for video streaming transmission. [Background technology]

[0002] In recent years, online video distribution services and real-time video streaming services have become popular. When distributing video, international standard coding standards such as H.264 and HEVC (High Efficiency Video Coding) are often used to compress moving images to reduce data volume.

[0003] In these coding methods, in order to improve coding efficiency, coding is performed using both intra-predicted pictures and inter-predicted pictures. An intra-predicted picture is a picture that can be decoded with only one picture. An inter-predicted picture is a picture that improves coding efficiency by referencing other pictures and using inter-picture differences. In particular, an intra-predicted picture is generally called a key frame to distinguish it from other frames, and in H.264 and HEVC, it is an IDR picture (see Non-Patent Document 1).

[0004] One of the benefits of key frames in video is that they can serve as the starting point for random access playback. That is, playback is possible from a key frame even in the middle of an encoded stream. For example, when a client makes a new playback request during video streaming and tries to play back from the middle of the encoded stream, playback will begin from the position of the key frame.

[0005] In order to respond to a client's playback request as quickly as possible without compromising real-time performance, it is important for the server to perform encoding using key frames as quickly as possible in response to the playback request, as described in Patent Document 1.

[0006] For example, in network video distribution, it is required that IDR pictures be encoded within approximately 100 msec of a new client's playback request. This means that if the encoding performance is 30 frames / sec (encoding 30 pictures per second), it takes 33 msec per picture, so encoding must be performed as IDR pictures within three frames.

[0007] In H.264, the IBBP structure is used as a standard GOP (Group Of Picture) structure. Fig. 12 shows the reference relationship of the IBBP structure. A P picture 204 is encoded with reference to an I picture 201. B pictures 202 and 203 are encoded with reference to the I picture 201 as forward prediction and with reference to the P picture 204 as backward prediction. In this way, B pictures are encoded using backward reference, i.e., they refer to pictures that come later in input order, so the input order and encoding order are reversed for encoding.

[0008] 13 shows the input order and encoding order of frame images in the IBBP structure. Frame image 301 is encoded as an I-picture. Frame images 302 and 303 are encoded as B-pictures. Frame image 304 is encoded as a P-picture. Frame images 302 and 303, which are encoded as B-pictures, refer to P-picture 314, which comes later in input order, so they are rearranged and encoded after P-picture 314. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] H.265 / HEVC Textbook, Supervised by Sakae Okubo, co-edited by Teruhiko Suzuki, Masayuki Takamura, and Ken Nakajo, published October 21, 2013 [Patent documents]

[0010] [Patent Document 1] JP 2014-143496 A Summary of the Invention [Problem to be solved by the invention]

[0011] When encoding using the IBBP structure in H.264, the standard reference relationship is as shown in Figure 12, and encoding is performed in the order shown in Figure 13. Furthermore, to support network video distribution, as mentioned above, there is a condition that encoding with an IDR picture must be completed within 100 msec of receiving an IDR picture insertion request, that is, within three pictures at 30 frames / sec.

[0012] For example, suppose a request to insert an IDR picture is made while encoding frame image 304, which should have been encoded as a P picture. In this case, to satisfy the condition of inserting an IDR picture within three frames after the IDR picture insertion request, it is necessary to change either frame image 304, frame image 302, or frame image 303 to an IDR picture and then encode them. However, if either frame image 302 or 303 is changed to an IDR picture, P picture 314, which has already been encoded, will be encoded by referencing I picture 311 across the IDR pictures of frame images 302 and 303, which does not comply with the standard.

[0013] Therefore, in this case, the frame image 304 must be changed to an IDR picture, but because the frame image 304 is in the middle of being coded, it must be discarded and re-coded as an IDR picture, which creates the problem of not being able to meet the processing time.

[0014] The present invention has been made in consideration of the above problems, and has as its object to generate an encoded stream of key frames within a predetermined time after a request for inserting a key frame is received, and to meet the processing time. [Means for solving the problem]

[0015] In order to achieve the above object, an image coding apparatus of the present invention for predictively coding a moving image comprises: an intra-frame coding means for performing intra-frame coding on input frame images; an inter-frame coding means for performing inter-frame coding on input frame images; Refer to the key frame image and instruct all macroblocks to skip B pictures. The image coding system includes a holding means for holding a copy picture, and a control means for, when the inter-picture coding means is coding a P frame image to be coded as a P picture as a P picture upon receiving a key frame insertion request, suspending the coding and re-coding the P frame image as a key frame by the intra-picture coding means, and for controlling to replace one of the B frame images to be coded as a B picture with the copy picture held in the holding means without coding it within a predetermined time after receiving the key frame insertion request. [Effects of the Invention]

[0016] According to the present invention, it is possible to generate an encoded stream of key frames within a predetermined time after a request for inserting a key frame is received, and to meet the processing time. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a diagram showing an overview of a network camera system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram showing the configuration of an encoding device according to an embodiment. [Figure 3] 3A to 3C are diagrams for explaining an encoding method for each frame image in the first embodiment. [Figure 4] FIG. 10 is a diagram showing an IDR picture insertion request and a problem in the first embodiment. [Figure 5] FIG. 10 is a diagram showing the encoding order when an IDR picture insertion request is received during encoding of a P picture in the first embodiment. [Figure 6] FIG. 2 is a diagram showing a stream generation method according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing the encoding order when an IDR picture insertion request is received during encoding of a B picture in the first embodiment. [Figure 8] 10 is a flowchart showing processing when an IDR picture insertion request is received in the first embodiment. [Figure 9] FIG. 10 is a diagram showing a GOP structure in a network video distribution mode in the second embodiment. [Figure 10] FIG. 11 is a diagram showing the encoding order when an IDR picture insertion request is received during the encoding of a P picture in the second embodiment. [Figure 11] FIG. 11 is a diagram showing the encoding order when an IDR picture insertion request is received during the encoding of a B picture in the second embodiment. [Figure 12] FIG. 1 is a diagram showing the reference relationship of pictures in a conventional IBBP structure. [Figure 13] FIG. 1 is a diagram showing the input order and encoding order of frame images in a conventional IBBP structure. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0019] First Embodiment 1 shows the overall configuration of a network system according to this embodiment. A network camera 11 distributes a video stream to multiple clients via a network 12. Clients 13 and 14 receive the video stream and play back the video.

[0020] In this system, a new client may be added during video distribution. When a client 15 is added during video distribution, information that a new client has been added is notified to the network camera 11 via the network 12. The network camera 11 performs encoding processing in response to the information that a new client has been added. Details of the encoding processing performed here will be described later.

[0021] 2 is a block diagram illustrating the configuration of an image encoding device installed in the network camera 11 according to this embodiment. Here, an image encoding device that performs predictive encoding in the H.264 encoding format is shown as an example.

[0022] The encoding unit 100 performs encoding processing on a picture-by-picture basis. The control unit 150 controls the encoding unit 100 on a picture-by-picture basis, controlling picture types, reference relationships, and the like. When a new client is added, the control unit 150 is notified by the network communication unit 118 that a client has been added. When a new client is added, the control unit 150 controls the encoding unit 100 in a manner different from normal. Details of the operation when a notification of the addition of a new client is received will be explained later.

[0023] First, we will explain how one frame image is encoded. Light from a subject enters the imaging unit 102 through a lens 101 and is converted into an image signal. The imaging unit 102 converts the image signal into digital pixel data and sends it to the development processing unit 103. The development processing unit 103 performs image processing such as debayering, blemish correction, noise removal, and color conversion to YCbCr format. After image processing, the image data is in a format suitable for compression encoding, and is input to the encoding frame buffer 104 of the encoding unit 100 as the encoding target image that will now be encoded.

[0024] The intra prediction unit 105 obtains a frame image to be coded (hereinafter referred to as a "target image to be coded") from the coding frame buffer 104 and performs intra prediction (intra-picture coding). H.264 intra prediction includes DC prediction, planar prediction, and nine types of directional prediction, including horizontal, vertical, and diagonal prediction. Among these prediction modes, a prediction mode with the highest coding efficiency is determined using a predetermined cost formula. The intra prediction unit 105 calculates the pixel difference between the target image to be coded and a predicted image obtained by intra prediction in the determined prediction mode, and outputs the difference image and the predicted image to the prediction selection unit 107. Since a locally decoded image must be used as the predicted image used to generate the difference image, a locally decoded image already generated is obtained from the reference frame buffer 117 and used to generate the predicted image.

[0025] The inter prediction unit 106 first obtains the image to be coded stored in the coding frame buffer 104. It also obtains a reference image from the reference frame buffer 117, performs block matching between the image to be coded and the reference image, and detects a motion vector. The inter prediction unit 106 then finds the pixel difference between the image to be coded and a predicted image at the position of the detected motion vector, and outputs the difference image and the predicted image to the prediction selection unit 107 (inter coding).

[0026] The prediction selection unit 107 compares the results of intra prediction and inter prediction, selects which prediction to use, and outputs the selected differential image to the orthogonal transformation unit 108. The selected predicted image is also output to the motion compensation unit 115 for creating a locally decoded image. However, since I pictures must be coded using intra prediction, the prediction selection unit 107 always selects intra prediction. For P pictures and B pictures, it is possible to select from the results of both intra prediction and inter prediction.

[0027] The orthogonal transformation unit 108 performs a discrete cosine transform on the difference image sent thereto to generate transformation coefficients, which are output to the quantization unit 109 . The quantization unit 109 quantizes the transform coefficients sent from the orthogonal transform unit 108 in accordance with a quantization step size output by a quantization control unit 110. The quantized transform coefficients are output to a variable-length coding unit 111 to create a coded stream and to an inverse quantization unit 113 to create a locally decoded image.

[0028] The variable-length coding unit 111 performs zigzag scanning, alternate scanning, etc. on the quantized transform coefficients to perform variable-length coding. To this, variable-length coded coding method information such as motion vectors, quantization step sizes, and macroblock division information is added to generate a coded stream. The generated coded stream is sent to a network communication unit 118 for video distribution. The network communication unit 118 packetizes the coded stream for network communication and transmits it to the network 12. The coded stream is also recorded on a recording medium 112 for recording. When generating the coded stream, the amount of code generated for each macroblock is calculated and output to the quantization control unit 110.

[0029] The quantization control unit 110 uses the generated code amount sent from the variable length coding unit 111 to determine a quantization step size so as to achieve a target code amount, and outputs the determined quantization step size to the quantization unit 109 .

[0030] Meanwhile, the inverse quantization unit 113 performs inverse quantization on the quantized transform coefficients sent from the quantization unit 109 to generate transform coefficients for local decoding. These transform coefficients are output to the inverse orthogonal transform unit 114. The inverse orthogonal transform unit 114 performs inverse discrete cosine transform on the sent transform coefficients to generate a difference image. The generated difference image is output to the motion compensation unit 115. The motion compensation unit 115 generates image data for local decoding by adding the predicted image sent from the prediction selection unit 107 and the difference image sent from the inverse orthogonal transform unit 114. The generated image data is output to the deblocking filter unit 116.

[0031] The deblocking filter unit 116 applies a deblocking filter to the image data sent thereto, and the image after the deblocking filter is stored in the reference frame buffer 117 as a locally decoded image. Through these operations, a coded stream and a locally decoded image are created.

[0032] Next, normal picture-by-picture encoding processing will be explained using Fig. 3. Here, it is assumed that encoding is performed at 30 frames / sec, and one frame image is input and encoded every 33 msec.

[0033] Frame images 401, 402, 403, 404, 405, and 406 are input in sequence every 33 msec. Frame image 403 is input at time T1, and is coded as a P picture at time T2, when the next frame image 404 is input. Hereinafter, a frame image to be coded as a P picture will be referred to as a "P frame image."

[0034] At the next time T3, frame image 401, which was input at a timing earlier than P frame image 403, is coded as a B picture. Hereinafter, a frame image to be coded as a B picture will be referred to as a "B frame image." At the next time T4, B frame image 402 is coded as a B picture in the same manner.

[0035] At the next time, T5, the P frame image 406 is coded as a P picture that references the P picture 413. At the next time, T6, the B frame image 404 is coded as a B picture that references the P pictures 413 and 416 that are obtained by coding the P frame images 403 and 406 that were input before and after the B frame image 404. At the next timing, T7, the B frame image 405 is similarly coded as a B picture that references the P pictures 413 and 416. In this way, the order of frame images is changed before encoding so that it is possible to refer to images that have been input later and encoded as P pictures.

[0036] Next, we will explain the operation when a new client is added to the network system. When a new client is added, it is necessary to insert a key frame as early as possible so that the new client can immediately play the video stream being delivered. A key frame here refers to a picture that can be played back, i.e., decoded, from that frame, and in the H.264 standard, this corresponds to an IDR picture.

[0037] When a new client is added, the network communication unit 118 receives information that the client has been added via the network 12. The network communication unit 118 notifies the control unit 150 that a new client has been added. Upon receiving the notification that a new client has been added, the control unit 150 performs encoding control so that a key frame, i.e., an IDR picture, can be inserted as quickly as possible.

[0038] The earlier the timing of inserting an IDR picture, the better, but in normal network video distribution, the IDR picture must be inserted within 100 msec (a predetermined time) after receiving a notification that a new client has been added. In other words, the encoder must encode one of the frame images as an IDR picture within 100 msec of receiving an IDR picture insertion request (key frame insertion request) due to the addition of a new client. In the case of encoding at 30 frames per second, this means that the frame image must be encoded as an IDR picture within three pictures after receiving an IDR picture insertion request.

[0039] Typically, at 30 frames per second, an IDR picture is coded approximately once every 15 pictures. Therefore, when a request to insert an IDR picture is made, a frame image that would normally be coded as a P or B picture must be coded as an IDR picture. When a frame image that would normally be coded as a P or B picture is coded as an IDR picture, restrictions arise due to the reference relationship. This point will be explained with reference to Figure 4.

[0040] Consider the case where a request to insert an IDR picture is received while the P frame image 406 is being coded as a P picture at time T5. In this case, to insert the IDR picture within 100 msec, either the B frame images 404, 405, or the P frame image 406 must be coded as an IDR picture.

[0041] First, consider the case where the B frame image 404 or the B frame image 405 is coded as an IDR picture. The P frame image 406, which is coded as a P picture before the B frame image 404, uses the P picture 413 as a reference image. If the B frame image 404 or the B frame image 405 is coded as an IDR picture, the P picture 416 will reference the P picture 413 across the IDR picture, which violates the standard.

[0042] Therefore, if an IDR picture insertion request occurs at this timing, the P frame image 406 must be coded as an IDR picture rather than a P picture. However, if the P frame image 406 is to be coded again as an IDR picture, the coding process as a P picture up to that point must be interrupted and coding must be restarted from the beginning of the P frame image 406. This creates a problem in that the processing time will not be within the 33 msec limit.

[0043] A coding method to avoid this problem will be explained below. The original coding order and the coding order after an IDR picture insertion request are shown in Figure 5.

[0044] If an IDR picture insertion request is received at time T5 while the P frame image 406 is being coded as a P picture, the results of this coding process are discarded. The P frame image 406 is then coded as a new IDR picture. Depending on the timing of the IDR picture insertion request, the re-coding will be completed by time T6. At this point, there is only time left to code one frame image at time T7, compared to the coding process for the two B frame images 404 and 405.

[0045] Next, at time T7, the B frame image 404 is coded as a B picture. At this time, because the P frame image 406 has been coded as the IDR picture 426, the B frame image 404 is coded with reference to only the IDR picture 426. This is because a frame image coded after the IDR picture cannot reference a picture coded before the IDR picture.

[0046] Although there is no time to perform the encoding process for the next B frame image 405, an encoded stream for one picture is required. Therefore, instead of encoding the B frame image 405, an encoded stream for one picture called a copy picture is inserted. A copy picture is an encoded instruction to perform skip processing on all macroblocks during decoding. In this case, the IDR picture 426 is referenced during decoding, and all macroblocks undergo B picture skip processing. Therefore, an image exactly identical to the IDR picture 426 is generated.

[0047] The coded stream of this copy picture can be created in advance because the processing content is fixed and it does not depend on a reference picture, etc. The coded stream of the copy picture is created before the start of coding processing for the entire sequence, and instead of coding the corresponding frame image (in this case, the B frame image 405), the coded stream of the copy picture is inserted after the coded stream of the B picture 424.

[0048] This makes it possible to insert one coded stream of the portion corresponding to the B frame image 405 with almost no processing time. At time T7, two coded streams can be created: one for the B frame image 404 and one for a copy picture in place of the B frame image 405, making it possible to make up the processing time.

[0049] Next, with reference to FIG. 6, a stream generation when an IDR picture insertion request is received during the coding of a P picture will be described. First, before the start of the encoding operation, an encoded stream of the copy picture is generated and stored in the memory 712. This may be generated by the CPU 711, or may be generated by causing the encoding device to perform such an operation.

[0050] When the encoding operation starts, the encoding unit 100 generates an encoded stream for each frame image and sends the generated encoded stream to the network communication unit 118. The network communication unit 118 sends the encoded stream to the network 12. At the same time, the encoded stream is also sent to the recording medium 112, and is written sequentially.

[0051] As explained in Fig. 5, if a request to insert an IDR picture is received while the P frame image 406 is being coded, the result of the coding process is discarded and the P frame image 406 is coded again as an IDR picture. The coded stream 701 of the generated IDR picture 426 is sent to the network communication unit 118. At the same time, it is also sent to the recording medium 112 and written there. Next, the B frame image 404 is coded as a B picture, and a coded stream 702 is generated and sent to the network communication unit 118. At the same time, it is also sent to the recording medium 112 and written therein.

[0052] No encoding operation is performed on the next B frame image 405, and the copy picture in the memory 712 is sent to the network communication unit 118 as an encoded stream 703. At the same time, it is also sent to the recording medium 112 (recording medium) and written therein. Since no encoding process is performed and the stream in the memory 712 is simply sent to the network communication unit 118 and recording medium 112, this can be achieved with almost no processing time.

[0053] Next, the coding order when an IDR picture insertion request is made while a B frame image is being coded as a B picture will be described with reference to Fig. 7. Note that the following description will only focus on the differences from the case where an IDR picture insertion request is made while a P frame image is being coded as a P picture.

[0054] If a request to insert an IDR picture is received while the B frame image 901 is being coded as a B picture at time T3, as mentioned above, due to the constraints of the reference relationship, only frame images that were originally intended to be coded as P pictures can be changed to IDR pictures.

[0055] In this case, the P frame image input immediately after the IDR picture insertion request is received is P frame image 906, and this image is coded as an IDR picture. The B frame image 901 continues to be coded as a B picture as originally planned. The next B frame image 902 is also coded as a B picture as originally planned.

[0056] The P frame image 906 is encoded as an IDR picture at time T5, which occurs within 100 msec of the IDR picture insertion request. Thus, if an IDR picture insertion request occurs while a B frame image is being encoded as a B picture, a P frame image will always be present within three pictures. Therefore, if that P frame image is encoded as an IDR picture, the IDR picture can be encoded within 100 msec.

[0057] For example, if an IDR picture insertion request is received while B frame image 902, rather than B frame image 901, is being coded as a B picture (time T4), the P frame image 906 is similarly coded as an IDR picture. This makes it possible to insert an IDR picture within 100 msec after the IDR picture insertion request is received.

[0058] Note that because the P frame image 906 is coded as an IDR picture, the next two B frame images 904 and 905 are coded by referencing only the IDR picture 926. This is because images that precede the IDR picture in input order but are coded after the IDR picture can only refer to the IDR picture.

[0059] In this way, if an IDR picture insertion request is received while a B frame image is being coded as a B picture, the processing time can be met without interrupting or discarding the coding process. Therefore, unlike when an IDR picture insertion request is received while a P frame image is being coded as a P picture, no copy picture is used.

[0060] Next, the flow of the encoding process in the first embodiment will be described with reference to the flowchart in FIG. {Dabraku=OFF} First, before encoding the entire sequence, a copy picture is created (S101). Then, encoding begins (S102). Encoding is performed for each frame image, and it is determined whether an IDR picture insertion request has been received (S103). If an IDR picture insertion request has not been received (NO in S103), encoding is performed as usual (S104).

[0061] If an IDR picture insertion request is received (YES in S103), it is determined whether a P picture is currently being coded (S105). If a P picture is currently being coded (YES in S105), the process proceeds to S106, and if a P picture is not currently being coded (NO in S105), the process proceeds to S110.

[0062] In S106, the encoding process of the P picture is interrupted, and the encoded stream that was being generated is discarded.Then, the P frame image whose encoding process was interrupted is re-encoded as an IDR picture (S107). The next B frame image is coded as a B picture (S108). At this time, coding is performed by referring only to the IDR picture generated in S107 as a reference image. The next B frame image is not coded, and a copy picture is inserted instead (S109). In this way, coding is performed when a request to insert an IDR picture is received during coding of a P picture.

[0063] If the IDR picture insertion request does not come during the encoding of a P picture (NO in S105), the IDR picture insertion request comes during the encoding of a B picture, so the B picture is encoded as usual until the next P frame image comes (S110).

[0064] Then, the P frame image is coded as an IDR picture (S111). The two subsequent B frame images, i.e., the B frame images that are input earlier in the order of frames but coded later in the order of frames than the IDR picture coded in S111, are coded by referencing only the IDR picture coded in S111 (S112). In this way, coding is performed when an IDR picture insertion request is made while coding the B frame images as B pictures.

[0065] Then, it is determined whether the encoding operation has been completed up to the final frame image (S113), and if not, the process returns to S103 and repeats the above operation, and if completed, the process ends.

[0066] As described above, if a request to insert an IDR picture is received while a P frame image is being coded as a P picture, the P picture is discarded and re-coded as an IDR picture, and one B picture is replaced with a copy picture. If a request to insert an IDR picture is received while a B frame image is being coded as a B picture, the image is coded as a B picture as usual until the next P frame image arrives, and then the P frame image is coded as an IDR picture.Then, the next two B frame images are coded as B pictures by referencing only the IDR picture.

[0067] In the above example, the case where encoding is performed at 30 frames / sec has been described, but similar control can be performed even in the case of other frame rates.

[0068] As described above, according to the first embodiment, it is possible to realize encoding of an IDR picture within a specified time (for example, 100 msec) after receiving a request to insert an IDR picture, and also to meet the processing time.

[0069] <Second embodiment> Next, a second embodiment of the present invention will be described. The second embodiment differs from the first embodiment in that it accommodates an IDR picture insertion request by changing the GOP structure. Note that the following description will only cover the differences from the first embodiment.

[0070] The configuration of the encoding device according to this embodiment is basically the same as that of the first embodiment shown in Fig. 2. However, in addition to the IDR picture insertion request, an operation mode flag indicating whether the mode is a normal moving image recording mode or a network moving image distribution mode is input to the control unit 150. Furthermore, in this embodiment, the IDR picture insertion request is received only in the network moving image distribution mode.

[0071] In network video distribution mode, a different GOP structure from that used during normal video recording is used to accommodate playback requests due to the addition of new clients, i.e., requests for immediate insertion of IDR pictures.

[0072] The control unit 150 determines whether the mode is normal video recording mode or network video distribution mode from the operation mode flag input to the control unit 150, and uses the corresponding GOP structure. In normal video recording mode, operations compatible with network video distribution are not performed, and encoding is performed using the normal IBBP structure described in Figures 12 and 13. In the normal IBBP structure, B pictures are generally encoded using both forward prediction and backward prediction. In this normal video recording mode, no IDR picture insertion request is made.

[0073] The GOP structure in network video distribution mode will be explained using Figure 9. Here, encoding is performed at 30 frames / sec, with one frame image input every 33 msec and encoded. Compared to the GOP structure used in normal video recording, the GOP structure in network video distribution mode has different reference relationships and encoding order.

[0074] Frame images 1101, 1102, 1103, and 1104 are input in sequence every 33 msec.

[0075] First, the reference relationships will be explained. As with normal P frame images, they are coded as P pictures by referencing the preceding P picture. For example, the P frame image 1104 is coded by referencing the P picture 1111.

[0076] For B pictures, only the P picture or I picture of the frame picture that is closer to the front or back in input order of the frame pictures is referenced. For example, for the B frame picture 1102, when the front P frame picture 1101 is compared with the back P frame picture 1104, the front P frame picture 1101 is closer in input order. Therefore, encoding is performed by referencing only the P picture 1111 of the front P frame picture 1101. Furthermore, for the B frame picture 1103, when the front P frame picture 1101 is compared with the back P frame picture 1104, the back P frame picture 1104 is closer in input order. Therefore, encoding is performed by referencing only the P picture 1114 of the back P frame picture 1104.

[0077] Next, the coding order will be described. A P frame image 1101 is input at time T1 and is coded as a P picture at the following time T2. In this way, P pictures are coded at the timing immediately following their input. Similarly, a P frame image 1104 is input at time T4 and is coded as a P picture at the following time T5.

[0078] With regard to B pictures, when only forward referencing is performed, encoding is performed two pictures after the B frame image is input. When only backward referencing is performed, encoding is performed three pictures after the B frame image is input. For example, a B frame image 1102 input at time T2 is encoded as a B picture by referencing only the forward P picture 1111, so it is encoded two pictures later, i.e., at time T4. On the other hand, a B frame image 1103 input at time T3 is encoded as a B picture by referencing only the backward P picture 1114, so it is encoded three pictures later, i.e., at time T6.

[0079] What differs from the coding order of the normal IBBP structure described in Fig. 13 is that the coding timing of the B frame image 1102, which only references the previous frame, is earlier, and is coded before the P frame image 1104. In network video distribution mode, coding operations are performed in accordance with the reference relationships and coding order described above.

[0080] Next, the operation when an IDR picture insertion request is received will be described. FIG. 10 is a diagram showing the encoding process when a request to insert an IDR picture is received while a frame image is being encoded as a P picture.

[0081] If an IDR picture insertion request is received while the P frame image 1202 is being coded as a P picture, the P frame image 1202 is coded as a P picture at time T5. The next B frame image 1201 to be coded is also coded as a B picture at time T6 as originally planned. The next B frame image 1203 to be coded was originally scheduled to be coded as a B picture, but is now coded as an IDR picture.

[0082] In network video distribution mode, a different GOP structure is used and the encoding order is changed, so that by the time the B frame image 1203 is encoded, all frame images input before the B frame image 1203 have already been encoded. As a result, the B frame image 1203 can be encoded independently of the pictures encoded before it, and can be encoded as an IDR picture. Furthermore, because the B frame image 1203 is encoded at time T7, it is within three pictures after the IDR picture insertion request is received, satisfying the condition of inserting an IDR picture within 100 msec.

[0083] The P frame image 1204 was originally intended to be coded with reference to the P picture 1212, but since the B frame image 1203 was coded as an IDR picture, it is coded as a P picture with reference to the IDR picture 1213.

[0084] Next, with reference to FIG. 11, a description will be given of the encoding process when an IDR picture insertion request is received while a B frame image is being encoded as a B picture.

[0085] If a request to insert an IDR picture is received while a B frame image is being coded as a B picture, the frame image that is scheduled to be coded as the nearest P picture in coding order is changed to an IDR picture and coded.

[0086] If an IDR picture insertion request is received at time T5 while the B frame image 1301 is being coded as a B picture, the B frame image 1301 continues to be coded as a B picture as originally planned. The next B frame image 1302 is also coded as a B picture as originally planned.

[0087] Then, the P frame image 1304 is changed to an IDR picture and coded. The P frame image 1304 is coded at time T7, which is within 3 pictures after the IDR picture insertion request, so it will be coded within 100 msec. If the IDR picture insertion request comes while a B picture is being coded, there will always be a P frame image within 3 pictures, so if that P frame image is coded as an IDR picture, the IDR picture can be coded within 100 msec.

[0088] Also, if an IDR picture insertion request is received while the B frame image 1302, instead of the B frame image 1301, is being coded as a B picture (time T6), the P frame image 1304 can be coded as an IDR picture in the same manner.

[0089] The next B frame image 1303 to be coded is coded as a B picture as originally planned. The reference destination of the B frame image 1303 is the IDR picture 1324 as originally planned. In this way, the only change is that the coding of the P frame image 1304 is changed from a P picture to an IDR picture, but the reference relationship remains the same.

[0090] As described above, in the GOP structure of this embodiment, when a B frame image is coded as a B picture, only the P picture or I picture of the P frame image input at the timing closest to the B frame image is referenced. Because of this structure, even if a P picture is changed to an IDR picture, no reference is made across the P picture. Therefore, even if a P picture is changed to an IDR picture, there is no need to change the reference relationship.

[0091] As described above, if a request to insert an IDR picture is received while a B picture is being coded, the picture is coded as a B picture as usual until the next P frame image arrives, and then the next P frame image is coded as an IDR picture.

[0092] As described above, by receiving a flag indicating whether the video is in network video distribution mode or not, and encoding using a different GOP structure than normal if the video is in network video distribution mode, it is possible to insert an IDR picture within three frames when an IDR picture insertion request is received.

[0093] In the above example, the case where encoding is performed at 30 frames / sec has been described, but similar processing can be performed even in the case of other frame rates.

[0094] As described above, according to the second embodiment, it is possible to realize encoding of an IDR picture within a specified time (for example, 100 msec) after receiving a request to insert an IDR picture.

[0095] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0096] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0097] 11: network camera, 12: network, 13, 14, 15: client, 100: encoding unit, 101: lens, 102: imaging unit, 103: development processing unit, 104: encoding frame buffer, 105: intra prediction unit, 106: inter prediction unit, 107: prediction selection unit, 108: orthogonal transformation unit, 109: quantization unit, 110: quantization control unit, 111: variable length encoding unit, 112: recording medium, 113: inverse quantization unit, 114: inverse orthogonal transformation unit, 115: motion compensation unit, 116: deblocking filter unit, 117: reference frame buffer, 118: network communication unit, 150: control unit, 711: CPU, 712: memory

Claims

1. An image encoding device that predictively encodes moving images, comprising: an intra-frame coding means for performing intra-frame coding on input frame images; an inter-picture coding means for performing inter-picture coding on input frame images; a storage means for storing a copy picture which refers to a key frame image and instructs skip processing of B pictures in all macroblocks; a control means for, when said inter-picture coding means is coding a P frame image to be coded as a P picture as a P picture upon receiving a key frame insertion request, suspending said coding and re-coding said P frame image as a key frame by said intra-picture coding means, and for controlling said inter-picture coding means to replace one of the B frame images to be coded as a B picture with a copy picture held in said holding means without coding it within a predetermined time after receiving said key frame insertion request; An image encoding device comprising:

2. 2. The image coding device according to claim 1, wherein, when the key frame insertion request is received and the inter-picture coding means is coding a B frame image as a B picture, the control means controls the intra-picture coding means to code the P frame image that is to be coded first after the B picture as a key frame.

3. 3. The image coding device according to claim 1, wherein, when the key frame insertion request is received, the inter-picture coding means codes a B frame image as a B picture by referring only to the key frame from when the key frame is generated until the next time a P frame image is coded as a P picture by the inter-picture coding means.

4. An image encoding device that predictively encodes moving images, comprising: encoding means for predictively encoding input frame images; a control means for controlling the encoding means to change a reference destination when encoding a B frame image to be encoded as a B picture as a B picture according to an operation mode; When the operation mode is a video distribution mode for distributing encoded video images, the control means sets the reference destination of the B frame image to the P picture of the P frame image to be coded as a P picture, which is input at a timing closest to the B frame image; an encoding means for encoding, when receiving a key frame insertion request, a first frame image that is input within a predetermined time from receiving the key frame insertion request and that is to be encoded as a key frame, as a key frame;

5. 5. The image encoding device according to claim 4, wherein when the operating mode is a moving image recording mode in which encoded moving images are recorded on a recording medium, the control means controls the image encoding device to use a P picture obtained by encoding two P frame images input before and after a B frame image as a reference.

6. The encoding means an intra-frame coding means for performing intra-frame coding on input frame images; an inter-picture coding means for performing inter-picture coding on input frame images, 6. The image coding device according to claim 4, wherein when the operation mode is the moving image distribution mode and the inter-picture coding means is coding a P frame image as a P picture upon receiving a key frame insertion request, the first B frame image input after the P frame image is coded as a key frame by the intra-picture coding means.

7. The encoding means an intra-frame coding means for performing intra-frame coding on input frame images; an inter-picture coding means for performing inter-picture coding on input frame images, 7. The image coding device according to claim 4, wherein when the operating mode is the moving image distribution mode and the inter-picture coding means is coding a B frame image as a B picture upon receiving a key frame insertion request, the first P frame image input after the B frame image is coded as a key frame by the intra-picture coding means.

8. 8. The image encoding device according to claim 1, wherein encoding is performed according to an encoding method defined in H.264 or HEVC.

9. 9. The image encoding device according to claim 8, wherein the key frame is an IDR picture.

10. An image encoding device according to any one of claims 1 to 9; an imaging means for capturing the moving image; An imaging device comprising:

11. An image coding method for predictively coding a moving image, comprising: a step of generating a copy picture in which all macroblocks instruct skip processing of B pictures by referring to a key frame image; a step of interrupting inter-picture coding when a key frame insertion request is received and the frame image is being coded as a P picture; performing intra-picture coding on the frame image for which the inter-picture coding has been interrupted; replacing one of the frame images to be coded as a B picture with the copy picture without coding it within a predetermined time after receiving the key frame insertion request; An image encoding method comprising:

12. An image coding method for predictively coding a moving image, comprising: a control step of controlling the encoding means to change a reference when encoding a B frame image to be encoded as a B picture as a B picture according to an operation mode, wherein when the operation mode is a moving image distribution mode in which encoded moving images are distributed, the reference of the B frame image is set to a P picture of a P frame image to be encoded as a P picture, which is input at a timing closest to the B frame image; an encoding step of, when the operation mode is a moving image distribution mode for distributing encoded moving images and a key frame insertion request is received, encoding as a key frame a first frame image that is input within a predetermined time after receiving the key frame insertion request and is to be encoded as a key frame; An image encoding method comprising:

13. A program for causing a computer to function as each of the means of the image encoding device according to any one of claims 1 to 9.

14. A computer-readable storage medium storing the program according to claim 13.

Images