Data encoding device, data decoding device, and data processing system

Abstract

A data encoding device 10 which includes an image encoding unit 11 for generating an image bitstream by encoding an image, a feature encoding unit 12 for generating a feature bitstream by encoding the features of the image, and a synchronization unit 13 for synchronizing a random access point in the image bitstream and a random access point in the feature bitstream.

Description

Technical Field

[0001] The present invention relates to a data encoding device, a data decoding device, a data encoding method, a data decoding method, and a data processing system used for task processing of machine recognition.

Background Art

[0002] SC29, which is one of the SC (Sub Committee) of ISO / IEC JTC1, is responsible for standardizing the encoding of audio, images, and multimedia information. WG11, which is one of the WG (Working Group), is responsible for video encoding, media transmission, streaming, audio encoding, etc.

[0003] In WG11, the standardization of a video compression method specialized for machine learning called Video Coding for Machines (VCM) is under consideration. VCM has a function of video encoding suitable for task processing of machine recognition and a function of feature amount encoding used for task processing. Non-Patent Document ￥1 discloses use cases of VCM.

[0004] FIG. 17 is a block diagram showing a data processing system using VCM. The data processing system shown in FIG. 17 includes a VCM encoder 300 and a VCM decoder 400. The VCM encoder 300 and the VCM decoder 400 are communicably connected via, for example, a transmission path. For example, a case where the VCM encoder 300 generates a video bit stream and a feature amount bit stream regarding a feature amount A for a task A of a certain machine process is taken as an example.

[0005] The arrow in FIG. 17 simply indicates the direction of the flow of signals (data), but does not exclude bidirectionality. This also applies to other block diagrams.

[0006] The VCM encoder 300 has a video encoder 301, a feature amount extractor 302, and a feature amount encoder 303.

[0007] The video encoder 301 encodes the video frames (see Figure 17) of the digitized video signal at each time point to generate a video bitstream.

[0008] The feature extractor 302 extracts feature A of the video signal from the video frame at each time step. The feature encoder 303 encodes feature A to generate a feature bitstream.

[0009] The VCM decoder 400 includes a video decoder 401 and a feature decoder 402.

[0010] The video decoder 401 decodes the video bitstream to generate a decoded video signal. The decoded video signal is used for human viewing and visual verification. The decoded video signal may also be used for task X, a machine processing task separate from task A.

[0011] The feature decoder 402 decodes the feature bitstream to generate feature A (decoded feature A). Decoded feature A is used for machine processing task A. [Prior art documents] [Non-patent literature]

[0012] [Non-Patent Document 1] "Use cases and requirements for Video Coding for Machines", ISO / IEC JTC1 / SC29 / WG11 w19365, April 2020 [Non-Patent Document 2] Keiichi Chono, et al., "Reduced-reference image quality assessment using distributed source coding", IEEE International Conference on Multimedia and Expo, June 2008 [Overview of the project] [Problems that the invention aims to solve]

[0013] In the underlying technology, simultaneous random access to the video bitstream and feature bitstream at a given time point is not guaranteed.

[0014] Therefore, the present invention aims to guarantee simultaneous random access to the video bitstream and the feature bitstream. [Means for solving the problem]

[0015] The data encoding device according to the present invention includes video encoding means for encoding video and generating a video bitstream, feature encoding means for encoding video feature quantities and generating a feature quantity bitstream, and synchronization means for synchronizing random access points in the video bitstream and random access points in the feature quantity bitstream.

[0016] The data decoding device according to the present invention includes a video decoding means that receives and decodes a video bitstream in which video is encoded and generated, and in which random access points are set; and a feature decoding means that receives and decodes a feature bitstream in which video feature quantities are encoded and generated, and in which random access points that are temporally aligned with the random access points are set.

[0017] The data encoding method according to the present invention encodes video to generate a video bitstream, encodes video features to generate a feature bitstream, and synchronizes random access points in the video bitstream with random access points in the feature bitstream.

[0018] The data decoding method according to the present invention receives a video bitstream in which video is encoded and a random access point is set, and performs decoding, and receives a feature amount bitstream in which the feature amount of the video is encoded and a random access point synchronized with the random access point in terms of time, and performs decoding.

[0019] The data encoding program according to the present invention causes a computer to execute a process of encoding video to generate a video bitstream, a process of encoding a feature amount of the video to generate a feature amount bitstream, and a process of synchronizing a random access point in the video bitstream and a random access point in the feature amount bitstream.

Advantages of the Invention

[0020] According to the present invention, simultaneous random access to the video bitstream and the feature amount bitstream is guaranteed.

Brief Description of the Drawings

[0022] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0023] Figure 1 is an explanatory diagram illustrating an example of the temporal relationship between a video bitstream (video frame data) and a feature bitstream (feature frame data). In Figure 1, arrows indicate frame dependencies. Circles indicate frames that have no dependencies on other frames. That is, frames marked with circles are frames in which self-contained encoding and decoding processes are performed. In the temporal direction, the location where such frames exist is called a random access point. In Figure 1, the width of the rectangle representing a frame corresponds to the amount of data. For example, a random access point frame in a video bitstream is an encoded I (Intra-coded) video frame.

[0024] In the example shown in Figure 1, the random access points in the video bitstream and the random access points in the feature bitstream are not synchronized. That is, the random access points in the video bitstream and the random access points in the feature bitstream are out of sync in time. Synchronization of random access points means that the random access points in both bitstreams appear at the same time. Simultaneous random access means that the random access points in both bitstreams are at the same time.

[0025] If the random access points in the video bitstream and the random access points in the feature bitstream are out of sync, for example, the machine recognition task using feature A cannot be started at the time when the video decoder 401 plays back the frame of the random access point (for example, at time t+2 shown in Figure 1).

[0026] It is possible for both random access points to coincide. In that case, the total amount of data (total data volume) at the time the random access points appear increases. In other words, the amount of data transmitted increases instantaneously.

[0027] Next, we will discuss predictive coding. Predictive coding includes intra-predictive and inter-predictive coding. First, we will explain intra-predictive and inter-predictive coding. We will begin by explaining intra-predictive and inter-predictive coding in the context of a video bitstream.

[0028] Intra prediction is a prediction method that does not use images from reconstructed video frames whose display time differs from that of the video frame to be encoded. Hereinafter, image blocks encoded based on intra prediction are referred to as intra-encoded image blocks. Inter prediction is a prediction method that uses images from reconstructed video frames whose display time differs from that of the video frame to be encoded. Hereinafter, image blocks encoded based on inter prediction are referred to as inter-encoded image blocks. Inter prediction is also known as motion-compensated prediction.

[0029] An encoded video frame consisting only of intra-encoded image blocks is called an I-video frame. Encoded video frames that do not consist only of intra-encoded image blocks are called non-I-video frames. Non-I-video frames include P (Predictive) video frames and B (Bi-directional predicted) video frames. A B-video frame can include an inter-encoded image block that uses one reconstructed video frame for inter-prediction, and an inter-encoded image block that uses two reconstructed video frames simultaneously for inter-prediction.

[0030] The concepts of intra-prediction and inter-prediction described above can also be applied to feature bitstreams.

[0031] In the case of a feature bitstream, a prediction that does not use features from video frames whose display time differs from that of the video frame to be encoded is called an intra-prediction of features. Hereinafter, features encoded based on intra-predictions are referred to as intra-encoded features. Furthermore, a prediction that uses features from video frames whose display time differs from that of the video frame to be encoded is called an inter-prediction of features. Hereinafter, features encoded based on inter-predictions are referred to as inter-encoded features.

[0032] A frame consisting solely of intra-encoded features is called an I-feature frame. Frames that do not consist solely of intra-encoded features are called non-I-feature frames.

[0033] Embodiment 1. Figure 2 is a block diagram showing a data processing system according to the first embodiment. The data processing system shown in Figure 2 includes a VCM encoder 100 and a VCM decoder 200. The VCM encoder 100 and the VCM decoder 200 are communicated together, for example, via a transmission line. In this embodiment, we take the example where the VCM encoder 100 generates a video bitstream and a feature bitstream relating to feature A for a certain machine processing task A.

[0034] The VCM encoder 100 includes a video encoder 101, a feature extractor 102, a feature encoder 103, and a controller 104.

[0035] The video encoder 101 generates a video bitstream by encoding each video frame of the digitized video signal at each moment using the video frame type (I video frame type or non-I video frame type) supplied by the controller 104. For example, the video encoder 101 performs the encoding process based on the H.266 / VVC (Versatile Video Coding) standard. The video encoder 101 may also perform the encoding process based on other standards such as the H.265 / HEVC (High Efficiency Video Coding) standard or the H.264 / AVC (Advanced Video Coding) standard.

[0036] The feature extractor 102 extracts feature A of the video signal from the video frame at each time step. The feature encoder 103 encodes feature A using the feature frame type (I-feature frame type or non-I-feature frame type) supplied by the controller 104 to generate a feature bitstream.

[0037] The controller 104 determines the type of video frame (video frame type) and the type of feature frame (feature frame type) to be encoded by the video encoder 101 and the feature encoder 103, respectively. The controller 104 supplies the video frame type as a control signal to the video encoder 101. The controller 104 also outputs the feature frame type as a control signal to the feature encoder 103.

[0038] The controller 104 synchronizes the output timing of the control signal indicating a video frame with the output timing of the control signal indicating a feature frame, so that simultaneous random access to the video bitstream and the feature bitstream is guaranteed. For example, the controller 104 synchronizes when a predetermined period has elapsed. Guaranteeing simultaneous random access to the video bitstream and the feature bitstream means, for example, that the random access points in the feature bitstream and the random access points in the video bitstream are guaranteed to coincide in time.

[0039] Furthermore, synchronizing the random access points of the video frame and the random access points of the feature frame means aligning the random access points of both in time. Specifically, for example, in response to instructions from the controller 104, for a single video frame input at a certain time, the video encoder 101 performs encoding based on the I-video frame type, while the feature encoder 103 performs encoding based on the I-feature frame type.

[0040] The VCM decoder 200 includes a video decoder 201 and a feature decoder 202.

[0041] The video decoder 201 decodes the video bitstream to generate a decoded video signal. The decoded video signal is used for human viewing and visual verification. The decoded video signal may also be used for task X, a machine processing task separate from task A (see Figure 17).

[0042] The feature decoder 202 decodes the feature bitstream to generate decoded feature A. Decoded feature A is used for machine processing task A (see Figure 17).

[0043] Next, the operation of the VCM encoder 100 in the first embodiment will be described. Figure 3 is a flowchart of the operation of the VCM encoder 100. The process shown in Figure 3 is executed for each frame.

[0044] In the VCM encoder 100, the controller 104 determines the video frame type and the feature frame type (step S100). As described above, for example, the controller 104 periodically determines the video frame type to an I-video frame type and the feature frame type to an I-feature frame type. When periodicity is used to determine the frame type, if a predetermined period has not elapsed, the controller 104 determines the video frame type to a non-I-video frame type and the feature frame type to a non-I-feature frame type.

[0045] The controller 104 then outputs a control signal indicating the video frame type to the video encoder 101 and a control signal indicating the feature frame type to the feature encoder 103.

[0046] The video encoder 101 encodes video frames using the video frame type instructed by the control signal from the controller 104 to generate a video bitstream (step S101).

[0047] The feature extractor 102 extracts features from the video frame (step S102). The feature encoder 103 encodes the features using the feature frame type instructed by the control signal from the controller 104 to generate a feature bitstream (step S103).

[0048] The video encoder 101 sends the video bitstream to, for example, the transmission path (step S104). The feature encoder 103 sends the feature bitstream to, for example, the transmission path (step S105).

[0049] Next, the operation of the VCM decoder 200 in the first embodiment will be described. Figure 4 is a flowchart showing the operation of the VCM decoder 200.

[0050] In the VCM decoder 200, the video decoder 201 decodes the received video bitstream to generate a decoded video signal (step S201). The feature decoder 202 decodes the feature bitstream to generate decoded features (step S202).

[0051] The video decoder 201 outputs the decoded video signal (step S203). The feature decoder 202 outputs the decoded features (step S204).

[0052] Figure 5 is an explanatory diagram illustrating an example of the temporal relationship between the video bitstream and the feature bitstream in the first embodiment. In Figure 5, the arrows indicate frames at random access points. Frames at random access points are frames that do not have dependencies on other frames. The width of the rectangles representing frames corresponds to the amount of data.

[0053] As illustrated in Figure 5, in this embodiment, simultaneous random access to the video bitstream and the feature bitstream is guaranteed, so that the random access points in the video bitstream and the random access points in the feature bitstream are aligned.

[0054] As a result, a system that utilizes the video bitstream and feature bitstream output by the VCM decoder 200 can immediately begin a machine recognition task based on feature data at the same time as a given video frame.

[0055] Embodiment 2. Figure 6 is a block diagram showing a data processing system of the second embodiment. The data processing system shown in Figure 6 includes a VCM encoder 110 and a VCM decoder 210. The VCM encoder 110 and the VCM decoder 210 are communicated together, for example, via a transmission line. In this embodiment, we will take the example where the VCM encoder 110 generates a video bitstream and a feature bitstream relating to feature A for a certain machine processing task A.

[0056] The VCM encoder 110 includes a video encoder 101, a feature extractor 102, a side information generator 105, a feature encoder 106, and a controller 104. The functions of the video encoder 101, the feature extractor 102, and the controller 104 are the same as those in the first embodiment.

[0057] The side information generator 105 generates information related to feature A extracted by the feature extractor 102 from the reconstructed video signal generated inside the video encoder 101. Hereinafter, the information correlated with feature A extracted by the feature extractor 102 will be referred to as side information A'.

[0058] The reconstructed video signal in the video encoder 101 is the video signal (video frame) generated by the decoding function included in the video encoder 101. Furthermore, the side information generator 105 generates side information A' by, for example, the same process performed by the feature extractor 102.

[0059] In addition to the functions of the feature encoder 103 in the first embodiment, the feature encoder 106 has the following functions. That is, when encoding an I-type feature frame, the feature encoder 106 reduces the amount of data in the feature frame by encoding the feature frame based on the correlation between feature A and side information A'.

[0060] Let's consider an example where the side information generator 105 generates side information A' from the reconstructed video signal in the video encoder 101 using the same process as the feature extractor 102. In this case, the side information generator 105 extracts features from the reconstructed video signal as side information A'. The feature encoder 106 then reduces the amount of data in the feature frame by predictively encoding feature A with the features extracted by the side information generator 105 (side information A'). This is because the feature A of the input video signal and the features of the reconstructed video signal are similar. Note that the higher the bitrate of the video bitstream, the higher the similarity between the feature A of the video signal and the reconstructed video signal.

[0061] Furthermore, when the feature encoder 106 performs coding of non-I feature frame types, it may also encode the feature frame by utilizing the correlation between feature A and side information A'.

[0062] The VCM decoder 210 includes a video decoder 201, a side information generator 203, and a feature decoder 204. The video decoder 201 is the same as that in the first embodiment. That is, the video decoder 201 decodes the video bitstream and generates a decoded video signal. The decoded video signal is used for human viewing and visual verification. The decoded video signal may also be used for task X, which is a separate machine processing task from task A (see Figure 17).

[0063] The side information generator 203 generates side information A' in the same manner as the side information generator 105 in the VCM encoder 110 when an I-frame type video frame is decoded. However, while the side information generator 105 generates side information A' from the reconstructed video signal, the side information generator 203 generates side information A' from the decoded video signal.

[0064] In addition to the functions of the feature decoder 202 in the first embodiment, the feature decoder 204 has the following functions. That is, when decoding a feature frame of type I, the feature decoder 204 decodes the feature frame based on the correlation between feature A and side information A'.

[0065] The feature decoder 204 can decode the feature frame by performing the inverse operation of the predictive coding performed by the feature encoder 106. For example, the feature decoder 204 predictively decodes the features decoded from the feature bitstream using the features extracted by the side information generator 105.

[0066] Next, the operation of the VCM encoder 110 in the second embodiment will be described. Figure 7 is a flowchart showing the operation of the VCM encoder 110. The process shown in Figure 7 is executed for each frame.

[0067] The processing in steps S100 to S102 is the same as the processing in the first embodiment.

[0068] In this embodiment, the side information generator 105 generates side information A' from the reconstructed video signal generated inside the video encoder 101 (step S301).

[0069] The feature encoder 106 predictively encodes feature A using the feature extracted by the side information generator 105 (side information A') to generate a feature bitstream (step S302).

[0070] Similar to the first embodiment, the video encoder 101 sends the video bitstream to, for example, the transmission path (step S104). Similar to the first embodiment, the feature encoder 103 sends the feature bitstream to, for example, the transmission path (step S105).

[0071] Next, the operation of the VCM decoder 210 in the second embodiment will be described. Figure 8 is a flowchart showing the operation of the VCM decoder 210.

[0072] Similar to the first embodiment, the video decoder 201 decodes the received video bitstream to generate a decoded video signal (step S201). The side information generator 203 generates side information A' from the decoded video signal when an I-video frame type video frame is decoded (step S401).

[0073] The feature decoder 204 decodes the feature bitstream to generate decoded features, as in the first embodiment (step S402). However, as described above, in this embodiment, when decoding a feature frame of type I, the feature decoder 204 decodes the feature frame by utilizing the correlation between feature A and side information A'.

[0074] The video decoder 201 outputs the decoded video signal (step S203). The feature decoder 204 outputs the decoded features (step S204).

[0075] Figure 9 is an explanatory diagram illustrating an example of the temporal relationship between the video bitstream and the feature bitstream in the first embodiment. In Figure 9, arrows (excluding the arrow related to "reducing data volume") indicate frames at random access points. Frames at random access points are frames that do not have dependencies on other frames. The width of the rectangle representing a frame corresponds to the data volume.

[0076] As illustrated in Figure 9, in this embodiment as well, simultaneous random access to the video bitstream and the feature bitstream is guaranteed, so the random access points in the video bitstream and the random access points in the feature bitstream are synchronized.

[0077] Furthermore, in this embodiment, since the feature frame is encoded using the correlation between feature A and side information A', the amount of data in the feature frame at random access points is reduced. As a result, the increase in the total amount of data at random access points is suppressed.

[0078] [Example 1] In the first and second embodiments, as illustrated in Figure 10, the video encoder 101 and feature encoders 103 and 106 may add identification headers to the beginning of the video frame data and the feature frame data to identify random access points.

[0079] The header for identifying video frame data includes information indicating whether it is an I-frame, P-frame, or B-frame. The header for identifying feature frame data includes information indicating whether it is an I-feature frame or a non-I-feature frame.

[0080] In the second embodiment, the identification header of the feature frame data further includes information indicating whether or not it was encoded using side information.

[0081] By adding the above-mentioned identification header, the decoding side can verify random access without having to decode the contents of the video frame data and feature frame data. Therefore, interoperability between the encoding side and the decoding side is further enhanced.

[0082] [Differentiation 2] In the second embodiment, the features extracted from the video frames of the reconstructed video signal using the same process as the feature extractor 102 performs were defined as side information A'. However, it is also possible to use the encoded data itself as side information. Hereafter, the features extracted using the same process as the feature extractor 102 performs will be referred to as feature A'.

[0083] For example, error correction codes can be applied as described in Non-Patent Document 2.

[0084] Let the encoded data of feature quantity A extracted by feature quantity extractor 102 from a video frame at a certain time t of the input video signal be Y(A, t). Also, let the encoded data of feature quantity A’ be side information Y(A’, t).

[0085] The encoding of feature quantity A’ is as shown in FIG. 11. That is, the feature quantity encoder 106 uses the syndrome generated by multiplying the encoding data Y(A, t) to be sent by the parity check matrix as the encoding data.

[0086] As shown in FIG. 11, the feature quantity encoder 106 determines the number of columns M of the parity check matrix based on the correlation between the encoding data Y(A, t) (N-bit data) to be sent and the side information Y(A’, t). The feature quantity encoder 106 multiplies the encoding data Y(A, t) by a parity check matrix of N rows and M columns to generate a syndrome (M-bit data). Then, the feature quantity encoder 106 outputs the syndrome as the feature quantity frame data. If M < N, the data amount is reduced.

[0087] The decoding of feature quantity A’ is as shown in FIG. 12. That is, the feature quantity decoder 204 can obtain the decoded feature quantity by using the corrected Y(A, t) as the decoded value from the relationship between the side information Y(A’, t), the parity check matrix, and the syndrome (M-bit data).

[0088] Note that by pre-sharing the values of each element of the parity check matrix between the encoding side and the decoding side, the data amount of each feature quantity frame data can be further reduced. In that case, the feature quantity encoder 106 only needs to include the ID of the parity check matrix used with the number of columns M in the identification header.

[0089] In each of the above embodiments and each modification example, for the sake of simplicity of explanation, one feature quantity is taken as an example. However, the present invention is applicable to a plurality of feature quantities.

[0090] Furthermore, when applying the encoding and decoding of feature data using the error correction code of the second modification to multiple features, the syndrome may be calculated collectively on the concatenated data of all feature frames, rather than independently calculating the syndrome for each feature frame data. In this case, if the feature frame data for each feature is short, the overhead caused by applying a check matrix with a small number of columns can be suppressed.

[0091] Figure 13 is a block diagram showing a specific configuration example of a data processing system. The data processing system shown in Figure 13 is a system in which a data encoding device 10 (corresponding to the VCM encoder 100 in the first embodiment or the VCM encoder 110 in the second embodiment) and a data decoding device 20 (corresponding to the VCM decoder 200 in the first embodiment or the VCM decoder 210 in the second embodiment) are connected by a transmission line (wireless transmission line or wired transmission line) 30.

[0092] In the data processing system, the data encoding device 10 can generate a video bitstream and a feature bitstream having the characteristics described in each of the above embodiments. Furthermore, in the data processing system, the data decoding device 20 can decode the video bitstream and the feature bitstream having the characteristics described in each of the above embodiments.

[0093] Furthermore, while each of the above embodiments can be implemented using hardware, it can also be realized using computer programs.

[0094] The information processing system shown in Figure 14 comprises a processor 1001 such as a CPU (Central Processing Unit), a program memory 1002, a storage medium 1003 for storing video data, and a storage medium 1004 for storing bitstreams. The storage mediums 1003 and 1004 may be separate storage mediums or may be storage areas consisting of the same storage medium. Magnetic storage media such as hard disks can be used as storage media.

[0095] In the information processing system, the program memory 1002 stores programs (data encoding programs or data decoding programs) for realizing the functions of each block shown in the above embodiments.

[0096] The processor 1001 then performs processing according to the program stored in the program memory 1002, thereby realizing the functions of the data encoding device 10, VCM encoders 100, 110, data decoding device 20, and VCM decoders 200, 210 as shown in each embodiment.

[0097] For example, the VCM encoder 100's functionality is realized when the processor 1001 executes processing according to a data encoding program (specifically, a VCM encoding program) for realizing the functions of each block in the VCM encoder 100 shown in Figure 2. Similarly, the VCM decoder 200's functionality is realized when the processor 1001 executes processing according to a data decoding program (specifically, a VCM decoding program) for realizing the functions of each block in the VCM decoder 200 shown in Figure 2.

[0098] Furthermore, at least the program memory 1002 is a non-transitory computer-readable medium. However, the program may be stored in various types of transient computer-readable medium. The program is supplied to the transient computer-readable medium, for example, via a wired communication channel or a wireless communication channel, i.e., via electrical signals, optical signals, or electromagnetic waves.

[0099] Figure 15 is a block diagram showing the main parts of a data encoding device. The data encoding device 10 shown in Figure 15 comprises a video encoding unit (video encoding means) 11 (implemented by a video encoder 101 in this embodiment) that encodes video and generates a video bitstream, a feature encoding unit (feature encoding means) 12 (implemented by feature encoders 103 and 106 in this embodiment) that encodes video feature quantities and generates a feature quantity bitstream, and a synchronization unit (synchronization means) 13 (implemented by a controller 104 in this embodiment) that synchronizes random access points in the video bitstream and random access points in the feature quantity bitstream.

[0100] The data encoding device 10 includes a first feature extraction unit (first feature extraction means: in this embodiment, implemented by a side information generator 105) that extracts feature quantities from the input video, and a second feature extraction unit (second feature extraction means: in this embodiment, implemented by a feature encoder 106) that extracts feature quantities from the video generated by the decoding process included in the encoding process by the video encoding unit 11. The feature encoding unit 12 may be configured to encode features based on the correlation between the feature quantities extracted from the input video and the feature quantities extracted from the video generated by the decoding process.

[0101] Figure 16 is a block diagram showing the main parts of a data decoding device. The data decoding device 20 shown in Figure 16 includes a video decoding unit (video decoding means) 21 that receives and decodes a video bitstream in which video has been encoded and generated, and in which random access points have been set, and a feature decoding unit (feature decoding means) 22 that receives and decodes a feature bitstream in which video feature quantities have been encoded and generated, and in which random access points that are temporally aligned with the random access points have been set.

[0102] The data decoding device 20 includes a feature extraction unit (feature extraction means: in this embodiment, implemented by a side information generator 203) that extracts feature quantities of the decoded video obtained by decoding the video bitstream, and the feature decoding unit 22 may be configured to perform decoding based on the correlation between the feature quantities obtained by decoding the feature bitstream and the feature quantities extracted from the decoded video.

[0103] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0104] (Note 1) A video encoding means that encodes video and generates a video bitstream, A feature encoding means that encodes the feature quantities of the video and generates a feature quantity bitstream, A synchronization means for synchronizing random access points in the video bitstream and random access points in the feature bitstream. A data encoding device equipped with the following features.

[0105] (Note 2) A first feature extraction means for extracting feature quantities from the input video, The system includes a second feature extraction means for extracting feature quantities of the video generated by the decoding process included in the encoding process by the aforementioned video encoding means, The feature encoding means performs feature encoding based on the correlation between the feature quantities extracted from the input video and the feature quantities extracted from the video generated by the decoding process. The data encoding device mentioned in Appendix 1.

[0106] (Note 3) The feature encoding means predictively encodes the features extracted from the input video using the features extracted from the video generated by the decoding process. The data encoding device mentioned in Appendix 2.

[0107] (Note 4) Video decoding means that receives and decodes a video bitstream that has been encoded and generated, and to which random access points have been set. A feature decoding means receives and decodes a feature bitstream in which the feature quantities of the aforementioned video are encoded and generated, and in which random access points that are temporally aligned with the aforementioned random access points are set. A data decoding device equipped with the following features.

[0108] (Note 5) The system includes a feature extraction means for extracting feature quantities of the decoded video obtained by decoding the video bitstream, The feature decoding means performs decoding based on the correlation between the features obtained by decoding the feature bitstream and the features extracted from the decoded video. Data decoding device as described in Appendix 4.

[0109] (Note 6) The feature decoding means predicts and decodes the features decoded from the feature bitstream using the features extracted by the feature extraction means. Data decoding device as described in Appendix 5.

[0110] (Note 7) Encode the video to generate a video bitstream, The feature quantities of the aforementioned video are encoded to generate a feature quantity bitstream, The random access points in the video bitstream and the random access points in the feature bitstream are synchronized. Data encoding method.

[0111] (Note 8) Extract the feature quantities of the input video, The video features generated by the decoding process included in the video encoding process are extracted, The feature quantities are encoded based on the correlation between the feature quantities extracted from the input video and the feature quantities extracted from the video generated by the decoding process. Data encoding method as described in Appendix 7.

[0112] (Note 9) The video is encoded and generated, and a random access point is set to receive and decode the video bitstream. The feature quantities of the aforementioned video are encoded and generated, and a feature quantity bitstream in which random access points that are temporally aligned with the aforementioned random access points are received and decoded. Data decoding method.

[0113] (Note 10) The feature quantities of the decoded video obtained by decoding the video bitstream are extracted, Decoding is performed based on the correlation between the features obtained by decoding the feature bitstream and the features extracted from the decoded video. Data decoding method as described in Appendix 9.

[0114] (Note 11) A data encoding device from any of Notes 1 to 3, Data decoding device and A data processing system equipped with the following features.

[0115] (Note 12) To the computer, The process of encoding video and generating a video bitstream, The process involves encoding the feature quantities of the aforementioned video to generate a feature quantity bitstream, A process for synchronizing random access points in the video bitstream with random access points in the feature bitstream. A data encoding program to execute this.

[0116] (Note 13) To the computer, The process involves receiving and decoding a video bitstream that has been encoded and generated, and which has been configured with random access points. The process involves encoding and generating the feature quantities of the aforementioned video, receiving and decoding a feature quantity bitstream in which random access points are set to be temporally aligned with the aforementioned random access points, and A data encoding program to execute this.

[0117] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made, which can be understood by those skilled in the art within the scope of the present invention. [Explanation of Symbols]

[0118] 10 Data Encoding Device 11. Video Encoding Section 12 Feature Encoding Unit 13 Synchronization Unit 20 Data Decoder 21 Video Decoding Unit 22 Feature decoding unit 30 transmission lines 100,110 VCM encoder 101 Video encoder 102 Feature Extractor 103,106 Feature Encoder 104 Controller 105 Side Information Generator 200,210 VCM Decoders 201 Video Decoder 202,204 Feature Decoder 203 Side Information Generator 1001 Processor 1002 Program Memory 1003,1004 Storage medium

Claims

1. A video encoding means for encoding video and generating a video bitstream, A feature encoding means that encodes the feature quantities of the video and generates a feature quantity bitstream, A synchronization means for synchronizing random access points in the video bitstream and random access points in the feature bitstream. A data encoding device equipped with the following features.

2. A first feature extraction means for extracting feature quantities from the input video, The system includes a second feature extraction means for extracting feature quantities of the video generated by the decoding process included in the encoding process by the aforementioned video encoding means, The feature encoding means performs feature encoding based on the correlation between the feature quantities extracted from the input video and the feature quantities extracted from the video generated by the decoding process. The data encoding device according to claim 1.

3. A video decoding means that receives and decodes a video bitstream that has been encoded and generated, and to which random access points have been set. A feature decoding means receives and decodes a feature bitstream in which the feature quantities of the aforementioned video are encoded and generated, and in which random access points that are temporally aligned with the aforementioned random access points are set. A data decoding device equipped with the following features.

4. The system includes a feature extraction means for extracting feature quantities of the decoded video obtained by decoding the aforementioned video bitstream, The feature decoding means performs decoding based on the correlation between the features obtained by decoding the feature bitstream and the features extracted from the decoded video. The data decoding device according to claim 3.

5. Encode the video to generate a video bitstream, The feature quantities of the aforementioned video are encoded to generate a feature quantity bitstream, The random access points in the video bitstream and the random access points in the feature bitstream are synchronized. Data encoding method.

6. Extract the feature quantities of the input video, The video features generated by the decoding process included in the video encoding process are extracted, The feature quantities are encoded based on the correlation between the feature quantities extracted from the input video and the feature quantities extracted from the video generated by the decoding process. The data encoding method according to claim 5.

7. The video is encoded and generated, and a random access point receives and decodes the video bitstream. The feature quantities of the aforementioned video are encoded and generated, and a feature quantity bitstream in which random access points that are temporally aligned with the aforementioned random access points are received and decoded. Data decoding method.

8. The aforementioned video bitstream is decoded, and the feature quantities of the decoded video obtained are extracted. Decoding is performed based on the correlation between the features obtained by decoding the feature bitstream and the features extracted from the decoded video. The data decoding method according to claim 7.

9. A data encoding device according to claim 1 or claim 2, The data decoding device according to claim 3 or claim 4 and A data processing system equipped with the following features.

10. On the computer, The process of encoding video and generating a video bitstream, The process involves encoding the feature quantities of the aforementioned video to generate a feature quantity bitstream, A process for synchronizing random access points in the video bitstream with random access points in the feature bitstream. A data encoding program to execute this.

Images