Data encoding device, data decoding device and data processing system

Abstract

A data encoding device 10 which includes an image encoding unit 11 for generating an image bitstream by encoding an image, a feature encoding unit 12 for generating a feature bitstream by encoding the features of the image, and a synchronization unit 13 for synchronizing a random access point in the image bitstream and a random access point in the feature bitstream.

Description

Data encoding device, data decoding device and data processing system

[0001] The present invention relates to a data encoding device, a data decoding device, a data encoding method, a data decoding method, and a data processing system used for processing machine recognition tasks.

[0002] SC29, one of the subcommittees (SCs) of ISO / IEC JTC1, is responsible for standardizing the coding of audio, video, and multimedia information. WG11, one of the working groups (WGs), is responsible for video coding, media transmission, streaming, audio coding, etc.

[0003] WG11 is studying the standardization of a video compression method called Video Coding for Machines (VCM) that is specialized for machine learning. VCM has a video coding function suitable for machine recognition task processing and a feature coding function used for task processing. Non-Patent Document 1 discloses use cases of VCM.

[0004] Fig. 17 is a block diagram showing a data processing system that uses VCM. The data processing system shown in Fig. 17 includes a VCM encoder 300 and a VCM decoder 400. The VCM encoder 300 and the VCM decoder 400 are communicably connected via, for example, a transmission path. Take the example of a case in which the VCM encoder 300 generates a video bitstream and a feature bitstream related to feature A for task A of a certain machine processing.

[0005] 17 simply indicates the direction of signal (data) flow, but does not exclude bidirectionality. This also applies to other block diagrams.

[0006] The VCM encoder 300 includes a video encoder 301 , a feature extractor 302 , and a feature encoder 303 .

[0007] The video encoder 301 encodes video frames (see FIG. 17) at each time of the digitized video signal to generate a video bitstream.

[0008] The feature extractor 302 extracts a feature A of the video signal from the video frame at each time point. The feature encoder 303 encodes the feature A to generate a feature bit stream.

[0009] The VCM decoder 400 includes a video decoder 401 and a feature decoder 402 .

[0010] The video decoder 401 decodes the video bitstream to generate a decoded video signal. The decoded video signal is used for human viewing or visual confirmation. The decoded video signal may also be used for task X, which is a machine processing task separate from task A.

[0011] The feature decoder 402 decodes the feature bitstream to generate feature A (decoded feature A). The decoded feature A is used for task A of machine processing.

[0012] "Use cases and requirements for Video Coding for Machines", ISO / IEC JTC1 / SC29 / WG11 w19365, April 2020Keiichi Chono, et al., "Reduced-reference image quality assessment using distributed source coding", IEEE International Conference on Multimedia and Expo, June 2008

[0013] In the background art, simultaneous random access to the video bitstream and the feature bitstream at a given time is not guaranteed.

[0014] Therefore, an object of the present invention is to guarantee simultaneous random access to a video bitstream and a feature bitstream.

[0015] A data encoding device according to the present invention includes video encoding means for encoding video to generate a video bitstream, feature encoding means for encoding features of the video to generate a feature bitstream, and synchronization means for synchronizing random access points in the video bitstream with random access points in the feature bitstream.

[0016] A data decoding device according to the present invention includes a video decoding means for receiving and decoding a video bitstream in which video is encoded and a random access point is set, and a feature decoding means for receiving and decoding a feature bitstream in which video features are encoded and a random access point is set that is temporally aligned with the random access point.

[0017] A data encoding method according to the present invention encodes video to generate a video bitstream, encodes features of the video to generate a feature bitstream, and synchronizes random access points in the video bitstream with random access points in the feature bitstream.

[0018] The data decoding method according to the present invention receives and decodes a video bitstream in which video is encoded and generated and a random access point is set, and receives and decodes a feature bitstream in which video features are encoded and generated and a random access point that is temporally aligned with the random access point.

[0019] The data encoding program according to the present invention causes a computer to execute a process of encoding video to generate a video bitstream, a process of encoding features of the video to generate a feature bitstream, and a process of synchronizing random access points in the video bitstream with random access points in the feature bitstream.

[0020] According to the present invention, simultaneous random access to the video bitstream and the feature bitstream is guaranteed.

[0021] FIG. 1 is an explanatory diagram showing an example of the relationship in the time direction between a video bitstream and a feature bitstream. FIG. 2 is a block diagram showing a data processing system of a first embodiment. FIG. 3 is a flowchart showing the operation of a VCM encoder in the first embodiment. FIG. 4 is a flowchart showing the operation of a VCM decoder in the first embodiment. FIG. 5 is an explanatory diagram showing an example of the relationship in the time direction between a video bitstream and a feature bitstream. FIG. 6 is a block diagram showing a data processing system of a second embodiment. FIG. 7 is a flowchart showing the operation of a VCM encoder in the second embodiment. FIG. 8 is a flowchart showing the operation of a VCM decoder in the second embodiment. FIG. 9 is an explanatory diagram showing an example of the relationship in the time direction between a video bitstream and a feature bitstream. FIG. 10 is an explanatory diagram showing a first modified example of a data processing system. FIG. 11 is an explanatory diagram showing encoding of feature frame data using error correction codes. FIG. 12 is an explanatory diagram showing decoding of feature frame data using error correction codes. FIG. 13 is a block diagram showing a specific example of the configuration of a data processing system. FIG. 14 is a block diagram showing an example of the configuration of an information processing system. FIG. 15 is a block diagram showing the main parts of a data encoding device. FIG. 16 is a block diagram showing the main parts of a data encoding device. FIG. 17 is a block diagram showing a data processing system using VCM.

[0022] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0023] FIG. 1 is an explanatory diagram showing an example of the relationship in the time direction between a video bitstream (video frame data) and a feature bitstream (feature frame data). In FIG. 1, arrows indicate frame dependencies. Circles indicate frames that have no dependency relationship with other frames. In other words, frames marked with circles are frames for which closed encoding and decoding processes are performed within the frame. A position in the time direction where such a frame exists is called a random access point. Note that in FIG. 1, the width of the rectangle representing a frame corresponds to the amount of data. Furthermore, for example, a frame at a random access point in a video bitstream is an encoded I (Intra coded) video frame.

[0024] In the example shown in Figure 1, the random access points in the video bitstream and the random access points in the feature bitstream are not synchronized. That is, the random access points in the video bitstream and the random access points in the feature bitstream are shifted in time. Note that synchronized random access points mean that the random access points appear at the same time in both bitstreams. Simultaneous random access means that the random access points appear at the same time in both bitstreams.

[0025] If the random access point in the video bitstream is misaligned with the random access point in the feature bitstream, then, for example, it is not possible to start a machine recognition task using feature A at the time when the video decoder 401 plays back the frame at the random access point (e.g., time t+2 shown in Figure 1).

[0026] The random access points may coincide with each other. In this case, the total amount of data increases when the random access point appears. In other words, the amount of data transmitted increases instantaneously.

[0027] Next, predictive coding will be described. Predictive coding includes intra prediction and inter prediction. First, intra prediction and inter prediction will be described. First, intra prediction and inter prediction will be described with respect to a video bitstream.

[0028] Intra prediction is prediction that does not use an image of a reconstructed video frame whose display time is different from that of the video frame at the time of encoding. Hereinafter, an image block coded based on intra prediction is referred to as an intra-coded image block. Inter prediction is prediction that uses an image of a reconstructed video frame whose display time is different from that of the video frame at the time of encoding. Hereinafter, an image block coded based on inter prediction is referred to as an inter-coded image block. Note that inter prediction is also called motion compensated prediction.

[0029] An encoded video frame consisting only of intra-coded image blocks is called an I video frame. An encoded video frame that is not composed of intra-coded image blocks is called a non-I video frame. Non-I video frames include P (Predictive) video frames and B (Bi-directional predicted) video frames. B video frames may include inter-coded image blocks that use one reconstructed video frame for inter prediction and inter-coded image blocks that use two reconstructed video frames simultaneously for inter prediction.

[0030] The above concepts of intra-prediction and inter-prediction can also be applied to feature bitstreams.

[0031] In the case of a feature bitstream, prediction that does not use features of a video frame whose display time is different from that of the video frame at the target encoding time is referred to as intra-prediction for features. Hereinafter, features coded based on intra-prediction will be referred to as intra-coded features. Also, prediction that uses features of a video frame whose display time is different from that of the video frame at the target encoding time is referred to as inter-prediction for features. Hereinafter, features coded based on inter-prediction will be referred to as inter-coded features.

[0032] A frame consisting of only intra-coded features is called an I-feature frame, and other frames are called non-I-feature frames.

[0033] Embodiment 1. Fig. 2 is a block diagram showing a data processing system of a first embodiment. The data processing system shown in Fig. 2 includes a VCM encoder 100 and a VCM decoder 200. The VCM encoder 100 and the VCM decoder 200 are communicably connected via, for example, a transmission path. In this embodiment, an example is taken of a case in which the VCM encoder 100 generates a video bitstream and a feature bitstream related to feature A for task A of a certain machine processing.

[0034] The VCM encoder 100 includes a video encoder 101 , a feature extractor 102 , a feature encoder 103 , and a controller 104 .

[0035] The video encoder 101 generates a video bitstream by encoding each video frame of the digitized video signal using a video frame type (I video frame type or non-I video frame type) supplied from the controller 104. For example, the video encoder 101 performs encoding based on the H.266 / VVC (Versatile Video Coding) standard. The video encoder 101 may also perform encoding based on other standards, such as the H.265 / HEVC (High Efficiency Video Coding) standard or the H.264 / AVC (Advanced Video Coding) standard.

[0036] The feature extractor 102 extracts a feature A of the video signal from the video frame at each time point. The feature encoder 103 encodes the feature A using a feature frame type (I feature frame type or non-I feature frame type) supplied from the controller 104 to generate a feature bitstream.

[0037] The controller 104 determines the type of video frame (video frame type) and the type of feature frame (feature frame type) to be coded by the video encoder 101 and the feature encoder 103, respectively. The controller 104 supplies the video frame type as a control signal to the video encoder 101. The controller 104 also outputs the feature frame type as a control signal to the feature encoder 103.

[0038] The controller 104 synchronizes the output timing of a control signal indicating an I video frame with the output timing of a control signal indicating an I feature frame so as to ensure simultaneous random access to the video bitstream and the feature bitstream. For example, the controller 104 synchronizes when a predetermined period has elapsed. Note that ensuring simultaneous random access to the video bitstream and the feature bitstream means, for example, ensuring that random access points in the feature bitstream and the video bitstream are temporally coincident.

[0039] Furthermore, synchronizing the random access points of the video frames and the random access points of the feature frames means aligning the random access points of both. Specifically, for example, in response to an instruction from the controller 104, for one video frame input at a certain time, the video encoder 101 performs encoding based on the I video frame type, and at the same time the feature encoder 103 performs encoding based on the I feature frame type.

[0040] The VCM decoder 200 includes a video decoder 201 and a feature decoder 202 .

[0041] The video decoder 201 decodes the video bitstream to generate a decoded video signal. The decoded video signal is used for human viewing or visual confirmation. The decoded video signal may also be used for task X, which is a machine process separate from task A (see FIG. 17).

[0042] The feature decoder 202 decodes the feature bitstream to generate decoded feature A. The decoded feature A is used for machine processing task A (see FIG. 17).

[0043] Next, a description will be given of the operation of the VCM encoder 100 in the first embodiment. Fig. 3 is a flowchart showing the operation of the VCM encoder 100. The process shown in Fig. 3 is executed for each frame.

[0044] The controller 104 in the VCM encoder 100 determines a video frame type and a feature frame type (step S100). As described above, for example, the controller 104 periodically determines the video frame type to be an I video frame type and the feature frame type to be an I feature frame type. When a period is used to determine the frame type, the controller 104 determines the video frame type to be a non-I video frame type and the feature frame type to be a non-I feature frame type when a predetermined period has not elapsed.

[0045] The controller 104 then outputs a control signal indicating the video frame type to the video encoder 101 and outputs a control signal indicating the feature frame type to the feature encoder 103 .

[0046] The video encoder 101 encodes video frames in accordance with a video frame type designated by a control signal from the controller 104 to generate a video bitstream (step S101).

[0047] The feature extractor 102 extracts features from the video frames (step S102). The feature encoder 103 encodes the features using a feature frame type specified by a control signal from the controller 104 to generate a feature bitstream (step S103).

[0048] The video encoder 101 sends the video bit stream to, for example, a transmission path (step S104), and the feature encoder 103 sends the feature bit stream to, for example, a transmission path (step S105).

[0049] Next, the operation of the VCM decoder 200 in the first embodiment will be described with reference to a flowchart shown in FIG.

[0050] The video decoder 201 in the VCM decoder 200 decodes the received video bitstream to generate a decoded video signal (step S201). The feature decoder 202 decodes the feature bitstream to generate decoded features (step S202).

[0051] The video decoder 201 outputs a decoded video signal (step S203), and the feature decoder 202 outputs a decoded feature (step S204).

[0052] 5 is an explanatory diagram showing an example of the relationship in the time direction between a video bitstream and a feature bitstream in the first embodiment. In FIG. 5, arrows indicate frames at random access points. Frames at random access points are frames that have no dependency relationship with other frames. The width of the rectangles indicating frames corresponds to the amount of data.

[0053] As illustrated in FIG. 5, in this embodiment, simultaneous random access to the video bitstream and the feature bitstream is guaranteed, so that the random access points in the video bitstream and the random access points in the feature bitstream are aligned.

[0054] As a result, a system that uses the video bitstream and feature bitstream output by the VCM decoder 200 can immediately begin machine recognition tasks based on the features of a given video frame and at the same time.

[0055] Embodiment 2. Fig. 6 is a block diagram showing a data processing system of a second embodiment. The data processing system shown in Fig. 6 includes a VCM encoder 110 and a VCM decoder 210. The VCM encoder 110 and the VCM decoder 210 are communicably connected via, for example, a transmission path. In this embodiment, too, an example is taken of a case in which the VCM encoder 110 generates a video bitstream and a feature bitstream related to feature A for task A of a certain machine processing.

[0056] The VCM encoder 110 includes a video encoder 101, a feature extractor 102, a side information generator 105, a feature encoder 106, and a controller 104. The functions of the video encoder 101, the feature extractor 102, and the controller 104 are the same as those in the first embodiment.

[0057] The side information generator 105 generates information correlated with the feature A extracted by the feature extractor 102 from the reconstructed video signal generated inside the video encoder 101. Hereinafter, the information correlated with the feature A extracted by the feature extractor 102 is referred to as side information A'.

[0058] The reconstructed video signal in video encoder 101 is a video signal (video frame) generated by a decoding function included in video encoder 101. Side information generator 105 generates side information A′, for example, by the same process as that executed by feature extractor 102.

[0059] The feature encoder 106 has the following function in addition to the functions of the feature encoder 103 in the first embodiment: When encoding an I feature frame type, the feature encoder 106 reduces the amount of data of the feature frame by encoding the feature frame based on the correlation between the feature A and the side information A'.

[0060] Let us take an example in which the side information generator 105 generates side information A' from the reconstructed video signal in the video encoder 101 using the same process as the feature extractor 102. In this case, the side information generator 105 extracts features from the reconstructed video signal as side information A'. The feature encoder 106 then predictively encodes feature A using the features (side information A') extracted by the side information generator 105, thereby reducing the data amount of the feature frame. This is because feature A of the input video signal and feature A of the reconstructed video signal are similar. Note that the higher the bit rate of the video bitstream, the higher the similarity between feature A of the video signal and the reconstructed video signal.

[0061] Furthermore, when encoding a non-I feature frame type, the feature encoder 106 may also encode the feature frame by utilizing the correlation between the feature A and the side information A′.

[0062] The VCM decoder 210 includes a video decoder 201, a side information generator 203, and a feature decoder 204. The video decoder 201 is the same as that in the first embodiment. That is, the video decoder 201 decodes a video bitstream to generate a decoded video signal. The decoded video signal is used for human viewing or visual confirmation. The decoded video signal may also be used for machine processing task X, which is separate from task A (see FIG. 17 ).

[0063] Side information generator 203 operates similarly to side information generator 105 in VCM encoder 110 to generate side information A' when a video frame of I video frame type is decoded, except that while side information generator 105 generates side information A' from the reconstructed video signal, side information generator 203 generates side information A' from the decoded video signal.

[0064] The feature decoder 204 has the following function in addition to the functions of the feature decoder 202 in the first embodiment: When decoding a feature frame of the I feature frame type, the feature decoder 204 decodes the feature frame based on the correlation between the feature A and the side information A′.

[0065] The feature decoder 204 can decode feature frames by performing the inverse process of the predictive coding performed by the feature encoder 106. For example, the feature decoder 204 predictively decodes features decoded from the feature bitstream using features extracted by the side information generator 105.

[0066] Next, a description will be given of the operation of the VCM encoder 110 in the second embodiment. Fig. 7 is a flowchart showing the operation of the VCM encoder 110. The process shown in Fig. 7 is executed for each frame.

[0067] The processing in steps S100 to S102 is the same as the processing in the first embodiment.

[0068] In this embodiment, the side information generator 105 generates side information A' from the reconstructed video signal generated inside the video encoder 101 (step S301).

[0069] The feature encoder 106 predictively encodes the feature A using the feature (side information A') extracted by the side information generator 105 to generate a feature bitstream (step S302).

[0070] As in the first embodiment, the video encoder 101 transmits the video bitstream to, for example, a transmission path (step S104). As in the first embodiment, the feature encoder 103 transmits the feature bitstream to, for example, a transmission path (step S105).

[0071] Next, the operation of the VCM decoder 210 in the second embodiment will be described with reference to a flowchart shown in FIG.

[0072] As in the first embodiment, video decoder 201 decodes the received video bitstream to generate a decoded video signal (step S201). Side information generator 203 generates side information A' from the decoded video signal when a video frame of I video frame type is decoded (step S401).

[0073] The feature decoder 204 decodes the feature bitstream to generate decoded features (step S402), as in the first embodiment. However, as described above, in this embodiment, when decoding a feature frame of the I feature frame type, the feature decoder 204 decodes the feature frame by utilizing the correlation between the feature A and the side information A'.

[0074] The video decoder 201 outputs a decoded video signal (step S203), and the feature decoder 204 outputs a decoded feature (step S204).

[0075] 9 is an explanatory diagram showing an example of the relationship in the time direction between a video bitstream and a feature bitstream in the first embodiment. In FIG. 9, arrows (except for the arrow related to "reducing data volume") indicate frames at random access points. Frames at random access points are frames that have no dependency relationship with other frames. The width of the rectangles indicating frames corresponds to the data volume.

[0076] As illustrated in FIG. 9, in this embodiment as well, simultaneous random access to the video bitstream and the feature bitstream is guaranteed, so that the random access points in the video bitstream and the random access points in the feature bitstream are synchronized.

[0077] Furthermore, in this embodiment, the feature frame is encoded using the correlation between the feature A and the side information A′, which reduces the amount of data in the feature frame at the random access point, thereby suppressing an increase in the total amount of data at the random access point.

[0078] [Variation 1] In the first and second embodiments, as illustrated in Figure 10, the video encoder 101 and the feature encoders 103 and 106 may add identification headers to the beginning of the video frame data and the beginning of the feature frame data to identify random access points.

[0079] The identification header of the video frame data includes information indicating whether it is an I video frame, a P video frame, or a B video frame. The identification header of the feature frame data includes information indicating whether it is an I feature frame or a non-I feature frame.

[0080] In the second embodiment, the identification header of the feature frame data further includes information indicating whether or not the feature frame data has been coded using side information.

[0081] By adding the above-mentioned identification header, the decoding side can confirm random accessibility without decrypting the contents of the video frame data and feature frame data, thereby further improving interoperability between the encoding side and the decoding side.

[0082] [Variation 2] In the second embodiment, the feature extracted from the video frame of the reconstructed video signal by the same process as that executed by the feature extractor 102 is defined as side information A'. However, the encoded data itself can also be used as side information. Hereinafter, the feature extracted by the same process as that executed by the feature extractor 102 will be referred to as feature A'.

[0083] For example, an error correction code can be applied as described in Non-Patent Document 2.

[0084] Let Y(A, t) be the coded data of feature A extracted by feature extractor 102 from a video frame at time t of the input video signal, and let Y(A', t) be the coded data of feature A'.

[0085] The feature A' is coded as shown in Fig. 11. That is, the feature encoder 106 multiplies the coded data Y(A, t) to be sent by a check matrix to generate a syndrome, which is then converted into coded data.

[0086] As shown in Fig. 11, the feature encoder 106 determines the number of columns M of the parity check matrix based on the correlation between the encoded data Y(A,t) (N-bit data) to be sent and the side information Y(A',t). The feature encoder 106 multiplies the encoded data Y(A,t) by a parity check matrix with N rows and M columns to generate a syndrome (M-bit data). The feature encoder 106 then outputs the syndrome as feature frame data. Note that if M<N, the amount of data is reduced.

[0087] The feature A' is decoded as shown in Fig. 12. That is, the feature decoder 204 can obtain the decoded feature by using the error-corrected Y(A, t) as the decoded value based on the relationship between the side information Y(A', t), the parity check matrix, and the syndrome (M-bit data).

[0088] Note that the data volume of each feature frame data can be further reduced by sharing the values ​​of each element of the parity check matrix between the encoding side and the decoding side in advance. In this case, the feature encoder 106 only needs to include the ID of the parity check matrix used for the number of columns M in the identification header.

[0089] In the above-described embodiments and modifications, one feature amount is used as an example for the sake of simplicity, but the present invention can be applied to a plurality of feature amounts.

[0090] Furthermore, when applying the encoding and decoding of feature data using the error-correcting code of the second modification to a plurality of features, it is possible to calculate a syndrome for all of the feature frame data concatenated together, rather than calculating a syndrome for each feature frame data independently. In this case, when the feature frame data for each feature is short, it is possible to suppress the overhead caused by applying a check matrix with a small number of columns.

[0091] Fig. 13 is a block diagram showing a specific example of the configuration of a data processing system. The data processing system shown in Fig. 13 is a system in which a data encoding device 10 (corresponding to VCM encoder 100 in the first embodiment or VCM encoder 110 in the second embodiment) and a data decoding device 20 (corresponding to VCM decoder 200 in the first embodiment or VCM decoder 210 in the second embodiment) are connected by a transmission path (wireless transmission path or wired transmission path) 30.

[0092] In the data processing system, the data encoding device 10 can generate a video bitstream and a feature bitstream having the characteristics described in the above embodiments. Also, in the data processing system, the data decoding device 20 can decode the video bitstream and the feature bitstream having the characteristics described in the above embodiments.

[0093] Furthermore, each of the above embodiments can be configured by hardware, but can also be realized by a computer program.

[0094] The information processing system shown in Fig. 14 includes a processor 1001 such as a CPU (Central Processing Unit), a program memory 1002, a storage medium 1003 for storing video data, and a storage medium 1004 for storing a bitstream. The storage medium 1003 and the storage medium 1004 may be separate storage media or may be storage areas formed by the same storage medium. A magnetic storage medium such as a hard disk can be used as the storage medium.

[0095] In the information processing system, a program memory 1002 stores a program (a data encoding program or a data decoding program) for realizing the functions of each block shown in each of the above embodiments.

[0096] The processor 1001 then executes processing in accordance with the program stored in the program memory 1002, thereby realizing the functions of the data encoding device 10, VCM encoders 100, 110, data decoding device 20, and VCM decoders 200, 210 shown in each embodiment.

[0097] For example, the processor 1001 executes processing in accordance with a data encoding program (more specifically, a VCM encoding program) for implementing the functions of each block in the VCM encoder 100 shown in Fig. 2, thereby realizing the functions of the VCM encoder 100. Also, for example, the processor 1001 executes processing in accordance with a data decoding program (more specifically, a VCM decoding program) for implementing the functions of each block in the VCM decoder 200 shown in Fig. 2, thereby realizing the functions of the VCM decoder 200.

[0098] At least the program memory 1002 is a non-transitory computer-readable medium. However, the program may be stored in various types of transitory computer-readable medium. The program is supplied to the transitory computer-readable medium via, for example, a wired or wireless communication path, i.e., via an electrical signal, an optical signal, or an electromagnetic wave.

[0099] Fig. 15 is a block diagram showing the main components of a data encoding device 10. The data encoding device 10 shown in Fig. 15 includes a video encoding unit (video encoding means) 11 (realized by a video encoder 101 in the embodiment) that encodes video to generate a video bitstream, a feature encoding unit (feature encoding means) 12 (realized by feature encoders 103 and 106 in the embodiment) that encodes features of the video to generate a feature bitstream, and a synchronization unit (synchronization means) 13 (realized by a controller 104 in the embodiment) that synchronizes random access points in the video bitstream and random access points in the feature bitstream.

[0100] The data encoding device 10 includes a first feature extraction unit (first feature extraction means: in this embodiment, implemented by the side information generator 105) that extracts features of the input video, and a second feature extraction unit (second feature extraction means: in this embodiment, implemented by the feature encoder 106) that extracts features of the video generated by the decoding process included in the encoding process by the video encoding unit 11, and the feature encoding unit 12 may be configured to encode the features based on the correlation between the features extracted from the input video and the features extracted from the video generated by the decoding process.

[0101] Fig. 16 is a block diagram showing the main components of a data decoding device 20. The data decoding device 20 shown in Fig. 16 includes a video decoding unit (video decoding means) 21 that receives and decodes a video bitstream generated by encoding video and having random access points set therein, and a feature decoding unit (feature decoding means) 22 that receives and decodes a feature bitstream generated by encoding video features and having random access points set therein that are temporally aligned with the random access points.

[0102] The data decoding device 20 includes a feature extraction unit (feature extraction means: in the embodiment, this is realized by the side information generator 203) that extracts features of the decoded video obtained by decoding the video bitstream, and the feature decoding unit 22 may be configured to perform decoding based on the correlation between the features obtained by decoding the feature bitstream and the features extracted from the decoded video.

[0103] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0104] (Supplementary Note 1) A data encoding device comprising: a video encoding means for encoding video to generate a video bitstream; a feature encoding means for encoding features of the video to generate a feature bitstream; and a synchronization means for synchronizing random access points in the video bitstream with random access points in the feature bitstream.

[0105] (Supplementary Note 2) The data encoding device of Supplementary Note 1, comprising: a first feature extraction means for extracting features of the input video; and a second feature extraction means for extracting features of video generated by a decoding process included in the encoding process by the video encoding means, wherein the feature encoding means encodes the features based on a correlation between the features extracted from the input video and the features extracted from the video generated by the decoding process.

[0106] (Supplementary Note 3) The data encoding device of Supplementary Note 2, wherein the feature encoding means predictively encodes the feature extracted from the input video using the feature extracted from the video generated by the decoding process.

[0107] (Supplementary Note 4) A data decoding device comprising: a video decoding means for receiving and decoding a video bitstream in which video is encoded and a random access point is set; and a feature decoding means for receiving and decoding a feature bitstream in which features of the video are encoded and a random access point that is temporally aligned with the random access point.

[0108] (Supplementary Note 5) The data decoding device of Supplementary Note 4 further comprises a feature extraction means for extracting features of decoded video obtained by decoding the video bitstream, wherein the feature decoding means performs decoding based on a correlation between the features obtained by decoding the feature bitstream and the features extracted from the decoded video.

[0109] (Supplementary Note 6) The data decoding device of Supplementary Note 5, wherein the feature decoding means predictively decodes the feature decoded from the feature bit stream using the feature extracted by the feature extraction means.

[0110] (Supplementary Note 7) A data encoding method comprising: encoding a video to generate a video bitstream; encoding features of the video to generate a feature bitstream; and synchronizing random access points in the video bitstream and random access points in the feature bitstream.

[0111] (Supplementary Note 8) The data encoding method of Supplementary Note 7, comprising: extracting features of the input video; extracting features of video generated by a decoding process included in a video encoding process; and encoding the features based on a correlation between the features extracted from the input video and the features extracted from the video generated by the decoding process.

[0112] (Supplementary Note 9) A data decoding method comprising: receiving and decoding a video bitstream in which a video is generated by encoding and a random access point is set; and receiving and decoding a feature bitstream in which a feature of the video is generated by encoding and a random access point that is aligned in time with the random access point.

[0113] (Supplementary Note 10) The data decoding method of Supplementary Note 9, further comprising: extracting features of decoded video obtained by decoding the video bitstream; and performing decoding based on a correlation between the features obtained by decoding the feature bitstream and the features extracted from the decoded video.

[0114] (Supplementary Note 11) A data processing system comprising: a data encoding device according to any one of Supplementary Note 1 to Supplementary Note 3; and a data decoding device according to Supplementary Note 4 to Supplementary Note 6.

[0115] (Supplementary Note 12) A data encoding program for causing a computer to execute the following processes: encoding video to generate a video bitstream; encoding features of the video to generate a feature bitstream; and synchronizing random access points in the video bitstream and random access points in the feature bitstream.

[0116] (Supplementary Note 13) A data encoding program for causing a computer to execute the following processes: a process of receiving and decoding a video bitstream generated by encoding video and having a random access point set therein; and a process of receiving and decoding a feature bitstream generated by encoding features of the video and having a random access point that is aligned in time with the random access point set therein.

[0117] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0118] REFERENCE SIGNS LIST 10 Data encoding device 11 Video encoding unit 12 Feature encoding unit 13 Synchronization unit 20 Data decoding device 21 Video decoding unit 22 Feature decoding unit 30 Transmission path 100, 110 VCM encoder 101 Video encoder 102 Feature extractor 103, 106 Feature encoder 104 Controller 105 Side information generator 200, 210 VCM decoder 201 Video decoder 202, 204 Feature decoder 203 Side information generator 1001 Processor 1002 Program memory 1003, 1004 Storage medium

Claims

1. a video encoding means for encoding a video to generate a video bitstream; a feature encoding means for encoding the feature of the video to generate a feature bit stream; a synchronizing means for synchronizing random access points in the video bitstream with random access points in the feature bitstream; A data encoding device comprising:

2. a first feature extraction means for extracting a feature of the input image; a second feature extraction means for extracting a feature of the video generated by a decoding process included in the encoding process by the video encoding means, The feature encoding means encodes the feature based on a correlation between the feature extracted from the input video and the feature extracted from the video generated by the decoding process.

2. The data encoding device according to claim 1.

3. a video decoding means for receiving and decoding a video bitstream in which a video is encoded and a random access point is set; a feature decoding means for receiving and decoding a feature bit stream in which the feature of the video is generated by encoding and a random access point that is aligned in time with the random access point is set; A data decoding device comprising:

4. a feature extraction means for extracting a feature of a decoded video obtained by decoding the video bitstream, The feature decoding means performs decoding based on a correlation between a feature obtained by decoding the feature bit stream and a feature extracted from the decoded video.

4. The data decoding device according to claim 3.

5. encoding the video to generate a video bitstream; encoding the features of the video to generate a feature bitstream; Synchronizing random access points in the video bitstream with random access points in the feature bitstream. Data encoding method.

6. Extracting features of the input video; Extracting a feature quantity of a video generated by a decoding process included in a video encoding process; The feature is encoded based on a correlation between the feature extracted from the input image and the feature extracted from the image generated by the decoding process.

6. The data encoding method according to claim 5.

7. receiving and decoding a video bitstream in which a video is encoded and generated and in which a random access point is set; The feature amount of the video is encoded and generated, and a feature amount bit stream in which a random access point that is aligned in time with the random access point is set is received and decoded. Data decryption method.

8. extracting features of a decoded video obtained by decoding the video bitstream; Decoding is performed based on a correlation between the feature obtained by decoding the feature bitstream and the feature extracted from the decoded video.

8. The data decoding method according to claim 7.

9. A data encoding device according to claim 1 or 2, The data decoding device according to claim 3 or 4, A data processing system comprising:

10. On the computer, encoding the video to generate a video bitstream; A process of encoding the features of the video to generate a feature bitstream; synchronizing random access points in the video bitstream with random access points in the feature bitstream; A data encoding program for executing the above.