Method and system for encoding and decoding video data in conjunction with performing a search

Abstract

The present invention relates to a method and system for encoding and decoding video data in conjunction with performing a search, which facilitates performing a search in regions of interest, ROI, in encoded video data. In the encoded video data, there are independently decodable ROIs. These ROIs and the encoded video frames in which they are located are identified in metadata that is searched in response to a search query. The encoded video data further embeds information that associates the ROIs with groups of coding units, CUs, that spatially overlap the ROIs. In conjunction with independently decoding the ROIs found in the search, the embedded information is used to identify the group of CUs to decode.

Description

Technical Field

[0001] The present invention relates to the field of searching in encoded video data. In particular, the present invention relates to encoding of video data to facilitate searching in the encoded video data, and decoding the encoded video data in conjunction with performing searching in the encoded video data. Background Art

[0002] Video cameras are commonly used for surveillance purposes. They generate video data that is stored in an encoded format for later retrieval. For example, when an incident is reported, the stored video data can be searched to locate specific content, such as a specific person or someone with a specific color in their clothing. Manually searching for specific content in video data is tedious and, for larger camera installations, even impractical. Therefore, automated search processes for video data have been proposed.

[0003] These automated search processes are driven by metadata associated with the encoded video data. The metadata can indicate in which video frames a region of interest (ROI) exists, and possibly also the location of the ROI within those frames. For example, the metadata can indicate in which frames objects exist, along with the bounding boxes of those objects within those frames. By searching within the metadata, relevant video frames for further analysis can be identified without having to decode the video data. Once relevant video frames are found, the automated search process can then decode the relevant frames and analyze the decoded video data within the ROI to see if it matches the specified search criteria.

[0004] The disadvantage of these automatic search processes is that they are slow and processing-intensive, and therefore unsuitable for searching in larger video archives (such as those collected by camera apparatuses with more than a few cameras). One reason for this is the inefficient decoding of relevant video frames. In light of this, it should be noted that when encoding video data, most video frames are typically inter-coded, referencing other video frames. Such frames are also known as P-frames or B-frames. Video frames are only occasionally intra-coded, i.e., encoded independently of other video frames. Intra-coded frames are also known as I-frames. This means that to decode a specific frame in the video data, it is necessary to start decoding from the most recent preceding intra-coded frame and then decode the inter-coded frames that led to the specific frame. Consequently, to decode the video frame relevant to the search, multiple video frames unrelated to the search must be decoded. Furthermore, it should be noted that video frames are conventionally encoded and decoded macroblock by macroblock in the raster scan direction. Consequently, even if only a small portion of a video frame is of interest, the entire video frame will be decoded. For example, in surveillance scenarios, it is not uncommon for an object of interest in a video frame to cover only 100×50 pixels or even less, which is a negligible portion of a typical video frame of a video.Therefore, there is room for improvement. Summary of the Invention

[0005] In view of the above, it is therefore an object of the present invention to alleviate the above problems and to provide a video encoding and video decoding method that facilitates more efficient searching in encoded video data.

[0006] The invention is defined by the appended independent claims, while embodiments of the invention are defined by the dependent claims.

[0007] According to a first aspect of the present invention, there is provided a method and system for decoding video data in conjunction with performing a search in the video data.

[0008] Decoding methods include:

[0009] receiving coded video data comprising a coded video frame in which independently decodable regions of interest are present, wherein information associating each independently decodable region of interest with a group of coding units that spatially overlap with the independently decodable region of interest is embedded in the coded video data,

[0010] Receive metadata that identifies independently decodable regions of interest and the encoded video frames they are located in,

[0011] receiving a search query related to searching among regions of interest in encoded video data,

[0012] In response to receiving the search query, searching the metadata for one or more independently decodable regions of interest and encoded video frames in which the one or more regions of interest exist, and

[0013] For each independently decodable region present in the coded video frame and found during the search: identifying a group of coding units associated with the independently decodable region of interest by using information embedded in the coded video data, and decoding the group of coding units of the coded video frame independently of video data in other video frames and independently of video data in coding units outside the group of coding units in the coded video frame to produce decoded video data corresponding to the independently decodable region of interest.

[0014] According to this decoding method, at least some of the ROIs in the received encoded video data are independently decodable. This means that the video data within the ROI is encoded in such a way that it can be decoded without first decoding any previous video frame or any other region in the same video frame. In practice, to achieve independent decoding of video data in such an ROI, the decoder needs to be informed of which coding units in which video frame correspond to the ROI. Once the decoder knows this, it can selectively decode these coding units without first decoding any other coding units in the same video frame or another video frame.

[0015] A two-stage process is proposed to inform the decoder about which coding units in which video frames to decode in response to a search query. The first stage relies on metadata that identifies independently decodable ROIs and the coded video frames in which they are located. When a search query is received, the metadata can be searched to find one or more independently decodable ROIs and the coded video frames in which they are located. The second stage relies on information embedded in the coded video data that associates each independently decodable ROI identified by the metadata with a group of coding units that spatially overlaps with the ROI. Thus, the embedded information allows identification of the coding unit groups associated with the independently decodable ROIs in the coded video frames found during the search. Finally, the identified coding unit groups in those coded video frames can be selectively decoded.

[0016] Accordingly, metadata and information embedded in the video data allow selective decoding of independently decodable ROIs in response to a search query without having to decode any video data other than the selected ROI. In this way, searching in video data becomes more efficient.

[0017] The region of interest in a video frame refers to a pixel region in the video frame that a user may be particularly interested in. The region of interest may correspond to a pixel region of an object detected in the video frame, for example.

[0018] Searching in video data can generally involve searching the video data for ROIs that match a specific search query. This search can be conducted for forensic purposes, i.e., it can be a forensic search, where the purpose of the search is to find evidence in the video material. For example, one might want to find all people wearing blue jackets, or one might want to find whether a specific person is depicted in the video data.

[0019] A coding unit refers to a pixel block in an image frame that forms a basic processing unit used during encoding. Depending on the encoding standard, a coding unit is also called a macroblock (H.264) or a coding tree unit (H.265).

[0020] Metadata used to identify independently decodable regions of interest and the encoded video frames in which they are located is associated with the encoded video data, but the metadata is typically provided as a separate data entity (such as a separate metadata stream). Therefore, it is possible to search within the metadata without having to parse the encoded video data. Metadata can also be embedded within the video data.

[0021] Information embedded in coded video data means that the information is included in the coded video frames. However, the information can be read from the coded video data without first decoding the video data itself. For example, the information can be embedded in a coded video frame header (such as a picture parameter set (PPS) header) or in a dedicated supplemental enhancement information (SEI) frame inserted into the coded video data.

[0022] An independently decodable ROI in an encoded video frame means that it can be decoded using or referencing only the encoded video data within the coding units of the encoded video frame that spatially overlaps with the ROI. In other words, the video data within the ROI is encoded without referencing any video data outside the ROI in the same frame or video data in another frame. For example, video data within a coding unit group that spatially overlaps with the ROI can be intra-coded. Intra-coding can be spatially restricted to the independently decodable region of interest, that is, intra-coding only references the video data within the ROI.

[0023] A search query can involve searching for ROIs within the encoded video data that have specific attributes or a specific combination of attributes. An attribute of an ROI refers to a property or feature of the video data within the ROI. For example, if the ROI corresponds to an object, it can be an attribute or feature of the object, such as the object's class, the object's color, the object's identity, or an item the object is wearing or carrying (such as a hat or bag). In some cases, the video data within the ROI may have been analyzed and attributed before encoding. In this case, the metadata may further indicate the attributes of the independently decodable region of interest, and the search can include searching within the metadata to find one or more independently decodable ROIs with attributes that match the specific attributes of the search query. While this requires more processing on the encoding side, it saves processing power on the decoding side because only ROIs with the attributes of interest are found in the search and need to be decoded.

[0024] In other cases, when metadata doesn't tag ROIs with attributes, the method can analyze the decoded video data corresponding to independently decodable ROIs to determine whether they match the specific attributes of the search query. This requires more processing power on the decoding side, as video data for ROIs that ultimately don't match the search query also needs to be decoded. However, the processing load on the encoding side is simultaneously reduced.

[0025] The metadata may include an identifier for each independently decodable ROI in a coded video frame, and information embedded in the coded video data may associate the identifier of each independently decodable ROI with a coding unit that spatially overlaps with the independently decodable ROI. The use of identifiers is an efficient way to link ROIs to coding units in a coded video frame.

[0026] According to a second aspect of the present invention, a method and system for encoding video data are provided.

[0027] Encoding methods include:

[0028] receiving video data comprising a video frame in which a region of interest exists,

[0029] receiving first metadata, the first metadata identifying the region of interest and the video frame in which it is located,

[0030] selecting one or more video frames and one or more regions of interest in each of the video frames using the first metadata, and

[0031] Encoding the video data to generate encoded video data, wherein encoding each selected video frame and each selected region of interest therein comprises:

[0032] Associating a region of interest with a set of coding units that spatially overlaps the region of interest,

[0033] encoding video data of the video frames in the group of coding units independently of video data in other video frames and independently of video data in coding units outside the group of coding units in the video frames that spatially overlap with the region of interest,

[0034] embedding in the coded video data information associating the region of interest with a group of coding units that spatially overlap with the region of interest, and

[0035] Second metadata is generated that identifies a region of interest in the video frame as independently decodable.

[0036] Thus, the encoding method encodes at least some of the ROIs in a video frame independently of video data in other video frames and independently of video data in decoding units outside the ROI. Consequently, when encoding these ROIs, any references to other video data are confined to the ROI. Furthermore, the encoding method generates metadata identifying the independently decodable ROIs and the video frames in which they reside, and embeds information in the encoded video data associating each independently decodable ROI with a set of coding units that spatially overlap with the region of interest. Thus, as described above, the encoding method enables selective decoding of independently decodable ROIs at the decoder side in response to a search query without having to decode any video data other than the selected ROI. In this way, searching within the video data becomes more efficient.

[0037] In some embodiments, the selected one or more video frames can correspond to all video frames in which a ROI exists. For example, all video frames in which a ROI exists can be selected as well as all ROIs therein. In other embodiments, the selected one or more video frames can comprise a strict subset of the video frames in which a ROI exists. Accordingly, fewer ROIs than all ROIs can be encoded in an independent manner. Since independent encoding is typically more expensive in terms of bit rate, bit rate can be saved by not independently encoding all ROIs. To this end, it should be noted that ROIs that are not independently encoded are still encoded and can still be decoded at the decoder side, albeit at the cost of decoding the entire frame in which the ROI resides as well as any previous video frames starting from the most recent intra-coded frame.

[0038] Various methods can be used to select which ROIs to independently encode. According to a first method, the first metadata further identifies a plurality of regions of interest in subsequent video frames as belonging to a common track, wherein the selected one or more video frames comprise fewer video frames than all of the subsequent video frames. Thus, when ROIs move between subsequent video frames, such as when video data captures movement of an object, ROIs can be selected to be independently encoded in one or several but not all subsequent video frames. This saves bit rate while enabling efficient search of ROIs belonging to a common track. This is because it is often sufficient for search purposes to decode ROIs in one or several frames of a track rather than decoding the entire track.

[0039] According to a second method, the selected one or more video frames as well as the selected one or more regions of interest in each video frame are further identified based on proximity of image properties within the regions of interest to target image properties. The image properties can comprise one or more of lighting conditions, occlusion ratio of the depicted object, and pose of the depicted object. Thus, when selecting what ROIs to independently encode, those ROIs can be selected that have favorable image properties. The second method can be combined with the first method. For example, when selecting which ROIs in a common track to independently encode, those ROIs can be selected that have the most favorable image properties. In one example, the one or those ROIs in a common track can be selected in which the depicted object faces the camera, the lighting conditions are best, and / or the depicted object is least occluded.

[0040] While having independently coded ROIs in the encoded video data increases the bit rate of the encoded video, for typical surveillance scenarios the additional bit rate cost is very low as the area covered by the ROIs is small relative to the area of the video frames. Furthermore, in most surveillance scenarios the time intervals in which no ROIs are present in the frames are longer than the time intervals in which ROIs are present in the frames. However, to control the bit rate per frame or over time, an upper limit can be introduced on the number of pixels that can be independently coded per time unit or per frame. In particular, the number of pixels in the video data covered by the selected one or more ROIs in the selected one or more video frames can be less than a predetermined proportion of the total amount of pixels in the video data per time unit or per selected video frame. Accordingly, this allows to set a limit on how much additional bit rate can be tolerated, thereby optimizing the search time of the search application.

[0041] The encoding method can further comprise analyzing the video data within the ROIs to associate it with attributes, and including the attributes of the regions of interest in the second metadata. As mentioned above, this saves processing power at the decoding side as it allows to identify ROIs with attributes matching a search query before decoding takes place.

[0042] According to a third aspect of the application, there is provided a non-transitory computer readable medium having stored thereon computer code instructions which, when executed by an apparatus having processing capability, cause the apparatus to perform the decoding method according to the first aspect or the encoding method according to the second aspect.

[0043] The third aspect can generally have the same features and advantages as the first and second aspects. It should also be noted that the application relates to all possible combinations of features unless explicitly stated otherwise. BRIEF DESCRIPTION OF DRAWINGS

[0044] The above and additional objects, features and advantages of the application will be better understood through the following illustrative and non-limiting detailed description of embodiments of the application, with reference to the appended drawings, in which the same reference numerals will be used for similar elements, in which:

[0045] Figure 1 An encoding system and a decoding system according to embodiments are schematically illustrated;

[0046] Figure 2 An encoding system according to embodiments is schematically illustrated;

[0047] Figure 3 A decoding system according to embodiments is schematically illustrated;

[0048] Figure 4 Video data and associated first metadata are schematically illustrated;

[0049] Figure 5 schematically illustrating encoded video data and associated second metadata;

[0050] Figure 6 schematically illustrates the output of a video decoding system;

[0051] Figure 7 is a flowchart of an encoding method according to an embodiment; and

[0052] Figure 8 is a flowchart of a decoding method according to an embodiment. DETAILED DESCRIPTION

[0053] The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the invention are shown.The systems and apparatus disclosed herein will be described during operation.

[0054] Figure 1 A video encoding system 100 and a corresponding video decoding system 200 are shown. The video encoding system 100 and the video decoding system 200 are specifically designed to enable efficient searching in video data. The video encoding system 100 may be included in a camera. The decoding system 200 may be included in a video management system.

[0055] Figure 2 and Figure 3 The encoding system 100 and the decoding system 200 are shown in greater detail. Generally, the encoding system 100 includes circuitry configured to implement a reception function 101, an ROI selection function 102, and an encoding function 103. Optionally, the circuitry of the encoding system 100 may also be configured to implement an analysis function 106. Similarly, the decoding system 200 includes circuitry configured to implement a reception function 201, a search function 202, and a decoding function 203. Optionally, the circuitry of the decoding system 200 may also be configured to implement an analysis function 206.

[0056] In a hardware implementation, each of the functions 101, 102, 103, 106, 201, 202, 203, 206 may correspond to a circuit dedicated and specifically designed to perform that function. The circuit may be in the form of one or more integrated circuits (such as one or more application specific integrated circuits or one or more field programmable gate arrays). For example, the encoding function 103 may include circuitry that, when used, encodes video data to generate encoded video data.

[0057] In a software implementation, alternatively, the circuitry may be in the form of a processor 105, 205, such as a microprocessor, associated with computer code instructions stored on a (non-transitory) computer-readable medium 104, 204, such as a non-volatile memory, causing the encoding system 100 and the decoding system 200 to perform any of the methods disclosed herein. Examples of non-volatile memory include read-only memory, flash memory, ferroelectric RAM, magnetic computer storage devices, optical disks, and the like. In the software case, the functions 101, 102, 103, 106, 201, 202, 203, 206 may each correspond to a portion of computer code instructions stored on a computer-readable medium that, when executed by the processor 104 or 204, causes the encoding system 100 or the decoding system 200 to perform these functions.

[0058] It shall be understood that it is also possible to have a combination of hardware and software implementation, which means that the functionality of some of the components 101 , 102, 103, 106, 201 , 202, 203, 206 are implemented in hardware and other components are implemented in software.

[0059] Now refer to Figure 1 、 Figure 2 、 Figure 4 、 Figure 5 as well as Figure 7 The flowchart of FIG. 1 illustrates the operation of the encoding system 100 when performing the method S100 for encoding video data.

[0060] In steps S102 and S104, the receiving function 101 receives the video data 110 and the first metadata 120 respectively. Figure 4 As further shown in FIG. 1 , video data 110 includes video frames 110 - 1 , 110 - 2 , 110 - 3 , and 110 - 6 , each of which includes an ROI 112 . In the example shown, two ROIs are present in each of video frames 110 - 1 and 110 - 2 , and one ROI is present in each of video frames 110 - 3 and 110 - 6 . As is common in long-duration surveillance videos, video frames 110 - 4 and 110 - 5 also exist without ROIs. ROIs can correspond to objects detected in the video frames. In this case, the ROIs correspond to a person and a car.

[0061] The first metadata 120 includes information identifying the ROI 112 and the video frame in which it is located. The first metadata 120 is typically provided as a separate data entity from the video data, but it can also be embedded in the video data 110, for example, using a Supplemental Enhancement Information (SEI) frame. The first metadata 120 is generated upstream of the video coding system 100, for example, in conjunction with detecting objects in the video data 110. To this end, any object detection algorithm known in the art can be used. The first metadata 120 may include data units 120-1, 120-2, 120-3, and 120-6, each of which identifies a video frame in the video data 110 and includes information about the ROI in the video frame. More specifically, the first metadata 120 may include identifiers for the video frames 110-1, 110-2, 110-3, and 110-6 in which the ROI is located. The first metadata 120 may further include an identifier for each ROI in the video frames 110-1, 110-2, 110-3, and 110-6. These identifiers may be given by frame number and the number of each ROI in the frame.

[0062] In addition to the identifier, the first metadata 120 may also specify the spatial coordinates of the ROI within the video frame for each ROI. The spatial coordinates define the spatial coverage of the ROI within the video frame, i.e., the pixels within the video frame that are included in the ROI. For example, the spatial coordinates may specify the spatial coordinates of a bounding box that defines the outline of the ROI. The spatial coordinates may be given in terms of pixel coordinates within the video frame.

[0063] Optionally, the first metadata 120 may further include an object identifier associated with the ROI, wherein ROIs describing the same physical object are associated with the same object identifier. Thus, the object identifier may be used to identify ROIs describing the same object. As a specific example, in the case where tracking has been performed upstream of the video encoding system 100, the first metadata 120 may indicate whether the ROI in the video frame belongs to a track that spans multiple subsequent video frames, and if so, the first metadata 120 may include an identifier of the track associated with the ROI. Figure 4 In the example shown, the ROIs corresponding to the people in the video frames 110 - 1 , 110 - 2 , and 110 - 3 form a track, and the ROIs corresponding to the cars in the video frames 110 - 1 and 110 - 2 form another track.

[0064] In some embodiments, the video data 110 and first metadata 120 are input to an optional analysis function 106, which analyzes the video data within the ROI in step S105 to associate the ROI with attributes. Such analysis, as is known in the art, may include object classification, object recognition, color recognition, and the like. The information in the first metadata 120 may be updated to further specify the attributes of each ROI. Examples of attributes include object category (such as person or car) and color information. As will be explained, these attributes can be used for search purposes in the decoding system 200. In alternative embodiments, the analysis function 106 may alternatively be located upstream of the encoding system 100.

[0065] The first metadata 120 may also include image properties of the ROI, such as lighting conditions within the ROI, occlusion ratios of the depicted objects, and poses of the depicted objects. The image properties of the ROI can generally be considered to reflect the quality of the video data within the ROI for specific purposes (such as object recognition). The image properties can be determined by the analysis function 106 or may have been determined upstream in the encoding system 100.

[0066] In step S106, the ROI selection function 102 uses the information in the first metadata 120 to select one or more of the video frames and one or more ROIs in each video frame. The ROI selection function 102 presents an indication 122 of the selected ROI and the video frames in which the ROI is present to the encoding function 103. In some embodiments, the ROI selection function 102 may select all video frames in which the ROI is present and all ROIs in each video frame. In other embodiments, the ROI selection function 102 may alternatively identify or select a strict subset of ROIs, for example by selecting a strict subset of video frames in which the ROI is present or by selecting a strict subset of ROIs within video frames. That is, the ROI selection function 102 may select fewer than all video frames in which the ROI is present, and / or fewer than all ROIs. As described above, the reason for not selecting all frames and / or all ROIs is to save bit rate during subsequent encoding. Figure 4 In the example of FIG, ROI selection function 102 has selected ROI 112-1, as shown by the thicker borders around those ROIs, but has not selected ROI 112-2. The selected ROIs 112-1 include ROI 2 (car) in video frame 110-1, ROI 1 in video frame 110-2, and ROI 1 (sports car) in video frame 110-6.

[0067] In the case where multiple ROIs in different video frames describe the same object indicated by the object identifier in the first metadata 120, the ROI selection function 102 may select one or several, but not all, ROIs describing the object. As a special case, when multiple ROIs form a track spanning subsequent video frames as indicated by a track identifier in the first metadata 120, the ROI selection function 102 may select one or several, but not all, ROIs belonging to the track. Accordingly, among the ROIs having the same track identifier, the ROI selection function 120 may select one or several ROIs in the corresponding one or several video frames that will subsequently be the representation of the track. Figure 4 In the example of FIG, the ROIs corresponding to the people in the video frames 110-1, 110-2, and 110-3 form a track, and the ROI selection function 102 selects the ROI 112-1 in the second video frame 110-2 as a representation of the track. Similarly, the ROIs corresponding to the cars in the video frames 110-1 and 110-2 form a track, and the ROI selection function 102 selects the ROI 112-1 in the first video frame 110-1 as a representation of the track.

[0068] In order to identify which one or more ROIs to select among multiple ROIs describing the same object (such as among ROIs belonging to a common trajectory), the ROI selection function 102 may further consider image properties of the ROIs specified by the information in the first metadata 120. The ROI selection function 102 may compare the image properties of the ROIs with the target image properties and select one or more ROIs having image properties that are closest to the target image properties. The target image properties may reflect image properties required for a specific purpose, such as an object recognition purpose. According to some examples, one or more ROIs may be selected that have lighting conditions that are closest to the target lighting conditions, one or more ROIs may be selected whose pose of the described object best matches the target pose, or one or more ROIs may be selected that have the smallest occlusion ratio. Figure 4 In the example, for example, the person in the second video frame 110-2 is facing the camera and is therefore selected, while the people in video frames 110-1 and 110-3 are facing away from the camera and are therefore not selected. In this way, the ROI selection function 102 can select those ROIs that have the best properties of the video data for a given purpose. These examples can be combined so that several image properties, possibly with different priorities, are considered in the selection. Image properties can also be considered when deciding how many ROIs to select from the multiple ROIs. In many cases, it may be sufficient to select the ROI that best matches the target image property. However, in other approaches, all ROIs with image properties that are closer to the target image property than a predefined threshold can be selected.

[0069] The ROI selection function 102 may be further constrained to limit the proportion of the total number of pixels allowed to be covered by the selected ROI per video frame and / or per time unit (such as per group of pictures (GOP)). This proportion may be predefined and may be considered a tuning parameter of the encoding system 100. To meet this constraint, the ROI selection function 102 may compare the number of pixels in the video data included in the selected ROI, as specified by the information in the first metadata 120, with the total number of pixels per frame and / or per time unit. If the proportion exceeds the limit, the ROI selection function 102 may need to reduce the number of selected ROIs, and if the proportion is not exceeded, there may be room to select additional ROIs. The ROI selection function 102 may repeat this process until the limit is not exceeded, or until no further ROIs can be added without exceeding the limit. Another option for reducing the bitrate is to apply higher compression to the video data outside the ROI during encoding than to the video data inside the ROI.

[0070] Instead of setting a ratio based on the total amount of pixels allowed to be covered by the selected ROI per frame or per time unit, an upper limit on the bit rate can be set. The upper limit can be measured or estimated based on the bit rate in a given scene with a given resolution, frame rate, and video configuration. If the bit rate exceeds the upper limit, the ROI selection component 102 reduces the number of selected ROIs, and if the bit rate is below the upper limit, the ROI selection component 102 can select additional ROIs as long as the upper limit is not exceeded.

[0071] The encoding function 103 receives the video data 110, the first metadata 120 and the indication 122 of the ROI and the relevant video frame selected by the ROI selection function 102. The encoding function 103 encodes the video data 110 to generate encoded video data 130 and second metadata 140 in step S108.

[0072] The encoding function 103 may generally encode the video data in the video frames 110-1 to 110-6 into the following Figure 5Corresponding encoded video frames 130-1 to 130-6 are shown. As is known in the art of video coding, the video data in each video frame 110 can be divided into coding units (CUs), each of which comprises a block of pixels in the video frame, such as an 8×8 pixel, 16×16 pixel, or 32×32 pixel block. Each CU is then either intra-frame coded (intra-coded for short), meaning it is predictively coded using spatial predictions from already coded video data in the same video frame; or inter-frame coded (inter-coded for short), meaning it is predictively coded using temporal predictions from already coded video data in one or more other video frames. In some frames, referred to as intra frames, all CUs are intra-coded. In other frames, referred to as inter frames, some CUs are intra-coded while others are inter-coded. Selecting intra- and inter-coding for CUs in inter frames based on bitrate and distortion considerations is a standard implementation. The encoding function 103 also employs this standard implementation when encoding most CUs in the video data 120. However, the encoding function 103 does not apply this standard implementation when encoding a CU that spatially overlaps the ROI selected by the ROI selection function 102. Instead, the encoding function 103 always intra-encodes the CU that spatially overlaps the selected ROI.

[0073] In more detail, when encoding a video frame including one or more selected ROIs, the encoding function 103 performs sub-steps S108a, S108b, S108c and S108d for each selected ROI within the video frame.

[0074] In sub-step S108a, the encoding function 103 associates the selected ROI in the video frame with a CU group that spatially overlaps with the ROI. To find the CU group that spatially overlaps with the ROI, the encoding function 103 may compare the spatial coverage of each CU in the video frame with the spatial coordinates in the first metadata 110 that define the spatial coverage of the ROI in the video frame. Those CUs that have at least one pixel located in the ROI are included in the CU group.

[0075] In sub-step S108b, the encoding function 130 encodes the video data in the CU group that spatially overlaps the ROI. The CUs in the CU group are encoded independently of the video data in other video frames, that is, they are intra-coded. Furthermore, they are encoded independently of the video data in CUs outside the CU group in the same video frame. In other words, the encoding function 130 constrains the scope of spatial prediction used in intra-coding to within the CU group. Therefore, the CUs included in the CU group refer only to the video data in the CUs within that group. By encoding the CU group in this manner, the video data within the ROI can be independently decoded by the decoding system 200.

[0076] In Figure 5 the example, the stripe pattern indicates that the CU group corresponding to the selected ROI 112-1 is encoded in this independent manner. According to the standard implementation, the CUs within any unmarked ROI are encoded.

[0077] In sub-step S108c, the encoding function 103 embeds information 132 in the encoded video data 130, which associates the ROIs with the CU groups that spatially overlap with the ROIs. The information 132 can be included in the encoded video frame header where the ROI is present, or can be included in a dedicated supplemental enhancement information (SEI) frame that is inserted in the sequence of encoded video frames 130-1 to 130-6. For example, such SEI frame can be inserted before the encoded video frame where the ROI is present, or the SEI frame can be inserted once per GOP. In the latter case, the SEI frame can carry the information 132 about all independently encoded ROIs in the GOP, and the information 132 should not only specify the CU groups associated with the ROIs, but also indicate in which frame the ROI is present. In Figure 5 In the example shown, the information 132 includes, for each marked ROI 112-1, a list of CUs that spatially overlap with the marked ROI.

[0078] In step S108d, the encoding function 103 generates second metadata 140 that identifies the ROIs and the encoded video frames where the ROIs are present. The second metadata 140 can include data units 140-1, 140-2, 140-3, 140-6 that each identify an encoded video frame in the encoded video data 130 and include information about the independently decodable ROIs in that encoded video frame. As Figure 5 shown, the second metadata thus identifies ROI 2 in encoded video frame 130-1, ROI 1 in encoded video frame 130-2, and ROI 1 in encoded video frame 130-6 as independently decodable. The second metadata 140 can further identify the unmarked ROIs 112-2 that are not independently decodable and the encoded frames where these ROIs 112-2 are present. Similar to the first metadata 120, the second metadata 140 can include information about the spatial coordinates of the ROIs, the properties of the ROIs, object identifiers such as track identifiers, and / or image properties of the ROIs. In some embodiments, the encoding function 103 can generate the second metadata 140 by updating the first metadata 120, where the update involves adding the identification of which ROIs are independently decodable.

[0079] The encoded video data 130 and the second metadata 140 are input to the decoding system 200. Reference will now be made to Figure 1 , Figure 3 、 Figure 5 、 Figure 6 as well as Figure 8 The flowchart of 200 illustrates the operation of the decoding system 200 when performing the method S200 for decoding video data in conjunction with performing a search in the video data.

[0080] In step S202, the receiving function 201 receives the encoded video data 130 including the encoded video frames 130-1 to 130-6 in which the independently decodable ROI 112-1 exists. As described above, the encoded video data 130 may further include the independently non-decodable ROI 112-2.

[0081] In step S204, the receiving function 201 receives the second metadata 140 identifying the independently decodable region of interest 112-1 and the encoded frames 130-1, 130-2, 130-6 in which it is located. As described above, the second metadata 140 may also identify other independently decodable ROIs 112-2 and the encoded video frames in which they are located.

[0082] In step S206, the search function 202 receives a search query 160. The search query 160 may be input by a user who wants to perform a search in the encoded video data 130. The search may involve a search among ROIs in the encoded video data 130, in particular, it may be a search for ROIs that match a particular attribute or a particular set of attributes. The attribute or set of attributes may be specified by the search query. For example, the search may involve searching the encoded video data for all objects of a particular object category, such as searching for all cars or people in the encoded video data. The search may involve searching for all objects that belong to a particular category and additionally have another particular attribute, such as searching for all cars with a particular license plate number, all cars with a particular color, all people wearing clothing of a particular color, or all people wearing glasses. The search may also involve searching for all objects that match an image of a particular object. For example, the user may have found an instance of an object of interest in the video data and want to find out whether the object of interest is described elsewhere in the encoded video data. In combination Figure 5 and Figure 6 In the examples described, it will be assumed that the search query involves searching for all sports cars in the encoded video data.

[0083] In response to receiving the search query 160, the search function 202 searches the second metadata 140 to find one or more independently decodable ROIs 112-1 and the encoded video frames in which they are located. It should be noted that the search function 103 may also find one or more non-independently decodable ROIs 112-2 at this stage. Which of the ROIs 112-1 and 112-2 the search function 202 searches for and finds in step S208 depends on whether the second metadata 140 indicates the attributes of the ROI.

[0084] In a first set of embodiments, the metadata 140 indicates properties of the ROIs 112-1, 112-2. The search function 202 searches the metadata 140 in step S208 to find one or more independently decodable ROIs 112-1 and also to find one or more other ROIs 112-2 that have properties or sets of properties that match the specific properties or sets of properties of the search query. For example, if the second metadata 140 indicates which ROIs correspond to sports cars, the search function 202 will parse the second metadata 140 in response to a search query requesting sports cars to find ROIs 112-1, 112-2 that correspond to sports cars. Figure 5 In the example shown, in this case, the search function 202 will find ROI 1 in the encoded frame 130 - 6 .

[0085] In a second set of embodiments, the metadata 140 does not indicate any type of attributes for the ROIs 112-1, 112-2, or indicates some, but not all, types of attributes specified by the search query. In this case, the search function 202 cannot immediately infer from the metadata 140 which of the ROIs matches the attribute or attribute set specified by the search query. Instead, the search function 202 finds one or more independently decodable ROIs 112-1 and one or more non-independently decodable ROIs 112-2 that are candidates for matching the search query. In some cases, such as when the second metadata 140 does not include any type of ROI attributes, these candidate ROIs include all independently decodable ROIs 112-1. In Figure 5In the example of , the search function 103 then identifies all independently decodable ROIs 112-1 as candidate ROIs. In other cases, such as when the search query indicates a set of attributes of different types, and the metadata 140 only includes attributes of a subset of these types, the candidate ROIs may include a subset of the independently decodable ROIs 112-1. For example, the second metadata 140 may indicate the object class of the ROIs, such as whether they are people or cars. However, the second metadata 140 may not specify further attributes of the people or cars, such as whether the people are "male" or "female," or whether the cars are "classic" or "sports." Upon receiving a search query for sports cars, the search function 103 may parse the metadata 140 to identify independently decodable ROIs 112-1 that are cars, as these are also candidates for sports cars.

[0086] The search function 202 then forwards an indication 142 to the decoding function 203 regarding the independently decodable ROIs 112-2 found during the search, as well as any ROIs 112-2 that are not independently decodable. The indication 142 can be viewed as an instruction to the decoding function 203 to decode the found ROIs. The indication can be based on the identifier of the found ROI and the encoded video frame in which it is located. For example, the indication 142 can include pairs of identifiers, where each pair includes an identifier of an encoded video frame and an identifier of an ROI therein, such as (frame ID, ROI ID).

[0087] To decode independently decodable ROIs found during the search, the decoding function 203 performs steps S210a and S210b for each independently decodable ROI found. In step S210a, the decoding function 203 uses information 132 embedded in the coded video data 130 to identify the group of CUs associated with the independently decodable ROI. For example, the decoding function 203 may parse the information 132 embedded in the video data to find the ROI identifier associated with the coded video frame in which the ROI exists. Once the ROI identifier is found, the decoding function 203 may read the list of CUs associated with the ROI in the coded video frame. If the information 132 is embedded in a coded video frame header or included in an SEI frame inserted in conjunction with the coded video frame (such as immediately before or after the coded video frame), the decoding function 203 may use the coded video frame identifier to navigate to the header or SEI frame associated with the coded video frame and then use the ROI identifier to identify the CU associated with the ROI. In the case where the information 132 is embedded in a less frequent SEI frame (such as once per GOP), the decoding function 203 can parse the SEI frame to find an identifier pair consisting of an ROI identifier and a frame identifier, and then read out the CU list associated with the identifier pair.

[0088] Once the set of CUs associated with the independently decodable ROIs in the encoded video frame has been identified, the decoding function 203 proceeds to selectively decode these CUs in step S210b to produce decoded video data corresponding to the independently decodable ROIs. To this end, the decoding function 203 can parse the encoded video data 130 to find the encoded video data associated with the identified CUs and decode the encoded video data associated with the identified CUs independently of any other video data in other video frames or other CUs of the current video frame. This is possible because the ROIs were encoded by the encoder system 100 without reference to previously decoded video data in other frames or in CUs outside the identified set of CUs in the current frame. The decoding function 203 can therefore decode the CUs in the identified set of CUs without first decoding video data in other CUs in other encoded video frames or outside the set of CUs in the same encoded video frame.

[0089] It should be noted that the decoding function 203 can also decode video data for ROIs 112-2 found by the search function 202 but which are not independently decodable. Since the video data in these ROIs 112-2 is typically encoded according to standard implementations with reference to previously decoded video data in other video frames or CUs outside the ROIs in the same video frame, these ROIs 112-1 cannot be decoded independently but need to be decoded starting from the previously intra-coded frame.

[0090] Finally, the decoding system 200 outputs the decoded video data 150 that matches the attributes given by the search query. In a first set of embodiments, where the ROIs found by the search function 202 are known to match the attributes specified by the search query, the decoded video data produced by the decoding function 203 can be output to the user as decoded video 150 that matches the attributes of the search query. This option is represented by the solid arrow 150 leading from the decoding function 203 in Figure 3 Fig. 1.

[0091] In a second set of embodiments, where the ROIs found by the search function 202 are candidates for matching the attributes specified by the search query, the decoding function 203 forwards the decoded video data 132 of the candidate ROIs to the analysis function 206. The analysis function 206 analyzes the decoded video data 132 corresponding to each ROI to determine whether it matches the particular attribute or set of attributes specified by the search query 160. When a match is found, the decoded video data of the ROI is output as decoded video data 150 matching the search query. To this end, the analysis function 206 can generally implement any known analysis method, such as an analysis method implemented by a convolutional neural network. The method implemented by the analysis function 206 can depend on the type of search attributes to be supported by the decoding system 200. For example, if searching for objects of a particular predefined object class is to be supported, the analysis function 206 can implement an object classification method that takes as input the decoded video data 132 of the ROI and determines the object class of the object described by the video data. If the determined object category matches the object category of the search query, the decoded video data of the ROI may be output as decoded video data 150 matching the search query 160. Figure 5 In the example of , the search query 160 is for sports cars, which should be in the predefined categories supported by the object classification method. The decoding system 100 will then output Figure 6 Decoded video data 150 depicting a sports car is shown. In another example, where the search is for a specific object depicted by an image, analysis function 206 may implement a re-identification method that outputs a similarity score reflecting the similarity between the object depicted in the video data of the ROI and the image of the specific object. If the score is above a threshold, the decoded video data of the ROI may be output as decoded video data 150 matching the search query 160. It should be understood that there are many more examples of analysis methods that can be included in analysis function 206, including methods for identifying color, pose, items worn or carried by the subject, and the like. Such analysis methods (including standard classifiers trained to classify colors, OpenPose (arXiv:1812.08008) for identifying poses, Mask-RCNN (arXiv:1703.06870) for detecting objects of a given class, or Single Shot Detector (SSD) (arXiv:1512.02325)) are known in the art and are not further described herein.

[0092] It should be understood that those skilled in the art can modify the above-described embodiments in a variety of ways and still utilize the advantages of the present invention as shown in the above-described embodiments. Therefore, the present invention should not be limited to the embodiments shown, but should be limited only by the appended claims. In addition, as will be appreciated by those skilled in the art, the embodiments shown may be combined.

Claims

1. A method for decoding video data in conjunction with performing a search in the video data, comprising: receiving encoded video data comprising an encoded video frame in which an independently decodable region of interest exists, wherein video data in a coding unit group that spatially overlaps with an independently decodable region of interest in a video frame is always intra-coded to reference only video data within the coding unit group that overlaps with the independently decodable region of interest, thereby enabling the independently decodable region of interest in the video frame to be decoded by using or referencing only the coded video data in the coding units of the coded video frame that spatially overlap with the region of interest, receiving metadata comprising data units as separate data entities from the coded video data, the data units each comprising an identifier of a coded video frame, an identifier of each region of interest in the coded video frame, and an identification of which regions of interest in the coded video frame are independently decodable, wherein the coded video data embeds information associating the identifier of each region of interest identified in the metadata as independently decodable with a group of coding units that spatially overlap with the region of interest, receiving a search query related to searching among regions of interest in the encoded video data, In response to receiving the search query, searching the metadata to find and provide identifiers of one or more regions of interest identified as independently decodable and the encoded video frames in which the one or more regions of interest are present, For each found region present in a coded video frame: identifying a group of coding units associated with the identifier of the region of interest by using the information embedded in the coded video data, and decoding the identified group of coding units of the coded video frame independently of video data in other video frames and independently of video data in coding units other than the group of coding units in the coded video frame to produce decoded video data corresponding to the region of interest.

2. The method for decoding video data according to claim 1, wherein The metadata further indicates properties of the region of interest, and the search query involves searching among regions of interest in the encoded video having specific properties, and The search includes searching the metadata to find one or more regions of interest identified as independently decodable and having properties that match specific properties of the search query.

3. The method for decoding video data according to claim 1, wherein: The search query involves searching among regions of interest having specific properties in the encoded video data, and the method further comprises: The decoded video data corresponding to the region of interest is analyzed to determine whether the region of interest matches the specific attributes of the search query.

4. A method for encoding video data, comprising: receiving video data comprising a video frame in which a region of interest exists, receiving first metadata comprising an identifier of the region of interest and an identifier of the video frame in which the region of interest exists, selecting one or more video frames and one or more regions of interest in each of the one or more video frames using the received first metadata, Encoding the video data to generate encoded video data, wherein encoding each selected video frame and each selected region of interest in each selected video frame comprises: Associating the region of interest with a set of coding units that spatially overlap with the region of interest, encoding the video data of the video frames in the coding unit group by always intra-coding the video data in the coding unit group so as to refer only to video data within the coding unit group, independently of video data in other video frames and independent of video data in coding units outside the coding unit group in the video frame that spatially overlap with the region of interest, embedding information in the encoded video data, the information associating an identifier of the region of interest with the group of coding units that spatially overlap with the region of interest, and Second metadata is generated, the second metadata including the identifier of the region of interest, the identifier of the video frame, and an independently decodable identification of the region of interest, wherein the second metadata is a data entity separate from the encoded video data.

5. The method for encoding video data according to claim 4, wherein: The selected one or more video frames comprise a strict subset of the video frames where the region of interest is located.

6. The method for encoding video data according to claim 4, wherein: The received first metadata further identifies a plurality of regions of interest in subsequent video frames as belonging to a common track, wherein the selected one or more video frames include fewer than all of the subsequent video frames.

7. The method for encoding video data according to claim 4, wherein: The selected one or more video frames and the selected one or more regions of interest in each of the selected one or more video frames are further identified based on the proximity between the image attributes within the region of interest and the target image attributes.

8. The method for encoding video data according to claim 4, wherein: The number of pixels in the video data covered by the selected one or more regions of interest in the selected one or more video frames is less than a predetermined proportion of the total number of pixels per time unit in the video data or the total number of pixels of each identified video frame.

9. The method for encoding video data according to claim 4, further comprising: Video data within the region of interest is analyzed to associate the region of interest with attributes, and the attributes of the region of interest are included in the second metadata.

10. The method for encoding video data according to claim 4, wherein: The video data in the group of coding units that spatially overlaps the region of interest is intra-coded.

11. A decoding system for decoding video data in conjunction with performing a search in the video data, comprising circuitry configured to: A receiving function is configured to receive encoded video data including an encoded video frame in which an independently decodable region of interest exists, wherein video data in a coding unit group that spatially overlaps with an independently decodable region of interest in a video frame is always intra-coded to reference only video data within the coding unit group that overlaps with the independently decodable region of interest, thereby enabling the independently decodable region of interest in the video frame to be decoded using or referring only to the coded video data in the coding units of the coded video frame that spatially overlap with the region of interest, and receiving metadata comprising data units as data entities separate from the coded video data, the data units each comprising an identifier of the coded video frame, an identifier of each region of interest in the coded video frame, and an identifier of which regions of interest in the coded video frame are independently decodable, wherein the coded video data embeds information associating the identifier of each region of interest identified in the metadata as independently decodable with the coding unit group that spatially overlaps with the region of interest, a search function configured to receive a forensic search query relating to a search among regions of interest in the coded video data and, in response to receiving the forensic search query, search the metadata to find and provide identifiers of one or more regions of interest identified as independently decodable and the coded video frames in which the one or more regions of interest are present, A decoding function is configured to: for each found region of interest present in a coded video frame: identify a group of coding units associated with the identifier of the region of interest by using information embedded in the coded video data, and decode the identified group of coding units of the coded video frame independently of video data in other video frames and independently of video data in coding units outside the group of coding units in the coded video frame to produce decoded video data corresponding to the region of interest.

12. A coding system for encoding video data, comprising circuitry configured to: a receiving function configured to receive video data comprising a video frame in which a region of interest is present, and configured to receive first metadata comprising an identifier of the region of interest and an identifier of the video frame in which the region of interest is present, a region of interest selection function configured to select one or more video frames and one or more regions of interest in each of the one or more video frames by using the received first metadata, an encoding function configured to encode the video data to generate encoded video data, wherein Encoding each selected video frame and each selected region of interest in each selected video frame includes: Associating the region of interest with a set of coding units that spatially overlap with the region of interest, encoding the video data of the video frames in the coding unit group by always intra-coding the video data in the coding unit group so as to refer only to video data within the coding unit group, independently of video data in other video frames and independent of video data in coding units outside the coding unit group in the video frame that spatially overlap with the region of interest, embedding information in the encoded video data, the information associating an identifier of the region of interest with the group of coding units that spatially overlap with the region of interest, and Second metadata is generated, the second metadata including the identifier of the region of interest, the identifier of the video frame, and an independently decodable identification of the region of interest, wherein the second metadata is a data entity separate from the encoded video data.

13. A non-transitory computer-readable medium having computer code instructions stored thereon, which, when executed by a device having processing capabilities, cause the device to perform the decoding method according to any one of claims 1 to 3.

14. A non-transitory computer-readable medium having computer code instructions stored thereon, which, when executed by a device having processing capabilities, cause the device to perform the encoding method according to any one of claims 4 to 10.

Images