Reference pixel selection for dimd extension
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NOKIA TECHNOLOGIES OY
- Filing Date
- 2024-11-14
- Publication Date
- 2026-08-04
Smart Images
Figure CN122514941A_ABST
Abstract
Description
Technical Field
[0001] This example and non-limiting embodiment generally relates to image decoding, and more specifically to the selection of reference samples for decoding. Background Technology
[0002] It is known that in decoder-side intra-prediction mode derivation (DIMD), reference samples from the luminance and / or chrominance channels are used for various DIMD extensions. Summary of the Invention
[0003] The following description of the invention is intended to be illustrative only. It is not intended to limit the scope of the claims.
[0004] According to a first aspect, an apparatus includes: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: determine one or more reference samples, at least in part, based on at least one predetermined sparse arrangement of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determine at least one intra-frame prediction mode, at least in part based on the determined one or more reference samples.
[0005] According to one aspect, a method includes: determining one or more reference samples, at least in part based on at least one predetermined sparse permutation of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determining at least one intra-frame prediction mode, at least in part based on the determined one or more reference samples.
[0006] According to one aspect, an apparatus includes components for: determining one or more reference samples at least in part based on at least one predetermined sparse arrangement of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determining at least one intra-frame prediction mode at least in part based on the determined one or more reference samples.
[0007] According to one aspect, a non-transitory computer-readable medium includes program instructions stored thereon for at least performing the following operations: determining one or more reference samples, at least in part based on at least one predetermined sparse permutation of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determining at least one intra-frame prediction mode, at least in part based on the determined one or more reference samples.
[0008] The independent claims provide the subject matter for several aspects. Additional aspects are defined in the dependent claims. Attached Figure Description
[0009] The foregoing aspects and other features are explained in the following description in conjunction with the accompanying drawings, in which:
[0010] Figure 1 This is a block diagram of one possible, non-limiting, example system in which example embodiments can be practiced;
[0011] Figure 2 This is a block diagram of one possible, and non-limiting, exemplary system in which example embodiments can be practiced;
[0012] Figure 3 This is a schematic diagram illustrating the features described in this article;
[0013] Figure 4 This is a schematic diagram illustrating the features described in this article;
[0014] Figure 5 This is a schematic diagram illustrating the features described in this article;
[0015] Figure 6 This is a schematic diagram illustrating the features described in this article;
[0016] Figure 7 This is a schematic diagram illustrating the features described in this article;
[0017] Figure 8 This is a schematic diagram illustrating the features described in this article;
[0018] Figure 9 This is a schematic diagram illustrating the features described in this article;
[0019] Figure 10 This is a schematic diagram illustrating the features described in this article;
[0020] Figure 11 This is a schematic diagram illustrating the features described in this article;
[0021] Figure 12 This is a schematic diagram illustrating the features described in this article;
[0022] Figure 13 This is a schematic diagram illustrating the features described in this article;
[0023] Figure 14 This is a schematic diagram illustrating the features described in this article; and
[0024] Figure 15 This is a flowchart illustrating the steps described in this article. Detailed Implementation
[0025] The following abbreviations, which may be found in the specification and / or drawings, are defined as follows:
[0026] The following describes suitable apparatus and possible mechanisms for practicing exemplary embodiments of this disclosure. Therefore, reference is made first. Figure 1 The diagram illustrates an example block diagram of device 50. This device can be configured to perform various functions, such as, for example, collecting information via one or more sensors, encoding and / or decoding information, receiving and / or transmitting information, analyzing information collected or received by the device, etc. A device configured to encode a video scene may (optionally) include one or more microphones for capturing the scene and / or one or more sensors (such as a camera) for capturing information about a physical environment in which the scene is captured. Alternatively, a device configured to encode a video scene may be configured to receive information about an environment in which the scene is captured and / or that the environment is an analog environment. A device configured to decode and / or present a video scene may be configured to receive a Moving Image Experts Group Immersive Codec Family (MPEG-I) bitstream comprising the encoded video scene. A device configured to decode and / or present a video scene may include one or more speaker / audio transducers and / or displays, and / or may be configured to transmit the decoded scene or signal to a device comprising one or more speaker / audio transducers and / or displays. Devices configured to decode and / or present video scenes may include user devices, head-mounted displays, or other devices capable of presenting AR, VR, and / or MR experiences to users.
[0027] Electronic device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system. Alternatively, the electronic device may be a computer or part of a non-mobile computer. It should be understood that the exemplary embodiments of this disclosure can be implemented within any electronic device or apparatus capable of processing data. Electronic device 50 may include a device capable of accessing a network and / or cloud via a wired or wireless connection. Electronic device 50 may include one or more processors 56, one or more memories 58, and one or more transceivers 52 interconnected via one or more buses. The one or more processors 56 may include a central processing unit (CPU) and / or a graphics processing unit (GPU). Each of the one or more transceivers 52 includes a receiver and a transmitter. The one or more buses may be an address bus, a data bus, or a control bus, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics, or other optical communication devices. “Circuit” may include dedicated hardware or hardware associated with software executable thereon. The one or more transceivers may be connected to one or more antennas 44. The one or more memories 58 may include computer program code. The one or more memories 58 and the computer program code may be configured, together with the one or more processors 56, to cause electronic device 50 to perform one or more operations described herein.
[0028] Electronic device 50 can be connected to a node in a network. The network node may include one or more processors, one or more memories, and one or more transceivers interconnected via one or more buses. Each of the one or more transceivers includes a receiver and a transmitter. The one or more buses may be address buses, data buses, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optic cables, or other optical communication devices. The one or more transceivers may be connected to one or more antennas. The one or more memories may include computer program code. The one or more memories and computer program code may be configured, together with the one or more processors, to cause the network node to perform one or more operations described herein.
[0029] Electronic device 50 may include microphone 36 or any suitable audio input, which may be a digital or analog signal input. Electronic device 50 may also include audio output device 38, which, in the example embodiments of this disclosure, may be any of the following: headphones, speaker, or analog or digital audio output connection. Electronic device 50 may also include a battery (or, in other example embodiments of this disclosure, the device may be powered by any suitable mobile energy device, such as a solar cell, fuel cell, or clock generator). Electronic device 50 may also include camera 42 or other sensors capable of recording or capturing images and / or video. Additionally or alternatively, electronic device 50 may also include a depth sensor. Electronic device 50 may also include display 32. Electronic device 50 may also include an infrared port for short-range line-of-sight communication with other devices. In other example embodiments of this disclosure, device 50 may also include any suitable short-range communication solution, such as Bluetooth. TM Wireless connection or USB / FireWire wired connection.
[0030] It should be understood that the electronic device 50 configured to perform the example embodiments of this disclosure may have fewer and / or additional components, which may correspond to the processes that the electronic device 50 is configured to perform. For example, a device configured to encode video may not include a speaker or audio transducer and may include a microphone; however, a device configured to present decoded video may not include a microphone and may include a speaker or audio transducer.
[0031] Now refer to Figure 1 The electronic device 50 may include a controller 56, a processor, or a processor circuitry for controlling the device 50. The controller 56 may be connected to a memory 58, which in exemplary embodiments of this disclosure may store data in both image and audio data formats, and / or may also store instructions for implementation on the controller 56. The controller 56 may also be connected to a codec circuitry 54 adapted to perform encoding and / or decoding of audio and / or video data, or to assist in encoding and / or decoding performed by the controller.
[0032] Electronic device 50 may also include a card reader 48 and a smart card 46 (e.g., UICC and UICC card reader) for providing user information and suitable for providing authentication information for authenticating and authorizing the user / electronic device 50 at the network. Electronic device 50 may also include an input device 34 (such as a keyboard, one or more input buttons, or a touch screen input device) for providing information to controller 56.
[0033] Electronic device 50 may include a wireless interface circuitry system 52 connected to a controller and adapted to generate wireless communication signals, such as for communicating with a cellular communication network, a wireless communication system, or a wireless local area network. Device 50 may also include an antenna 44 connected to the wireless interface circuitry system 52 for transmitting radio frequency signals generated at the wireless interface circuitry system 52 to other devices(s) and / or for receiving radio frequency signals from other devices(s).
[0034] Electronic device 50 may include microphone 38, camera 42, and / or other sensors capable of recording or detecting audio signals, image / video signals, and / or other information about the local / virtual environment, which are then passed to codec 54 or controller 56 for processing. Before transmission and / or storage, electronic device 50 may receive audio / image / video signals and / or information about the local / virtual environment from another device for processing. Electronic device 50 may also receive audio / image / video signals and / or information about the local / virtual environment for encoding / decoding via a wireless or wired connection. The structural elements of electronic device 50 described above represent examples of components used to perform corresponding functions.
[0035] Memory 58 can be of any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as semiconductor-based storage devices, flash memory, magnetic storage devices and systems, optical storage devices and systems, fixed memory, and removable memory. Memory 58 can be non-transitory memory. Memory 58 can be a component for performing storage functions. Controller 56 can be one or more processors, or include one or more processors, which can be of any type suitable for the local technical environment and, as a non-limiting example, can include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture. Controller 56 can be a component for performing functions.
[0036] According to an example embodiment of this disclosure, electronic device 50 may be configured to perform volumetric scene capture. For example, electronic device 50 may include camera 42 or other sensors capable of recording or capturing images and / or video. Electronic device 50 may also include one or more transceivers 52 to enable the transmission of captured content to be processed at another device. Such electronic device 50 may or may not include... Figure 1 All modules shown in the diagram.
[0037] According to exemplary embodiments of this disclosure, electronic device 50 may be configured to perform volumetric video content processing. For example, electronic device 50 may include a controller 56 for processing images to generate volumetric video content, a controller 56 for processing volumetric video content to project 3D information onto 2D information, tiles, and auxiliary information, and / or a codec 54 for encoding 2D information, tiles, and auxiliary information into a bitstream (for transmission to another device via wireless interface 52). Such electronic device 50 may or may not include... Figure 1 All modules shown in the diagram.
[0038] According to exemplary embodiments of this disclosure, electronic device 50 may be configured to perform encoding or decoding of 2D information representing volumetric video content. For example, electronic device 50 may include a codec 54 for encoding or decoding 2D information representing volumetric video content. Such electronic device 50 may or may not include... Figure 1 All modules shown in the diagram.
[0039] According to exemplary embodiments of this disclosure, electronic device 50 may be configured to perform the presentation of decoded 3D volumetric video. For example, electronic device 50 may include a controller for projecting 2D information to reconstruct 3D volumetric video, and / or a display 32 for presenting the decoded 3D volumetric video. Such electronic device 50 may or may not include... Figure 1 All modules shown in the diagram.
[0040] about Figure 2This illustrates an example of a system in which exemplary embodiments of the present disclosure can be utilized. System 10 includes multiple communication devices that can communicate via one or more networks. System 10 may include any combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, E-UTRA, LTE, CDMA, 4G, 5G networks, etc.), wireless local area networks (WLANs) (such as those defined by any IEEE 802.x standard), Bluetooth (BLUETOOTH™) personal area networks, Ethernet LANs, Token Ring LANs, wide area networks, and / or the Internet. Wireless networks can implement network virtualization, which is the process of combining hardware and software network resources and network functions into a single, software-based management entity (virtual network). Network virtualization involves platform virtualization (often combined with resource virtualization). Network virtualization is classified as external virtualization (combining many networks or parts of networks into virtual units) or internal virtualization (providing network-like functionality to software containers on a single system). For example, a network can be deployed in a telecommunications cloud, featuring Virtualized Network Functions (VNFs) running on, for example, data center servers. For instance, core network functions and / or (multiple) radio access networks (e.g., CloudRAN, O-RAN, edge cloud) can be virtualized. It should be noted that the virtualized entities resulting from network virtualization are still implemented (to some extent, using hardware such as processors and memory), and these virtualized entities also create technological effects.
[0041] It should also be noted that the operations of the exemplary embodiments of this disclosure can be performed by multiple cooperating devices (e.g., cRAN).
[0042] System 10 may include both wired and wireless communication devices and / or electronic devices suitable for implementing example embodiments of this disclosure.
[0043] For example, Figure 2 The system shown illustrates a representation of mobile phone network 11 and Internet 28. The connection to Internet 28 can include, but is not limited to, long-range wireless connections, short-range wireless connections, and various wired connections (including but not limited to telephone lines, cables, power lines, and similar communication paths).
[0044] The example communication devices shown in System 10 may include, but are not limited to, device 15, a combination of a personal digital assistant (PDA) and a mobile phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22, and a head-mounted display (HMD) 17. Electronic device 50 may include any one of these example communication devices. In one example embodiment of this disclosure, more than one of these devices, or multiple devices of one or more of these devices, may perform the disclosed processes(s). These devices may be connected to the Internet 28 via wireless connection 2.
[0045] The exemplary embodiments of this disclosure can also be implemented in: set-top boxes (i.e., digital TV receivers, which may or may not have a display or wireless capabilities), tablet computers or (laptop) personal computers (PCs) (which have hardware and / or software for processing neural network data), various operating systems, and chipsets, processors, DSPs, and / or embedded systems that provide hardware / software-based coding. The exemplary embodiments of this disclosure can also be implemented in cellular phones (such as smartphones), tablet computers, personal digital assistants (PDAs) with wireless communication capabilities, portable computers with wireless communication capabilities, image capture devices (such as digital cameras) with wireless communication capabilities, gaming devices with wireless communication capabilities, music storage and playback devices with wireless communication capabilities, internet devices that allow wireless internet access and browsing, tablet computers with wireless communication capabilities, and portable units or terminals incorporating combinations of these functions.
[0046] Some or other devices can send and receive calls and messages and communicate with service providers via the wireless connection 25 to base station 24, which can be, for example, an eNB, gNB, access point, access node, or other node. Base station 24 can connect to a network server 26 that allows communication between mobile phone network 11 and Internet 28. The system may include additional communication equipment and various types of communication devices.
[0047] Communication devices can communicate using various transmission technologies, including but not limited to Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Transmission Control Protocol-Internet Protocol (TCP-IP), Short Message Service (SMS), Multimedia Message Service (MMS), email, Instant Message Service (IMS), Bluetooth™, IEEE 802.11, 3GPP Narrowband IoT, and any similar wireless communication technologies. Communication devices implementing the various example embodiments of this disclosure can communicate using various media, including but not limited to radio, infrared, laser, cable connections, and any suitable connection.
[0048] In telecommunications and data networks, a channel can refer to either a physical channel or a logical channel. A physical channel can refer to a physical transmission medium (such as a wire), while a logical channel can refer to a logical connection over a multiplexed medium (capable of transmitting multiple logical channels). A channel can be used to transmit information signals (such as bitstreams, which may be MPEG-I bitstreams) from one or more transmitters to one or more receivers.
[0049] Having thus introduced a suitable but non-limiting technical context for the practice of exemplary embodiments of this disclosure, the exemplary embodiments will now be described in more detail.
[0050] The features described in this article can typically relate to processes that occur at the decoder. Figure 3A schematic diagram of an example encoder (302) and an example decoder (340) is illustrated. In the encoder (302), the input image (304) can be divided into CUs or CTUs (306), and the prediction blocks can be subtracted (308) to form a residual (310). This residual can be transformed (312) and quantized (314) before being encoded (316) as compressed bits (318) into a bitstream. The quantized transform coefficients can also be dequantized / inverse quantized (320) and inverse transformed (322), and combined with the output of the prediction block (324). This result can then be used for intra-frame prediction (326) and can be (e.g., in parallel) loop filtered (328), included in the decoded image buffer (330), and used for inter-frame prediction (332). In the decoder (340), the compressed bits (342) can be decoded (344), dequantized (346), inverse transformed (348), and combined with the output of the prediction block (350). The result can then be used for intra-frame prediction (352) and can be (e.g., in parallel) loop filtered (354), included in the decoded image buffer (356), and used for inter-frame prediction (358). The contents of the decoded image buffer (356) can be output (360).
[0051] like Figure 3 As shown, the decoder (340) can actually become part of the encoding loop of the encoder (302) in the reverse way (e.g., 320-332). The quantization transform coefficients in the encoder (e.g., after the quantization block (314), or the output of CABAC (344) in the decoder) can be dequantized (320, 346) and inverse transformed (322, 348) to generate encoded residual blocks (e.g., 324, 350). Then (intra-frame or inter-frame) prediction blocks (326, 332, 352, 358) can be added to the encoded residual blocks (324, 350) to generate reconstructed blocks. Loop filtering can be performed on the reconstructed blocks (328, 354) to form the final reconstructed blocks. The final reconstructed blocks can be stored in the decoded image buffer (330, 356) for output (360) and for possible use in future encoding.
[0052] Although Figure 3 Not illustrated, but the decoder may include a mode selector (e.g., if more than one intra-prediction mode exists). Therefore, each mode can perform intra-prediction and provide a prediction signal to the mode selector.
[0053] The features described in this paper typically involve decoder-side intra-prediction mode derivation (DIMD), an intra-coding tool in the Enhanced Compression Model (ECM), a video coding standard currently under development sponsored by JVET. DIMD forms a prediction block by fusing a planar mode predictor with predictors of up to five intra-prediction modes (e.g., DIMD modes). To conserve overhead bits, DIMD (e.g., a module or decoder implementing DIMD) analyzes reference samples to derive the DIMD mode. The reference samples are the reconstructed neighboring samples available at the decoder. Figure 4 The location of the reference sample in DIMD is shown. For example, for block 410, the reconstructed neighboring samples around the block (e.g., the gray block at 420) can be used as reference samples. 430 illustrates a 3x3 vertical gradient operator with a reference sample located in the middle. 440 illustrates a 3x3 horizontal gradient operator with a reference sample located in the middle. Using the 3x3 gradient operator requires that the reference sample be at least one sample away from block 410, whether it is above or below the block, or to the left or right of the block.
[0054] The DIMD pattern is determined from the gradient histogram, where each histogram bar corresponds to the magnitude of the gradient intensity of the angle prediction pattern. A gradient operator is applied to each reference sample to derive the gradient intensity and the angle prediction pattern. The derived gradient intensity for each reference sample is accumulated into the associated histogram bar corresponding to the derived angle prediction pattern.
[0055] DIMD extensions have been included in several ECM coding tools, such as chroma prediction, intra template matching (TMP), and matrix-weighted intra prediction (MIP). DIMD extensions in ECM occur after the luma prediction block and luma reconstruction block have been formed. Unlike the original DIMD, DIMD extensions utilize this information by using reconstructed samples from the luma block as reference samples.
[0056] For blocks that use MIP or intra-TMP prediction, DIMD is used to derive the intra-prediction mode of the current block based on MIP or intra-TMP prediction samples. For MIP, this is done before upsampling.
[0057] ECM currently supports three types of DIMD extensions. For example... Figure 5 As shown in the gray sample at position 510, both the MIP and the DIMD used for intra-frame TMP employ the same reference sample location. These samples are located within the luma block and / or channel. Notably, the outermost sample point of gray sample 510 is separated from the boundary of block 520 by one sample.
[0058] Chromaticity DIMD uses both reference samples from the same color channel and reference samples from the juxtaposed blocks in the luminance channel. Figure 6 The location of the reference sample used for chroma DIMD is shown. Gray sample 610 shows the luma reference sample. Gray sample 630 about block 620 shows the chroma reference sample. The chroma block and luma block can be juxtaposed. If the video is in 420 format (i.e., the luma channel has twice the number of pixel rows and twice the number of pixel columns compared to the chroma channel), then the top left pixel coordinates of the juxtaposed luma block are (2x, 2y), and the top left pixel coordinates of the chroma block are (x, y).
[0059] It is worth noting that the outermost sample of gray sample 610 is separated from the top and left boundaries of the relevant block by one sample, and is separated from the bottom and right boundaries of the relevant block by at least one sample. It is also worth noting that the outermost gray sample 630 is separated from the boundary of the relevant block by one sample.
[0060] Because DIMD mode derivation involves calculating gradient strength and prediction mode for each reference sample, it is a computationally intensive operation. Furthermore, the current design is not well optimized for complexity because DIMD for MIP and intra-TMP uses most samples in the block (except for all samples at block boundaries) as reference samples. Additionally, current DIMD extensions use different reference sample sets for different modes (e.g., DIMD for MIP and intra-TMP can use...). Figure 5 The set shown, however, DIMD for chroma can be used Figure 6 The set shown leads to multiple implementations of gradient derivation. These issues cause DIMD to require a large number of additional cycles for processing.
[0061] It can be noted that intra-prediction modes derived from DIMD used for intra-frame TMP and / or MIP can be used to determine the set of low-frequency non-separable transforms (LFNSTs) in the transform process. It can also be noted that intra-prediction modes derived using chroma DIMD can be used in the prediction process.
[0062] The technical effect of the exemplary embodiments of this disclosure is that the number of reference samples in DIMD can be kept low, for example, without sacrificing too much coding efficiency.
[0063] In this disclosure, the terms “sample” and “pixel” are used interchangeably.
[0064] In this disclosure, the terms “block boundary”, “block edge”, and “block border” are used interchangeably.
[0065] In one example embodiment, a specific set of reference pixels may be involved in a block, for example, according to a predefined arrangement. In one example embodiment, a specific set of reference samples may be used to derive a gradient histogram for DIMD expansion, according to a predefined arrangement. Since the goal of DIMD is to find edge patterns in the encoded block, the sample arrangement can be configured such that only a small number of samples are used to derive edge patterns.
[0066] Compared to samples near the block boundaries, samples in the middle of the block are generally not represented as edge patterns; using only samples in the middle of the block may miss edges occurring near the boundary regions. Therefore, using all samples within the block may not be efficient, because edge patterns detected in the middle part can often be detected in the region near the boundary. Therefore, coding efficiency can be largely maintained when only reference samples near the boundary regions are used to construct the gradient histogram, and reference samples in the middle of the block are skipped.
[0067] In one example embodiment, the arrangement of reference samples can be set to samples in the region near the boundary. One example could use an arrangement comprising each reference sample one pixel away from the block boundary. Now referencing Figure 7 The diagram illustrates a square block of width N and height N, and an example of a sample with coordinates x = 0, ..., N-1 and y = 0, ..., N-1. Accordingly, the following samples can be included as the central samples for gradient calculation:
[0068] It is shown at 710. It is shown at 730. It is shown at 740. It is shown at 720.
[0069] In the pseudocode, the center reference sample selection and gradient calculation for using the function `calculateGradiantAt(x, y)` to compute the gradient at position (x, y) within the block can be performed as follows: for x = range(1, N-2) calculateGradiantAt(x, 1) calculateGradiantAt(x, N-2) for y = range(2, N-3) calculateGradiantAt(1, y) calculateGradiantAt(N-2, y)
[0070] It can be noted that, Figure 7 The examples are not restrictive; different sample ranges are possible.
[0071] Alternatively, sparse sampling of the reference block can be used as a sample permutation to include representative samples from both the block boundaries and the regions within the block. This can be achieved, for example, by selecting reference samples that extend symmetrically from the center of the block toward the corners and / or different edges of the block. Now refer to Figure 8 The diagram illustrates a square block of width N and height N, and an example of samples with coordinates x = 0, ..., N-1 and y = 0, ..., N-1. Accordingly, the following samples can be included as the center samples for gradient calculation:
[0072] It is shown at position 810. It is shown at position 820.
[0073] The center reference sample selection and gradient calculation for using the function calculateGradiantAt(x, y) to compute the gradient at position (x, y) within the block can be performed as follows: for y = range(1, N-2) for x = range(1, N-2) if x == y or x == N-1-y calculateGradiantAt(x, y)
[0074] Alternatively, within the same iteration, a single loop can be used to execute both x == y and x == N-1-y cases: for x = range(1, N-2) calculateGradiantAt(x, x) calculateGradientAt(x,N-1-x)
[0075] It can be noted that, Figure 8 The examples are not restrictive; different sample ranges are possible.
[0076] Different methods can also be combined to achieve a technique that covers a wider range of reference samples while still maintaining the sparsity of the process. For example, both reference samples extending radially from the block center and reference samples at a defined, constant distance from the block boundary can be included in the gradient calculation (e.g., see...). Figure 13 ).
[0077] The sparse permutation of reference samples refers to a permutation of reference samples that includes fewer samples than all the internal samples of the block. For example, if the permutation of reference samples includes more samples than such... Figure 5 If there are fewer reference samples (where all samples at least one pixel away from the block boundary are used as reference samples), the arrangement of reference samples may be sparse. In one example embodiment, the arrangement of samples may be sparse if the arrangement of samples omits at least one sample that is determined to be a repeating edge pattern for a given block.
[0078] In this disclosure, the arrangement of reference samples may refer to at least one of the following: a set of reference samples, a layout of reference samples, a configuration of reference samples, a pattern of reference samples, a constellation diagram of reference samples, and / or a combination of reference samples.
[0079] In this disclosure, a predetermined and / or sparse arrangement of reference samples can be a predetermined minimum or minimal set of reference samples used for intra-frame prediction patterns (e.g., excluding or omitting any samples that are unnecessary or repetitive for determining the pattern). The predetermined sparse arrangement can be used in a DIDM to indicate the location of the reference samples. This arrangement can be fixed per processing block dimension. The DIDM can analyze the reference samples to derive edge patterns corresponding to one or more prediction directions; each direction can be aligned with an angular prediction pattern.
[0080] In one example embodiment, the predetermined sparse permutation may be fixed and may need to be able to work efficiently with any type of video content (e.g., video content with high or low levels of variation). In one example embodiment, the predetermined sparse permutation may include only a subset of the internal samples.
[0081] In one example embodiment, the predetermined sparse arrangement may include all internal samples required for accurate edge pattern detection within a processing block. In another example embodiment, the predetermined sparse arrangement may exclude one or more samples that do not provide useful information for edge pattern detection / derivation / determination. For example, in a flat region where all samples have the same value, it may not be necessary to include all samples in the predetermined sparse arrangement.
[0082] Figures 9 to 11 Three arrangements of the reference sample (grey block) positions for DIMD expansion are shown. These arrangements allow the reference sample to be positioned one row / column away from the block boundary due to the 3x3 gradient operator used in ECM. Alternatively, the reference sample can be located in only one row / column (i.e., the width / thickness of the sample set is one). Alternatively or alternatively, the reference sample can be located more than one row / column away from the boundary, and / or may be located in more than one row / column.
[0083] Alternatively, not every reference sample in a given permutation can be used for gradient calculation. For example, in Figures 9 to 11 In this context, all pixels in the selected row / column can be used as a reference sample. However, it is also possible to use a subset of the pixels in that row / column instead. For example, each odd-numbered pixel (the 1st, 3rd, 5th, etc.) can be used, or the left n pixels and right m pixels in 1010 can be used, and the top p pixels and bottom q pixels in 1020 can be used.
[0084] In one example embodiment, the reference sample may not be located on the block's border or boundary. In other words, the arrangement of the reference samples can exclude any samples or pixels on the block's border or boundary.
[0085] Now for reference Figure 9 The illustration shows an example of an arrangement specifying the location of reference samples for DIMD expansion. In this example, the reference samples can be located near the top (910), left (940), bottom (930), and right (920) boundaries. Note that the reference samples are located one row / column away from the block boundaries.
[0086] Now for reference Figure 10 The illustration shows an example of an arrangement specifying the location of reference samples for DIMD expansion. In this example, the reference samples could be located near the top (1010) and left (1020) boundaries. Note that the reference samples are located one row / column away from the block boundaries.
[0087] Now for reference Figure 11 The illustration shows an example of an arrangement specifying the location of reference samples for DIMD expansion. In this example, the reference samples could be located near the bottom (1120) and right (1110) boundaries. Note that the reference samples are located one row / column away from the block boundaries.
[0088] Now for reference Figure 12 The illustration shows an example of an arrangement of reference samples selected symmetrically from the block center for DIMD expansion. It can be noted that the reference samples are located at least one row / column away from the block boundary, but may be located more than one row / column away.
[0089] Now for reference Figure 13 The diagram illustrates the combination. Figure 9 and Figure 12 An example of the method is shown. An example of an arrangement specifying the positions of reference samples used for DIMD expansion is illustrated. In this example, the reference samples can be selected symmetrically from the block center (light gray) and also include additional samples (dark gray) around the block edges.
[0090] In one example embodiment, the UE can be configured with one sparse arrangement of reference samples for luma blocks and another distinct sparse arrangement of reference samples for chroma blocks. For example, in the case of chroma DIMD, two sparse arrangements may exist. For DIMD used for intra-frame TMP and MIP, only one luma arrangement may be required.
[0091] Now for reference Figure 14 The illustration shows an example of an arrangement specifying the locations of reference samples used for DIMD expansion. In this example, reference samples can be located at the top and left borders of a block. Reference samples along these borders can use samples from adjacent reconstructed blocks(s) in their gradient calculations.
[0092] The exemplary embodiments disclosed herein can be applied in the context of artificial intelligence (AI) and / or machine learning (ML). For example, AI / ML neural networks (NNs) or other models can be used to design the arrangement of reference samples in an adaptive manner, such as reconstructing pixels from luminance blocks.
[0093] Figure 15 The illustration depicts potential steps of example method 1500. Example method 1500 may include: determining one or more reference samples, at least in part based on at least one predetermined sparse permutation of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode 1510; and determining at least one intra-frame prediction mode 1520, at least in part based on the determined one or more reference samples. Example method 1500 may be performed, for example, by a decoder, UE, network entity, base station, etc.
[0094] According to one example embodiment, the apparatus may include: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: determine one or more reference samples, at least in part, based on at least one predetermined sparse arrangement of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determine at least one intra-frame prediction mode, at least in part based on the determined one or more reference samples.
[0095] At least one intra-frame prediction mode may include a mode for at least one of the following: chroma prediction, a first mode-dependent transformation based on intra-frame template matching, a second mode-dependent transformation based on matrix-weighted intra-frame prediction, a transformation process, and a prediction process.
[0096] One or more reference samples may include one or more reconstructed brightness samples.
[0097] At least one predetermined sparse permutation may omit at least one sample, which is determined to be a repeating edge pattern for a given block.
[0098] At least one predetermined sparse permutation may include a set of internal samples less than a block.
[0099] At least one predetermined sparse permutation may omit at least one sample, which is determined to have the same value as at least one other sample in the block.
[0100] At least one predetermined sparse permutation may include an arrangement of one or more samples that are one sample away from at least one block boundary.
[0101] At least one predetermined sparse permutation may include an arrangement of one or more samples that are the same number of samples away from at least one block boundary.
[0102] At least one predetermined sparse permutation may include an arrangement of: one or more first samples, which are a first number of samples away from the boundary of a first block; and one or more second samples, which are a second number of samples away from the boundary of a second block.
[0103] At least one predetermined sparse permutation may include an arrangement of: one or more first samples, the one or more first samples being a first number of samples away from a first block boundary; one or more second samples, the one or more second samples being a second number of samples away from a second block boundary; one or more third samples, the one or more third samples being a third number of samples away from a third block boundary; and one or more fourth samples, the one or more fourth samples being a fourth number of samples away from a fourth block boundary.
[0104] At least one predetermined sparse permutation may include an arrangement of one or more samples that extend symmetrically from the center of the block toward at least one of the multiple corners of the block.
[0105] At least one predetermined sparse permutation may include an arrangement of one or more samples that extend symmetrically from the center of the block toward at least one of the multiple boundaries of the block.
[0106] At least one predetermined sparse permutation may include an arrangement of: one or more first samples that are the same number of samples away from at least one block boundary; and one or more second samples that extend symmetrically from the middle of the block toward at least one of a plurality of corners of the block.
[0107] At least one predetermined sparse permutation may include an arrangement of: one or more first samples that are the same number of samples away from at least one block boundary; and one or more second samples that extend symmetrically from the middle of the block toward at least one of the multiple boundaries of the block.
[0108] At least one predetermined sparse permutation may include an arrangement of one or more samples that are not on the boundaries of the block.
[0109] At least one predetermined sparse permutation may include an arrangement of one or more samples on at least one block boundary.
[0110] At least one predetermined sparse arrangement may include at least one first predetermined sparse arrangement of one or more reference samples for the luminance block, and at least one second predetermined sparse arrangement of reference samples for the chrominance block.
[0111] According to one aspect, an example method may be provided, comprising: determining one or more reference samples using a user equipment, at least in part based on at least one predetermined sparse permutation of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determining at least one intra-frame prediction mode based at least in part on the determined one or more reference samples.
[0112] At least one intra-frame prediction mode may include a mode for at least one of the following: chroma prediction, a first mode-dependent transformation based on intra-frame template matching, a second mode-dependent transformation based on matrix-weighted intra-frame prediction, a transformation process, or a prediction process.
[0113] One or more reference samples may include one or more reconstructed brightness samples.
[0114] At least one predetermined sparse permutation may omit at least one sample, which is determined to be a repeating edge pattern for a given block.
[0115] At least one predetermined sparse permutation may include a set of internal samples less than a block.
[0116] At least one predetermined sparse permutation may omit at least one sample, which is determined to have the same value as at least one other sample in the block.
[0117] At least one predetermined sparse permutation may include an arrangement of one or more samples that are one sample away from at least one block boundary.
[0118] At least one predetermined sparse permutation may include an arrangement of one or more samples that are the same number of samples away from at least one block boundary.
[0119] At least one predetermined sparse permutation may include an arrangement of: one or more first samples, which are a first number of samples away from a first block boundary; and one or more second samples, which are a second number of samples away from a second block boundary.
[0120] At least one predetermined sparse permutation may include an arrangement of: one or more first samples, the one or more samples being a first number of samples away from the boundary of the first block; one or more second samples, the one or more second samples being a second number of samples away from the boundary of the second block; one or more third samples, the one or more third samples being a third number of samples away from the boundary of the third block; and one or more fourth samples, the one or more fourth samples being a fourth number of samples away from the boundary of the fourth block.
[0121] At least one predetermined sparse permutation may include an arrangement of one or more samples that extend symmetrically from the center of the block toward at least one of the multiple corners of the block.
[0122] At least one predetermined sparse permutation comprises an arrangement of one or more samples that extend symmetrically from the center of the block toward at least one of the multiple boundaries of the block.
[0123] At least one predetermined sparse permutation may include an arrangement of: one or more first samples that are the same number of samples away from at least one block boundary; and one or more second samples that extend symmetrically from the middle of the block toward at least one of a plurality of corners of the block.
[0124] At least one predetermined sparse permutation may include an arrangement of: one or more first samples that are the same number of samples away from at least one block boundary; and one or more second samples that extend symmetrically from the middle of the block toward at least one of the multiple boundaries of the block.
[0125] At least one predetermined sparse permutation may include an arrangement of one or more samples that are not on the boundaries of the block.
[0126] At least one predetermined sparse permutation may include an arrangement of one or more samples on at least one block boundary.
[0127] At least one predetermined sparse arrangement may include at least one first predetermined sparse arrangement of reference samples for a luminance block, and at least one second predetermined sparse arrangement of one or more reference samples for a chrominance block.
[0128] According to one example embodiment, an apparatus may include: a circuit system configured to perform: determining one or more reference samples using a user equipment, at least in part based on at least one predetermined sparse arrangement of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and a circuit system configured to perform: determining at least one intra-frame prediction mode based at least in part on the determined one or more reference samples.
[0129] According to one example embodiment, an apparatus may include: a processing circuit system; and a storage circuit system including computer program code, the storage circuit system and the computer program code being configured together with the processing circuit system such that the apparatus is capable of: determining one or more reference samples, at least in part based on at least one predetermined sparse permutation of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determining at least one intra-frame prediction mode, at least in part based on the determined one or more reference samples.
[0130] As used herein, the terms “circuit system” or “apparatus” may refer to one or more or all of the following: (a) a hardware circuit implementation only (such as an implementation only in analog and / or digital circuit systems), and (b) a combination of hardware circuitry and software, such as (if applicable): (i) a combination of (multiple) analog and / or digital hardware circuitry having software / firmware, and (ii) any portion of (multiple) hardware processors having software (including (multiple) digital signal processors), software, and (multiple) memories, which work together to enable an apparatus (such as a mobile phone or server) to perform various functions, and (c) (multiple) hardware circuitry and / or (multiple) processors, such as (multiple) microprocessors or portions of (multiple) microprocessors, which require software (e.g., firmware) to operate, but may be absent when operation is not required. This definition of “circuit system” applies to all uses of the term in this application (including in any claim). As another example, as used herein, the term circuit system also covers an implementation of hardware circuitry or a processor (or multiple processors) or a portion of hardware circuitry or a processor and its accompanying software and / or firmware. For example, and if applicable to specific elements of the particular claim, the term "circuit" also covers baseband integrated circuits or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices.
[0131] According to one example embodiment, an apparatus may include components for: determining one or more reference samples, at least in part based on at least one predetermined sparse permutation of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determining at least one intra-frame prediction mode, at least in part based on the determined one or more reference samples.
[0132] At least one intra-frame prediction mode may include a mode for at least one of the following: chroma prediction, a first mode-dependent transformation based on intra-frame template matching, a second mode-dependent transformation based on matrix-weighted intra-frame prediction, a transformation process, or a prediction process.
[0133] One or more reference samples may include one or more reconstructed brightness samples.
[0134] At least one predetermined sparse permutation may omit at least one sample, which is determined to be a repeating edge pattern for a given block.
[0135] At least one predetermined sparse permutation may include a set of internal samples less than a block.
[0136] At least one predetermined sparse permutation may omit at least one sample, which is determined to have the same value as at least one other sample in the block.
[0137] At least one predetermined sparse permutation may include an arrangement of one or more samples that are one sample away from at least one block boundary.
[0138] At least one predetermined sparse permutation may include an arrangement of one or more samples that are the same number of samples away from at least one block boundary.
[0139] At least one predetermined sparse permutation may include an arrangement of: one or more first samples, which are a first number of samples away from the boundary of a first block; and one or more second samples, which are a second number of samples away from the boundary of a second block.
[0140] At least one predetermined sparse permutation may include an arrangement of: one or more first samples, the one or more first samples being a first number of samples away from a first block boundary; one or more second samples, the one or more second samples being a second number of samples away from a second block boundary; one or more third samples, the one or more third samples being a third number of samples away from a third block boundary; and one or more fourth samples, the one or more fourth samples being a fourth number of samples away from a fourth block boundary.
[0141] At least one predetermined sparse permutation may include an arrangement of one or more samples that extend symmetrically from the center of the block toward at least one of the multiple corners of the block.
[0142] At least one predetermined sparse permutation comprises an arrangement of one or more samples that extend symmetrically from the center of the block toward at least one of the multiple boundaries of the block.
[0143] At least one predetermined sparse permutation may include an arrangement of: one or more first samples that are the same number of samples away from at least one block boundary; and one or more second samples that extend symmetrically from the middle of the block toward at least one of a plurality of corners of the block.
[0144] At least one predetermined sparse permutation may include an arrangement of: one or more first samples that are the same number of samples away from at least one block boundary; and one or more second samples that extend symmetrically from the middle of the block toward at least one of the multiple boundaries of the block.
[0145] At least one predetermined sparse permutation may include an arrangement of one or more samples that are not on the boundaries of the block.
[0146] At least one predetermined sparse permutation may include an arrangement of one or more samples on at least one block boundary.
[0147] At least one predetermined sparse arrangement may include at least one first predetermined sparse arrangement of reference samples for a luminance block, and at least one second predetermined sparse arrangement of one or more reference samples for a chrominance block.
[0148] A processor, memory, and / or an example algorithm (which may be encoded as instructions, a program, or code) may be provided as an example component for providing or causing the execution of an operation.
[0149] According to one example embodiment, a non-transitory computer-readable medium includes instructions stored thereon that, when executed by at least one processor, cause at least one processor to: determine one or more reference samples, at least in part, based on at least one predetermined sparse permutation of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determine at least one intra-frame prediction mode, at least in part based on the determined one or more reference samples.
[0150] According to an example embodiment, a non-transitory computer-readable medium includes program instructions stored thereon for at least performing the following operations: determining one or more reference samples, at least in part based on at least one predetermined sparse permutation of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determining at least one intra-frame prediction mode, at least in part based on the determined one or more reference samples.
[0151] According to another example embodiment, a machine-readable non-transient program storage device may be provided, which tangibly embodies machine-executable instructions for performing operations including: determining one or more reference samples, at least in part based on at least one predetermined sparse permutation of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determining at least one intra-frame prediction mode, at least in part based on the determined one or more reference samples.
[0152] According to another example embodiment, a non-transitory computer-readable medium includes instructions that, when executed by a device, cause the device to perform at least the following operations: determining one or more reference samples, at least in part based on at least one predetermined sparse arrangement of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determining at least one intra-frame prediction mode, at least in part based on the determined one or more reference samples.
[0153] A computer-implemented system includes: at least one processor and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the system to at least: determine one or more reference samples, at least in part based on at least one predetermined sparse permutation of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and determine at least one intra-frame prediction mode, at least in part based on the determined one or more reference samples.
[0154] A computer-implemented system includes: means for determining one or more reference samples at least partially based on at least one predetermined sparse permutation of one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and means for determining at least one intra-frame prediction mode at least partially based on the determined one or more reference samples.
[0155] The term “non-transient” as used in this article refers to the limitation on the medium itself (i.e., tangible, non-signal), rather than the limitation on the persistence of data storage (e.g., RAM versus ROM).
[0156] It should be understood that the foregoing description is illustrative only. Those skilled in the art can devise various alternatives and modifications. For example, the features stated in the dependent claims can be combined with each other in any suitable combination(s). Furthermore, features from the different embodiments described above can be selectively combined to form new embodiments. Therefore, this specification is intended to cover all such alternatives, modifications, and variations falling within the scope of the appended claims.
Claims
1. An apparatus comprising: at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: determine one or more reference samples based at least in part on at least one predetermined sparse arrangement of the one or more reference samples, the one or more reference samples being used to derive at least one intra prediction mode; and determine the at least one intra prediction mode based at least in part on the determined one or more reference samples.
2. The apparatus of claim 1, wherein the at least one intra prediction mode comprises a mode for at least one of: chroma prediction; a first mode-dependent transform according to intra template matching; a second mode-dependent transform according to matrix weighted intra prediction; a transform process; or a prediction process.
3. The apparatus of claim 1 or 2, wherein the one or more reference samples comprise one or more reconstructed luma samples.
4. The apparatus of any of claims 1 to 3, wherein the at least one predetermined sparse arrangement omits at least one sample determined to be repetitive for an edge mode of a determined block.
5. The apparatus of any of claims 1 to 3, wherein the at least one predetermined sparse arrangement comprises fewer than a set of internal samples of a block.
6. The apparatus of any of claims 1 to 3, wherein the at least one predetermined sparse arrangement omits at least one sample determined to have a same value as at least one other sample of a block.
7. The apparatus of any of claims 1 to 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are one sample from at least one block boundary.
8. The apparatus of any of claims 1 to 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are a same number of samples from at least one block boundary.
9. The apparatus of any of claims 1 to 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a first number of samples from a first block boundary; and one or more second samples that are a second number of samples from a second block boundary.
10. The apparatus of any of claims 1 to 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a first number of samples from a first block boundary; one or more second samples that are a second number of samples from a second block boundary; one or more third samples that are a third number of samples from a third block boundary; and one or more fourth samples that are a fourth number of samples from a fourth block boundary.
11. The apparatus according to any one of claims 1 to 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that extend symmetrically from the center of the block toward at least one of a plurality of corners of the block.
12. The apparatus according to any one of claims 1 to 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that extend symmetrically from the center of the block toward at least one of a plurality of boundaries of the block.
13. The apparatus according to any one of claims 1 to 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of the following: One or more first samples, wherein the one or more first samples are at the same number of samples as at least one block boundary; and One or more second samples, the one or more second samples extending symmetrically from the middle of the block toward at least one of the multiple corners of the block.
14. The apparatus according to any one of claims 1 to 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of the following: One or more first samples, wherein the one or more first samples are at the same number of samples as at least one block boundary; and One or more second samples, which extend symmetrically from the middle of the block toward at least one of the plurality of boundaries of the block.
15. The apparatus according to any one of claims 1 to 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples, the one or more samples not being on the boundary of the block.
16. The apparatus according to any one of claims 1 to 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples on at least one block boundary.
17. The apparatus of claim 1 or 2, wherein the at least one predetermined sparse arrangement comprises at least one first predetermined sparse arrangement of the one or more reference samples for the luminance block, and at least one second predetermined sparse arrangement of the reference samples for the chroma block.
18. A method comprising: The one or more reference samples are determined using a user equipment, based at least in part on at least one predetermined sparse permutation of one or more reference samples, which are used to derive at least one intra-frame prediction mode. as well as The at least one intra-frame prediction mode is determined based at least in part on the determined one or more reference samples.
19. The method of claim 18, wherein the at least one intra-frame prediction mode comprises a mode for at least one of the following: Colorimetric prediction; Based on the first mode-dependent transformation of intra-frame template matching; Based on the second mode-dependent transformation of matrix-weighted intra-frame prediction; Transformation process; or Prediction process.
20. The method of claim 18 or 19, wherein the one or more reference samples comprise one or more reconstructed luminance samples.
21. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement omits at least one sample, the at least one sample being determined as a repeating edge pattern for a defined block.
22. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement comprises a set of internal samples of less than a block.
23. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement omits at least one sample, the at least one sample being determined to have the same value as at least one other sample of the block.
24. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples, the one or more samples being one sample away from at least one block boundary.
25. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples, the one or more samples being a number of samples away from at least one block boundary.
26. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of the following: One or more first samples, wherein the one or more first samples are a first number of samples that are a distance of a first block boundary from the first block boundary; and One or more second samples, wherein the one or more second samples are a second number of samples away from the boundary of the second block.
27. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of the following: One or more first samples, wherein the one or more first samples are a first number of samples away from the boundary of the first block; One or more second samples, wherein the one or more second samples are a second number of samples away from the boundary of the second block; One or more third samples, wherein the one or more third samples are a third number of samples away from the boundary of the third block; and One or more fourth samples, wherein the one or more fourth samples are a fourth number of samples away from the fourth block boundary.
28. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that extend symmetrically from the center of the block toward at least one of a plurality of corners of the block.
29. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that extend symmetrically from the center of the block toward at least one of a plurality of boundaries of the block.
30. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of the following: One or more first samples, wherein the one or more first samples are at the same number of samples as at least one block boundary; and One or more second samples, the one or more second samples extending symmetrically from the middle of the block toward at least one of the multiple corners of the block.
31. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of the following: One or more first samples, wherein the one or more first samples are at the same number of samples as at least one block boundary; and One or more second samples, which extend symmetrically from the middle of the block toward at least one of the plurality of boundaries of the block.
32. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are not on the boundary of the block.
33. The method according to any one of claims 18 to 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples on at least one block boundary.
34. The method of claim 18 or 19, wherein the at least one predetermined sparse arrangement comprises at least one first predetermined sparse arrangement of reference samples for a luminance block, and at least one second predetermined sparse arrangement of the one or more reference samples for a chroma block.
35. An apparatus comprising components for: The one or more reference samples are determined, at least in part, based on at least one predetermined sparse permutation of the one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and The at least one intra-frame prediction mode is determined based at least in part on the determined one or more reference samples.
36. The apparatus of claim 35, wherein the at least one intra-frame prediction mode includes modes for at least one of the following: Colorimetric prediction; Based on the first mode-dependent transformation of intra-frame template matching; Based on the second mode-dependent transformation of matrix-weighted intra-frame prediction; Transformation process; or Prediction process.
37. The apparatus of claim 35 or 36, wherein the one or more reference samples comprise one or more reconstructed luminance samples.
38. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement omits at least one sample, the at least one sample being determined as repeating the edge pattern for a defined block.
39. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement comprises a set of internal samples of less than a block.
40. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement omits at least one sample, the at least one sample being determined to have the same value as at least one other sample of the block.
41. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples, the one or more samples being one sample away from at least one block boundary.
42. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples, the one or more samples being a number of samples away from at least one block boundary.
43. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of the following: One or more first samples, wherein the one or more first samples are a first number of samples that are a distance of a first block boundary from the first block boundary; and One or more second samples, wherein the one or more second samples are a second number of samples away from the boundary of the second block.
44. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of the following: One or more first samples, wherein the one or more first samples are a first number of samples away from the boundary of the first block; One or more second samples, wherein the one or more second samples are a second number of samples away from the boundary of the second block; One or more third samples, wherein the one or more third samples are a third number of samples away from the boundary of the third block; and One or more fourth samples, wherein the one or more fourth samples are a fourth number of samples away from the fourth block boundary.
45. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that extend symmetrically from the center of the block toward at least one of a plurality of corners of the block.
46. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that extend symmetrically from the center of the block toward at least one of a plurality of boundaries of the block.
47. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of the following: One or more first samples, wherein the one or more first samples are at the same number of samples as at least one block boundary; and One or more second samples, the one or more second samples extending symmetrically from the middle of the block toward at least one of the multiple corners of the block.
48. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of the following: One or more first samples, wherein the one or more first samples are at the same number of samples as at least one block boundary; and One or more second samples, which extend symmetrically from the middle of the block toward at least one of the plurality of boundaries of the block.
49. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples, the one or more samples not being on the boundary of the block.
50. The apparatus according to any one of claims 35 to 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples on at least one block boundary.
51. The apparatus of claim 35 or 36, wherein the at least one predetermined sparse arrangement comprises at least one first predetermined sparse arrangement of reference samples for a luminance block, and at least one second predetermined sparse arrangement of the one or more reference samples for a chroma block.
52. A non-transitory computer-readable medium comprising program instructions stored thereon, the program instructions being configured to perform at least the following operations: The one or more reference samples are determined, at least in part, based on at least one predetermined sparse permutation of the one or more reference samples, the one or more reference samples being used to derive at least one intra-frame prediction mode; and The at least one intra-frame prediction mode is determined based at least in part on the determined one or more reference samples.