Reference pixels selection for DIMD extensions

Abstract

An apparatus configured to: determine one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determine the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

Description

REFERENCE PIXELS SELECTION FOR DIMD EXTENSIONSTECHNICAL FIELD

[0001] The example and non-limiting embodiments relate generally to image decoding and, more particularly, to reference sample selection for decoding.BACKGROUND

[0002] It is known, in decoder-side intra prediction mode derivation (DIMD), to use reference samples in the luma channel and / or the chroma channel for various DIMD extensions.SUMMARY

[0003] The following summary is merely intended to be illustrative. The summary is not intended to limit the scope of the claims.

[0004] In accordance with one aspect, an apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: determine one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determine the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0005] In accordance with one aspect, a method comprising: determining, with a user equipment, one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0006] In accordance with one aspect, an apparatus comprising means for: determining one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0007] In accordance with one aspect, a non-transitory computer-readable medium comprising program instructions stored thereon for performing at least the following: determining one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0008] According to some aspects, there is provided the subject matter of the independent claims. Some further aspects are defined in the dependent claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The foregoing aspects and other features are explained in the following description, taken in connection with the accompanying drawings, wherein:

[0010] FIG. 1 is a block diagram of one possible and non-limiting example system in which the example embodiments may be practiced;

[0011] FIG. 2 is a block diagram of one possible and non-limiting exemplary system in which the example embodiments may be practiced;

[0012] FIG. 3 is a diagram illustrating features as described herein;

[0013] FIG. 4 is a diagram illustrating features as described herein;

[0014] FIG. 5 is a diagram illustrating features as described herein;

[0015] FIG. 6 is a diagram illustrating features as described herein;

[0016] FIG. 7 is a diagram illustrating features as described herein;

[0017] FIG. 8 is a diagram illustrating features as described herein;

[0018] FIG. 9 is a diagram illustrating features as described herein;

[0019] FIG. 10 is a diagram illustrating features as described herein;

[0020] FIG. 11 is a diagram illustrating features as described herein;

[0021] FIG. 12 is a diagram illustrating features as described herein;

[0022] FIG. 13 is a diagram illustrating features as described herein;

[0023] FIG. 14 is a diagram illustrating features as described herein; and

[0024] FIG. 15 is a flowchart illustrating steps as described herein.DETAILED DESCRIPTION OF EMBODIMENTS

[0025] The following abbreviations that may be found in the specification and / or the drawing figures are defined as follows:3 GPP third generation partnership project4G fourth generation5G fifth generation5GC 5G core networkAl artificial intelligenceAR augmented realityCDMA code division multiple accessCPU central processing unit cRAN cloud radio access networkDIMD decoder-side intra (prediction) mode derivationECM enhanced compression model eNB (or eNodeB) evolved Node B (e.g., an LTE base station)EN-DC E-UTRA-NR dual connectivity en-gNB or En-gNB node providing NR user plane and control plane protocol terminations towards the UE, and acting as secondary node in EN- DCE-UTRA evolved universal terrestrial radio access, i.e., the LTE radio access technology FDMA frequency division multiple access gNB (or gNodeB) base station for 5G / NR, i.e., a node providing NR user plane and control plane protocol terminations towards the UE, and connected via the NG interface to the 5GCGPU graphical processing unit GSM global systems for mobile communicationsHMD head-mounted displayIEEE Institute of Electrical and Electronics EngineersIMD integrated messaging deviceIMS instant messaging service loT Internet of ThingsJVET Joint Video Experts TeamLFNST low frequency non-separable transformLIE long term evolutionMIP matrix weighted intra prediction ML machine learningMMS multimedia messaging serviceMPEG-I Moving Picture Experts Group immersive codec familyMR mixed reality ng or NG new generation ng-eNB or NG-eNB new generation eNBNN neural networkNR new radioN / W or NW networkO-RAN open radio access networkPC personal computerPDA personal digital assistantSMS short messaging serviceTCP -IP transmission control protocol-internet protocolTDMA time division multiple accessTMP template matchingUE user equipment (e.g., a wireless, typically mobile device)UMTS universal mobile telecommunications systemUSB universal serial busVNR virtualized network functionVR virtual realityWC versatile video codingWLAN wireless local area network

[0026] The following describes suitable apparatus and possible mechanisms for practicing example embodiments of the present disclosure. Accordingly, reference is first made to FIG. 1, which shows an example block diagram of an apparatus 50. The apparatus may be configured to perform various functions such as, for example, gathering information by one or more sensors, encoding and / or decoding information, receiving and / or transmitting information, analyzing information gathered or received by the apparatus, or the like. A device configured to encode a video scene may (optionally) comprise one or more microphones for capturing the scene and / or one or more sensors, such as cameras, for capturing information about the physical environment in which the scene is captured. Alternatively, a device configured to encode a video scene may be configured to receive information about an environment in which a scene is captured and / or a simulated environment. A device configured to decode and / or render the video scene may beconfigured to receive a Moving Picture Experts Group immersive codec family (MPEG-I) bitstream comprising the encoded video scene. A device configured to decode and / or render the video scene may comprise one or more speakers / audio transducers and / or displays, and / or may be configured to transmit a decoded scene or signals to a device comprising one or more speakers / audio transducers and / or displays. A device configured to decode and / or render the video scene may comprise a user equipment, a head / mounted display, or another device capable of rendering to a user an AR, VR and / or MR experience.

[0027] The electronic device 50 may for example be a mobile terminal or user equipment of a wireless communication system. Alternatively, the electronic device may be a computer or part of a computer that is not mobile. It should be appreciated that example embodiments of the present disclosure may be implemented within any electronic device or apparatus which may process data. The electronic device 50 may comprise a device that can access a network and / or cloud through a wired or wireless connection. The electronic device 50 may comprise one or more processors 56, one or more memories 58, and one or more transceivers 52 interconnected through one or more buses. The one or more processors 56 may comprise a central processing unit (CPU) and / or a graphical processing unit (GPU). Each of the one or more transceivers 52 includes a receiver and a transmitter. The one or more buses may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. A “circuit” may include dedicated hardware or hardware in association with software executable thereon. The one or more transceivers may be connected to one or more antennas 44. The one or more memories 58 may include computer program code. The one or more memories 58 and the computer program code may be configured to, with the one or more processors 56, cause the electronic device 50 to perform one or more of the operations as described herein.

[0028] The electronic device 50 may connect to a node of a network. The network node may comprise one or more processors, one or more memories, and one or more transceivers interconnected through one or more buses. Each of the one or more transceivers includes a receiver and a transmitter. The one or more buses may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit,fiber optics or other optical communication equipment, and the like. The one or more transceivers may be connected to one or more antennas. The one or more memories may include computer program code. The one or more memories and the computer program code may be configured to, with the one or more processors, cause the network node to perform one or more of the operations as described herein.

[0029] The electronic device 50 may comprise a microphone 36 or any suitable audio input which may be a digital or analogue signal input. The electronic device 50 may further comprise an audio output device 38 which in example embodiments of the present disclosure may be any one of: an earpiece, speaker, or an analogue audio or digital audio output connection. The electronic device 50 may also comprise a battery (or in other example embodiments of the present disclosure the device may be powered by any suitable mobile energy device such as solar cell, fuel cell, or clockwork generator). The electronic device 50 may further comprise a camera 42 or other sensor capable of recording or capturing images and / or video. Additionally or alternatively, the electronic device 50 may further comprise a depth sensor. The electronic device 50 may further comprise a display 32. The electronic device 50 may further comprise an infrared port for short range line of sight communication to other devices. In other example embodiments of the present disclosure the apparatus 50 may further comprise any suitable short-range communication solution such as for example a BLUETOOTH™ wireless connection or a USB / firewire wired connection.

[0030] It should be understood that an electronic device 50 configured to perform example embodiments of the present disclosure may have fewer and / or additional components, which may correspond to what processes the electronic device 50 is configured to perform. For example, an apparatus configured to encode a video might not comprise a speaker or audio transducer and may comprise a microphone, while an apparatus configured to render the decoded video might not comprise a microphone and may comprise a speaker or audio transducer.

[0031] Referring now to FIG. 1, the electronic device 50 may comprise a controller 56, processor or processor circuitry for controlling the apparatus 50. The controller 56 may be connected to memory 58 which in example embodiments of the present disclosure may store both data in the form of image and audio data and / or may also store instructions for implementation onthe controller 56. The controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and / or decoding of audio and / or video data or assisting in coding and / or decoding carried out by the controller.

[0032] The electronic device 50 may further comprise a card reader 48 and a smart card 46, for example a UICC and UICC reader, for providing user information and being suitable for providing authentication information for authentication and authorization of the user / electronic device 50 at a network. The electronic device 50 may further comprise an input device 34, such as a keypad, one or more input buttons, or a touch screen input device, for providing information to the controller 56.

[0033] The electronic device 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system, or a wireless local area network. The apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and / or for receiving radio frequency signals from other apparatus(es).

[0034] The electronic device 50 may comprise a microphone 38, camera 42, and / or other sensors capable of recording or detecting audio signals, image / video signals, and / or other information about the local / virtual environment, which are then passed to the codec 54 or the controller 56 for processing. The electronic device 50 may receive the audio / image / video signals and / or information about the local / virtual environment for processing from another device prior to transmission and / or storage. The electronic device 50 may also receive either wirelessly or by a wired connection the audio / image / video signals and / or information about the local / virtual environment for encoding / decoding. The structural elements of electronic device 50 described above represent examples of means for performing a corresponding function.

[0035] The memory 58 may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-basedmemory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The memory 58 may be a non-transitory memory. The memory 58 may be means for performing storage functions. The controller 56 may be or comprise one or more processors, which may be of any type suitable to the local technical environment, and may include one or more of general-purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on a multicore processor architecture, as non-limiting examples. The controller 56 may be means for performing functions.

[0036] The electronic device 50 may be configured to perform capture of a volumetric scene according to example embodiments of the present disclosure. For example, the electronic device 50 may comprise a camera 42 or other sensor capable of recording or capturing images and / or video. The electronic device 50 may also comprise one or more transceivers 52 to enable transmission of captured content for processing at another device. Such an electronic device 50 may or may not include all the modules illustrated in FIG. 1.

[0037] The electronic device 50 may be configured to perform processing of volumetric video content according to example embodiments of the present disclosure. For example, the electronic device 50 may comprise a controller 56 for processing images to produce volumetric video content, a controller 56 for processing volumetric video content to project 3D information into 2D information, patches, and auxiliary information, and / or a codec 54 for encoding 2D information, patches, and auxiliary information into a bitstream for transmission to another device with radio interface 52. Such an electronic device 50 may or may not include all the modules illustrated in FIG. 1.

[0038] The electronic device 50 may be configured to perform encoding or decoding of 2D information representative of volumetric video content according to example embodiments of the present disclosure. For example, the electronic device 50 may comprise a codec 54 for encoding or decoding 2D information representative of volumetric video content. Such an electronic device 50 may or may not include all the modules illustrated in FIG. 1.

[0039] The electronic device 50 may be configured to perform rendering of decoded 3D volumetric video according to example embodiments of the present disclosure. For example, the electronic device 50 may comprise a controller for projecting 2D information to reconstruct 3D volumetric video, and / or a display 32 for rendering decoded 3D volumetric video. Such an electronic device 50 may or may not include all the modules illustrated in FIG. 1.

[0040] With respect to FIG. 2, an example of a system within which example embodiments of the present disclosure can be utilized is shown. The system 10 comprises multiple communication devices which can communicate through one or more networks. The system 10 may comprise any combination of wired or wireless networks including, but not limited to a wireless cellular telephone network (such as a GSM, UMTS, E-UTRA, LIE, CDMA, 4G, 5G network etc.), a wireless local area network (WLAN) such as defined by any of the IEEE 802.x standards, a BLUETOOTH™ personal area network, an Ethernet local area network, a token ring local area network, a wide area network, and / or the Internet. A wireless network may implement network virtualization, which is the process of combining hardware and software network resources and network functionality into a single, software-based administrative entity, a virtual network. Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is categorized as either external, combining many networks, or parts of networks, into a virtual unit, or internal, providing network-like functionality to software containers on a single system. For example, a network may be deployed in a tele cloud, with virtualized network functions (VNF) running on, for example, data center servers. For example, network core functions and / or radio access network(s) (e.g. CloudRAN, O-RAN, edge cloud) may be virtualized. Note that the virtualized entities that result from the network virtualization are still implemented, at some level, using hardware such as processors and memories, and also such virtualized entities create technical effects.

[0041] It may also be noted that operations of example embodiments of the present disclosure may be carried out by a plurality of cooperating devices (e.g. cRAN).

[0042] The system 10 may include both wired and wireless communication devices and / or electronic devices suitable for implementing example embodiments of the present disclosure.

[0043] For example, the system shown in FIG. 2 shows a mobile telephone network 11 and a representation of the internet 28. Connectivity to the internet 28 may include, but is not limited to, long range wireless connections, short range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication pathways.

[0044] The example communication devices shown in the system 10 may include, but are not limited to, an apparatus 15, a combination of a personal digital assistant (PDA) and a mobile telephone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22, and a head-mounted display (HMD) 17. The electronic device 50 may comprise any of those example communication devices. In an example embodiment of the present disclosure, more than one of these devices, or a plurality of one or more of these devices, may perform the disclosed process(es). These devices may connect to the internet 28 through a wireless connection 2.

[0045] The example embodiments of the present disclosure may also be implemented in a set-top box; i.e. a digital TV receiver, which may / may not have a display or wireless capabilities, in tablets or (laptop) personal computers (PC), which have hardware and / or software to process neural network data, in various operating systems, and in chipsets, processors, DSPs and / or embedded systems offering hardware / software based coding. The example embodiments of the present disclosure may also be implemented in cellular telephones such as smart phones, tablets, personal digital assistants (PDAs) having wireless communication capabilities, portable computers having wireless communication capabilities, image capture devices such as digital cameras having wireless communication capabilities, gaming devices having wireless communication capabilities, music storage and playback appliances having wireless communication capabilities, Internet appliances permitting wireless Internet access and browsing, tablets with wireless communication capabilities, as well as portable units or terminals that incorporate combinations of such functions.

[0046] Some or further apparatus may send and receive calls and messages and communicate with service providers through a wireless connection 25 to a base station 24, which may be, for example, an eNB, gNB, access point, access node, other node, etc. The base station 24 may beconnected to a network server 26 that allows communication between the mobile telephone network 11 and the internet 28. The system may include additional communication devices and communication devices of various types.

[0047] The communication devices may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), BLUETOOTH™, IEEE 802.11, 3GPP Narrowband loT and any similar wireless communication technology. A communications device involved in implementing various example embodiments of the present disclosure may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.

[0048] In telecommunications and data networks, a channel may refer either to a physical channel or to a logical channel. A physical channel may refer to a physical transmission medium such as a wire, whereas a logical channel may refer to a logical connection over a multiplexed medium, capable of conveying several logical channels. A channel may be used for conveying an information signal, for example a bitstream, which may be a MPEG-I bitstream, from one or several senders (or transmitters) to one or several receivers.

[0049] Having thus introduced one suitable but non-limiting technical context for the practice of the example embodiments of the present disclosure, example embodiments will now be described with greater specificity.

[0050] Features as described herein may generally relate to processes occurring at a decoder. FIG. 3 illustrates diagrams of an example encoder (302) and an example decoder (340). In the encoder (302), input pictures (304) may be divided into a CU or CTU (306), a prediction block may be subtracted (308) to form a residual (310), which may be transformed (312) and quantized (314) before coding (316) as compressed bits (318) into a bitstream. The quantized transformedcoefficients may also be dequantized / inverse quantized (320) and inverse transformed (322), and combined with the output of a prediction block (324). The result may then be used in intra prediction (326) and may be (e.g. in parallel) in-loop filtered (328), included in a decoded picture buffer (330), and used in inter prediction (332). In the decoder (340), compressed bits (342) may be decoded (344), dequantized (346), inverse transformed (348), and combined with the output of a prediction block (350). The result may then be used in intra prediction (352) and may be (e.g. in parallel) in-looped filtered (354), included in a decoded picture buffer (356), and used in inter prediction (358). The contents of the decoded picture buffer (356) may be output (360).

[0051] As shown in FIG. 3, the decoder (340) may actually be part of the coding loop of the encoder (302) in a reverse way (e.g. 320-332). The quantized transformed coefficients in the encoder (e.g. after quantization block (314), or the output of CAB AC (344) in the decoder) may be dequantized (320, 346) and inverse transformed (322, 348), generating the coded residual block (e.g. 324, 350). The (intra or inter) prediction block (326, 332, 352, 358) may then be added to the coded residual block (324, 350), generating the reconstructed block. In-loop filtering may be performed over the reconstructed block (328, 354), forming the final reconstructed block. The final reconstructed blocks may be stored in a decoded picture buffer (330, 356) for output (360), as well as for possible use of future coding.

[0052] While not illustrated in FIG. 3, a decoder may include a mode selector, for example if there is more than one intra-prediction mode. Hence, each mode may perform the intraprediction, and provide the predicted signal to the mode selector.

[0053] Features as described herein may generally relate to Decoder-side Intra prediction Mode Derivation (DIMD), an intra coding tool in Enhanced Compression Model (ECM), a video coding standard that is currently under the development sponsored by JVET. DIMD forms a prediction block by fusing a planar mode predictor with predictors from up to five intra prediction modes (e.g. DIMD modes). To save overhead bits, DIMD (e.g. a module or decoder implementing DIMD) analyses reference samples to derive DIMD modes. The reference samples are the reconstructed neighbor samples that are available at the decoder side. FIG. 4 shows locations of reference samples in DIMD. For example, for block 410, the reconstructed neighbor samplesaround the block (e.g. gray blocks at 420) may be used as reference samples. 430 illustrates a 3x3 vertical gradient operator with a reference sample in the middle. 440 illustrates a 3x3 horizontal gradient operator with a reference sample in the middle. Employing 3x3 gradient operators requires that the reference sample be separated from the block 410 at least by one, either above or below, and either to the left or to the right of the block.

[0054] DIMD mode is determined from a histogram of gradients, where each histogram bin corresponds to the amplitude of gradient strength for an angular prediction mode. Gradient operators are applied to each reference sample to derive gradient strength and an angular prediction mode. The derived gradient strength of each reference sample is accumulated to an associated histogram bin corresponding to the derived angular prediction mode.

[0055] Extensions of DIMD have been included in several ECM coding tools, such as chroma prediction, intra template matching (TMP), and matrix weighted intra prediction (MIP). DIMD extensions in ECM take place after the luma prediction block and the luma reconstructed block have already been formed. DIMD extensions take advantage of this information by using reconstructed samples in the luma block as reference samples, unlike in the original DIMD.

[0056] For blocks using MIP or intra TMP prediction, DIMD is used to derive the intra prediction mode of the current block based on the MIP or intra TMP predicted samples. For MIP, this is done before upsampling.

[0057] Three types of DIMD extension are currently supported in ECM. Both DIMD for MIP and for intra TMP employ the same locations of reference samples, as shown in FIG. 5 with the gray samples at 510. These samples are in the luma block and / or channel. It is noted that the outermost samples of the gray samples 510 are separated by one from the borders of the block 520.

[0058] Chroma DIMD uses both reference samples from the same color channel and from the collocated block in the luma channel. FIG. 6 shows locations of reference samples for chroma DIMD. The gray samples 610 show the luma reference samples. The gray samples 630 about block 620 show the chroma refence samples. The chroma block and the luma block may be collocated. If the video is in 420 format (i.e. luma channel has twice as many rows of pixels andtwice as many columns of pixels, compared to the chroma channel), the top left pixel coordinate of the collocated luma block is (2x, 2y), and the top left pixel coordinate of a chroma block is (x, y).

[0059] It is noted that the outermost samples of the gray samples 610 are separated by one from the top and left borders of the relevant block, and separated by at least one from the bottom and right borders of the relevant block. It is noted that the outermost gray samples 630 are separated by one from the borders of the relevant block.

[0060] DIMD mode derivation is a computationally intensive operation, as it involves gradient strength and prediction mode calculations for every reference sample. The current design is not well optimized in terms of complexity, as DIMD for MIP and intra TMP employ most samples in the block (all except those on the block boundaries) as reference samples. Additionally, current DIMD extensions employ different sets of reference samples for different modes (e.g. DIMD for MIP and intra TMP may use the set shown in FIG. 5, while DIMD for chroma may use the set shown in FIG. 6), resulting in multiple implementations of gradient derivation. These issues cause DIMD to require a lot of additional cycles to process.

[0061] It may be noted that the intra prediction mode derived from DIMD for intra TMP and / or MIP may be used to determine a Low Frequency Non-Separable Transform (LFNST) set in a transform process. It may be noted that the intra prediction mode derived with chroma DIMD may be used in a prediction process.

[0062] A technical effect of example embodiments of the present disclosure may be to keep a number of reference samples in DIMD low, for example without sacrificing too much coding efficiency.

[0063] In the present disclosure, the terms “sample” and “pixel” are used interchangeably.

[0064] In the present disclosure, the terms “block boundary”, “block edge”, and “block border” are used interchangeably.

[0065] In an example embodiment, only a certain set of reference pixels may be involved in the block, for example according to a predefined arrangement. In an example embodiment, a certain set of reference samples may be used, according to a predefined arrangement, to derive a histogram of gradients for DIMD extensions. Since the goal of DIMD is to find the edge pattern in the coding block, a sample arrangement may be configured such that only a small number of samples are used to derive the edge pattern.

[0066] Samples in the middle of the block are usually not as representative of the edge pattern compared to the samples close to block boundaries; using only samples in the middle of the block may miss an edge that occurs in the area close to the boundary. Hence, using all the samples inside the block may not be efficient, as an edge pattern detected in the middle part may often be detected in the area close to the boundary. As a result, coding efficiency may be mostly retained when building a histogram of gradients using only reference samples in the area close to the boundary, and skipping the reference samples in the middle of the block.

[0067] In an example embodiment, an arrangement of reference samples may be set to samples in the area close to the boundary. One example may use an arrangement that includes every reference sample that is one pixel away from the block boundaries. Referring now to FIG. 7, illustrated is an example of a square block with a width of N and a height of N, and samples with coordinates x = 0, ... , N-l and y = 0, ... , N-l . Accordingly, the following samples may be included as center samples for the gradient calculations:

[0068] Samples (x, l), where x = 1, ..., 1V — 2 are illustrated at 710. Samples (x, N — 2), where x = 1, ..., N — 2 are illustrated at 730. Samples (1, y), where y = 2, ... , N — 3 are illustrated at 740. Samples(N — 2, y), where y = 2, ... , 1V — 3 are illustrated at 720.

[0069] In pseudo code, the center reference sample selection and gradient calculation using a function calculateGradiantAt(x, y) to calculate gradient at position (x, y) inside the block may be performed, for example, as follows: for x = range(l, N-2) calculateGradiantAt(x, 1) calculateGradiantAt(x, N-2) for y = range(2, N-3) calculateGradiantAt(l, y) calculateGradiantAt(N-2, y)

[0070] It may be noted that the example of FIG. 7 is not limiting; different ranges of samples are possible.

[0071] Alternatively, a sparse sampling of the reference block may be used as a sample arrangement to include representative samples both on the boundary and in the inner areas of the block. This may be achieved, for example, by selecting reference samples symmetrically extending from the middle of the block towards the corners, and / or different edges, of the block. Referring now to FIG. 8, illustrated is an example of a square block with a width of N and a height of N and samples with coordinates x = 0, ... , N-l and y = 0, ... , N-l . Accordingly, the following samples may be included as center samples for the gradient calculations: t Samples for which x = y and x = 1, ... , N — 2(Samples for which x = N — 1 — y and y = 1, ... , N — 2

[0072] Samples for which x = y and x = 1, .... N — 2 are illustrated at 810. Samples for which x = N — 1 — y and y = 1, ..., N — 2 are illustrated at 820.

[0073] In pseudo code, the center reference sample selection and gradient calculation using a function calculateGradiantAt(x, y) to calculate gradient at position (x, y) inside the block may be performed, for example, as follows:for y = range(l, N-2) for x = range(l, N-2) if x == y or x == N-l-y calculateGradiantAt(x, y)

[0074] Or, with a single loop performing both x == y and x == N-l-y cases within the same iteration: for x = range(l, N-2) calculateGradiantAt(x, x) calculateGradiantAt(x, N-l-x)

[0075] It may be noted that the example of FIG. 8 is not limiting; different ranges of samples are possible.

[0076] Different approaches may also be combined together, which may have the technical effect of covering a wider range of reference samples, while still maintaining the sparse nature of the process. For example, both refence samples extending radially from the center of the block and reference samples with a determined constant distance from the borders of the block may be included in the gradient calculation (see, e.g., FIG. 13).

[0077] A sparse arrangement of reference samples refers to an arrangement of reference samples comprising fewer samples than all interior samples of a block. For example, an arrangement of reference samples may be sparse if it includes fewer reference samples than in an example such as FIG. 5, where all samples that are at least one pixel away from a border of the block are used as reference samples. In an example embodiment, an arrangement of samples may be sparse if it omits at least one sample that is determined to be duplicative for determining an edge pattern of the block.

[0078] In the present disclosure, an arrangement of reference samples may refer to at least one of: a set of reference samples, a layout of reference samples, a configuration of referencesamples, a patern of reference samples, a constellation of reference samples, and / or a combination of reference samples.

[0079] In the present disclosure, a predetermined and / or sparse arrangement of reference samples may be a minimal or minimum set of reference samples predetermined for use for an intra prediction mode (e.g. excluding or omitting any samples unnecessary or duplicative for determining the mode). A predetermined sparse arrangement may be used in DIMD to indicate the location of reference samples. This arrangement may be fixed per processing block dimension. DIMD may analyze reference samples to derive an edge pattern corresponding to one or more prediction directions; each direction may align with an angular prediction mode.

[0080] In an example embodiment, the predetermined sparse arrangement may be fixed, and may need to be able to work with any type of video content effectively (e.g. video content with high or low levels of variation). In an example embodiment, the predetermined sparse arrangement may include only a subset of internal samples.

[0081] In an example embodiment, the predetermined sparse arrangement may include all internal samples needed to accurately detect edge patern within a processing block. In an example embodiment, the predetermined sparse arrangement may exclude one or more samples that do not provide useful information for edge pattern detection / derivation / determination. For example, in a flat area where all samples have the same value, it may be unnecessary to include all samples in the predetermined sparse arrangement.

[0082] FIGs. 9-11 show three arrangements of reference sample (gray blocks) locations for DIMD extensions. These arrangements may set reference samples one row / column away from the block boundaries, due to the 3x3 gradient operator used in ECM. Additionally, the reference samples may he on only one row / column (i.e. the width / thickness of the set of samples is one). Additionally or alternatively, the reference samples may be more than one row / column away from the boundaries, and / or may be located in more than one row / column.

[0083] Additionally or alternatively, not every reference samples in a given arrangement may be used in gradient calculation. For example, in Figs. 9-11, all pixels in the selected row / columnmay be used as reference samples. However, it may also be possible to use a subset of the pixels in the row / column instead. For example, every odd pixel (1st, 3rd, 5th, ... ) may be used, or left n pixels and right m pixels in 1010 may be used, and top p pixels and bottom q pixels in 1020 may be used.

[0084] In an example embodiment, reference samples may not be located on the border or boundary of a block. In other words, an arrangement of reference samples may exclude any samples or pixels on the border or boundary of a block.

[0085] Referring now to FIG. 9, illustrated is an example of an arrangement specifying locations of reference samples for DIMD extensions. In this example, reference samples may be situated close to the top (910), left (940), bottom (930), and right (920) boundaries. It may be noted that the reference samples are situated one row / column away from the boundaries of the block.

[0086] Referring now to FIG. 10, illustrated is an example of an arrangement specifying locations of reference samples for DIMD extensions. In this example, reference samples may be situated close to the top (1010) and left (1020) boundaries. It may be noted that the reference samples are situated one row / column away from the boundaries of the block.

[0087] Referring now to FIG. 11, illustrated is an example of an arrangement specifying locations of reference samples for DIMD extensions. In this example, reference samples may be situated close to the bottom (1120) and right (1110) boundaries. It may be noted that the reference samples are situated one row / column away from the boundaries of the block.

[0088] Referring now to FIG. 12, illustrated is an example of an arrangement of reference samples, for DIMD extensions, selected symmetrically extending from the center of the block. It may be noted that the reference samples are situated at least one row / column away from the boundaries of the block, but may be situated more than one row / column away from the boundaries of the block.

[0089] Referring now to FIG. 13, illustrated is an example combining the approaches of FIGs. 9 and 12. Illustrated is an example of an arrangement specifying locations of reference samples for DIMD extensions. In this example, reference samples may be selected in symmetric fashion extending from the center of the block (light gray), including also additional samples around the block edges (dark gray).

[0090] In an example embodiment, a UE may be configured with one sparse arrangement of reference samples for a luma block, and another, different sparse arrangement of reference samples for a chroma block. For example, there may be two sparse arrangements in a case of chroma DIMD. For DIMD for intra TMP and MIP, only one luma arrangement may be needed.

[0091] Referring now to FIG. 14, illustrated is an example of an arrangement specifying locations of reference samples for DIMD extensions. In this example, reference samples may be situated along the top and left borders of the block. Reference samples along these borders may use samples from neighboring reconstructed block(s) in their gradient calculations.

[0092] Example embodiments of the present disclosure may be applicable in the context of artificial intelligence (Al) and / or machine learning (ML). For example, an AI / ML neural network (NN) or other model may be used to design an arrangement of reference samples in an adaptive manner, for example according to luma block reconstructed pixels.

[0093] FIG. 15 illustrates the potential steps of an example method 1500. The example method 1500 may include: determining one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples, 1510; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples, 1520. The example method 1500 may be performed, for example, with a decoder, a UE, a network entity, a base station, etc.

[0094] In accordance with one example embodiment, an apparatus may comprise: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: determine one or more reference samples for deriving atleast one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determine the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0095] The at least one intra prediction mode may comprise a mode for at least one of: chroma prediction, a first mode dependent transform according to intra template matching, a second mode dependent transform according to a matrix weighted intra prediction, a transform process, a prediction process.

[0096] The one or more reference samples may comprise one or more reconstructed luma samples.

[0097] The at least one predetermined sparse arrangement may omit at least one sample that is determined to be duplicative for determining an edge pattern of a block.

[0098] The at least one predetermined sparse arrangement may comprise fewer than a set of internal samples of a block.

[0099] The at least one predetermined sparse arrangement may omit at least one sample that is determined to have a same value as at least one other sample of a block.

[0100] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples that are one sample away from at least one block border.

[0101] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples that are a same number of samples away from at least one block border.

[0102] The at least one predetermined sparse arrangement may comprise an arrangement of: one or more first samples that are a first number of samples away from a first block border, and one or more second samples that are a second number of samples away from a second block border.

[0103] The at least one predetermined sparse arrangement may comprise an arrangement of: one or more first samples that are a first number of samples away from a first block border, one or more second samples that are a second number of samples away from a second block border, oneor more third samples that are a third number of samples away from a third block border, and one or more fourth samples that are a fourth number of samples away from a fourth block border.

[0104] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples extending symmetrically from a middle of a block towards at least one of a plurality of corners of the block.

[0105] The at least one predetermined sparse arrangement comprises an arrangement of one or more samples extending symmetrically from a middle of a block towards at least one of a plurality of borders of the block.

[0106] The at least one predetermined sparse arrangement may comprise an arrangement of: one or more first samples that are a same number of samples away from at least one block border, and one or more second samples extending symmetrically from a middle of a block towards at least one of a plurality of corners of the block.

[0107] The at least one predetermined sparse arrangement may comprise an arrangement of: one or more first samples that are a same number of samples away from at least one block border, and one or more second samples extending symmetrically from a middle of a block towards at least one of a plurality of borders of the block.

[0108] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples that are not on a border of a block.

[0109] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples that are on at least one block border.

[0110] The at least one predetermined sparse arrangement may comprise at least one first predetermined sparse arrangement of the one or more reference samples for a luma block, and at least one second predetermined sparse arrangement of reference samples for a chroma block.

[0111] In accordance with one aspect, an example method may be provided comprising: determining, with a user equipment, one or more reference samples for deriving at least one intraprediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0112] The at least one intra prediction mode may comprise a mode for at least one of: chroma prediction, a first mode dependent transform according to intra template matching, a second mode dependent transform according to a matrix weighted intra prediction, a transform process, or a prediction process.

[0113] The one or more reference samples may comprise one or more reconstructed luma samples.

[0114] The at least one predetermined sparse arrangement may omit at least one sample that is determined to be duplicative for determining an edge pattern of a block.

[0115] The at least one predetermined sparse arrangement may comprise fewer than a set of internal samples of a block.

[0116] The at least one predetermined sparse arrangement may omit at least one sample that is determined to have a same value as at least one other sample of a block.

[0117] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples that are one sample away from at least one block border.

[0118] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples that are a same number of samples away from at least one block border.

[0119] The at least one predetermined sparse arrangement may comprise an arrangement of: one or more first samples that are a first number of samples away from a first block border, and one or more second samples that are a second number of samples away from a second block border.

[0120] The at least one predetermined sparse arrangement may comprise an arrangement of: one or more first samples that are a first number of samples away from a first block border, one or more second samples that are a second number of samples away from a second block border, oneor more third samples that are a third number of samples away from a third block border, and one or more fourth samples that are a fourth number of samples away from a fourth block border.

[0121] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples extending symmetrically from a middle of a block towards at least one of a plurality of corners of the block.

[0122] The at least one predetermined sparse arrangement comprises an arrangement of one or more samples extending symmetrically from a middle of a block towards at least one of a plurality of borders of the block.

[0123] The at least one predetermined sparse arrangement may comprise an arrangement of: one or more first samples that are a same number of samples away from at least one block border, and one or more second samples extending symmetrically from a middle of a block towards at least one of a plurality of corners of the block.

[0124] The at least one predetermined sparse arrangement may comprise an arrangement of: one or more first samples that are a same number of samples away from at least one block border, and one or more second samples extending symmetrically from a middle of a block towards at least one of a plurality of borders of the block.

[0125] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples that are not on a border of a block.

[0126] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples that are on at least one block border.

[0127] The at least one predetermined sparse arrangement may comprise at least one first predetermined sparse arrangement of reference samples for a luma block, and at least one second predetermined sparse arrangement of the one or more reference samples for a chroma block.

[0128] In accordance with one example embodiment, an apparatus may comprise: circuitry configured to perform: determining, with a user equipment, one or more reference samples forderiving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and circuitry configured to perform: determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0129] In accordance with one example embodiment, an apparatus may comprise: processing circuitry; memory circuitry including computer program code, the memory circuitry and the computer program code configured to, with the processing circuitry, enable the apparatus to: determine one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determine the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0130] As used in this application, the term “circuitry” or “means” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry) and (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.” This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0131] In accordance with one example embodiment, an apparatus may comprise means for: determining one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0132] The at least one intra prediction mode may comprise a mode for at least one of: chroma prediction, a first mode dependent transform according to intra template matching, a second mode dependent transform according to a matrix weighted intra prediction, a transform process, or a prediction process.

[0133] The one or more reference samples may comprise one or more reconstructed luma samples.

[0134] The at least one predetermined sparse arrangement may omit at least one sample that is determined to be duplicative for determining an edge pattern of a block.

[0135] The at least one predetermined sparse arrangement may comprise fewer than a set of internal samples of a block.

[0136] The at least one predetermined sparse arrangement may omit at least one sample that is determined to have a same value as at least one other sample of a block.

[0137] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples that are one sample away from at least one block border.

[0138] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples that are a same number of samples away from at least one block border.

[0139] The at least one predetermined sparse arrangement may comprise an arrangement of: one or more first samples that are a first number of samples away from a first block border, and one or more second samples that are a second number of samples away from a second block border.

[0140] The at least one predetermined sparse arrangement may comprise an arrangement of: one or more first samples that are a first number of samples away from a first block border, one or more second samples that are a second number of samples away from a second block border, one or more third samples that are a third number of samples away from a third block border, and one or more fourth samples that are a fourth number of samples away from a fourth block border.

[0141] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples extending symmetrically from a middle of a block towards at least one of a plurality of corners of the block.

[0142] The at least one predetermined sparse arrangement comprises an arrangement of one or more samples extending symmetrically from a middle of a block towards at least one of a plurality of borders of the block.

[0143] The at least one predetermined sparse arrangement may comprise an arrangement of: one or more first samples that are a same number of samples away from at least one block border, and one or more second samples extending symmetrically from a middle of a block towards at least one of a plurality of corners of the block.

[0144] The at least one predetermined sparse arrangement may comprise an arrangement of: one or more first samples that are a same number of samples away from at least one block border, and one or more second samples extending symmetrically from a middle of a block towards at least one of a plurality of borders of the block.

[0145] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples that are not on a border of a block.

[0146] The at least one predetermined sparse arrangement may comprise an arrangement of one or more samples that are on at least one block border.

[0147] The at least one predetermined sparse arrangement may comprise at least one first predetermined sparse arrangement of reference samples for a luma block, and at least one second predetermined sparse arrangement of the one or more reference samples for a chroma block.

[0148] A processor, memory, and / or example algorithms (which may be encoded as instructions, program, or code) may be provided as example means for providing or causing performance of operation.

[0149] In accordance with one example embodiment, a non-transitory computer-readable medium comprising instructions stored thereon which, when executed with at least one processor, cause the at least one processor to: determine one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determine the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0150] In accordance with one example embodiment, a non-transitory computer-readable medium comprising program instructions stored thereon for performing at least the following: determining one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0151] In accordance with another example embodiment, a non-transitory program storage device readable by a machine may be provided, tangibly embodying instructions executable by the machine for performing operations, the operations comprising: determining one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0152] In accordance with another example embodiment, a non-transitory computer-readable medium comprising instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: determining one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement ofthe one or more reference samples; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0153] A computer implemented system comprising: at least one processor and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the system at least to perform: determining one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0154] A computer implemented system comprising: means for determining one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and means for determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

[0155] The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e. tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0156] It should be understood that the foregoing description is only illustrative. Various alternatives and modifications can be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modification and variances which fall within the scope of the appended claims.

Claims

CLAIMSWhat is claimed is:

1. An apparatus comprising: at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: determine one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determine the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

2. The apparatus of claim 1, wherein the at least one intra prediction mode comprises a mode for at least one of: chroma prediction, a first mode dependent transform according to intra template matching, a second mode dependent transform according to a matrix weighted intra prediction, a transform process, or a prediction process.

3. The apparatus of claim 1 or 2, wherein the one or more reference samples comprise one or more reconstructed luma samples.

4. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement omits at least one sample that is determined to be duplicative for determining an edge pattern of a block.

5. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement comprises fewer than a set of internal samples of a block.

6. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement omits at least one sample that is determined to have a same value as at least one other sample of a block.

7. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are one sample away from at least one block border.

8. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are a same number of samples away from at least one block border.

9. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a first number of samples away from a first block border, and one or more second samples that are a second number of samples away from a second block border.

10. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a first number of samples away from a first block border,one or more second samples that are a second number of samples away from a second block border, one or more third samples that are a third number of samples away from a third block border, and one or more fourth samples that are a fourth number of samples away from a fourth block border.

11. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples extending symmetrically from a middle of a block towards at least one of a plurality of corners of the block.

12. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples extending symmetrically from a middle of a block towards at least one of a plurality of borders of the block.

13. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a same number of samples away from at least one block border, and one or more second samples extending symmetrically from a middle of a block towards at least one of a plurality of corners of the block.

14. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a same number of samples away from at least one block border, andone or more second samples extending symmetrically from a middle of a block towards at least one of a plurality of borders of the block.

15. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are not on a border of a block.

16. The apparatus of any of claims 1 through 3, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are on at least one block border.

17. The apparatus of claim 1 or 2, wherein the at least one predetermined sparse arrangement comprises at least one first predetermined sparse arrangement of the one or more reference samples for a luma block, and at least one second predetermined sparse arrangement of reference samples for a chroma block.

18. A method comprising: determining, with a user equipment, one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

19. The method of claim 18, wherein the at least one intra prediction mode comprises a mode for at least one of: chroma prediction, a first mode dependent transform according to intra template matching, a second mode dependent transform according to a matrix weighted intra prediction,a transform process, or a prediction process.

20. The method of claim 18 or 19, wherein the one or more reference samples comprise one or more reconstructed luma samples.

21. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement omits at least one sample that is determined to be duplicative for determining an edge pattern of a block.

22. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement comprises fewer than a set of internal samples of a block.

23. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement omits at least one sample that is determined to have a same value as at least one other sample of a block.

24. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are one sample away from at least one block border.

25. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are a same number of samples away from at least one block border.

26. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a first number of samples away from a first block border, and one or more second samples that are a second number of samples away from a second block border.

27. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a first number of samples away from a first block border, one or more second samples that are a second number of samples away from a second block border, one or more third samples that are a third number of samples away from a third block border, and one or more fourth samples that are a fourth number of samples away from a fourth block border.

28. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples extending symmetrically from a middle of a block towards at least one of a plurality of corners of the block.

29. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples extending symmetrically from a middle of a block towards at least one of a plurality of borders of the block.

30. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a same number of samples away from at least one block border, and one or more second samples extending symmetrically from a middle of a block towards at least one of a plurality of corners of the block.

31. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a same number of samples away from at least one block border, and one or more second samples extending symmetrically from a middle of a block towards at least one of a plurality of borders of the block.

32. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are not on a border of a block.

33. The method of any of claims 18 through 20, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are on at least one block border.

34. The method of claim 18 or 19, wherein the at least one predetermined sparse arrangement comprises at least one first predetermined sparse arrangement of reference samples for a luma block, and at least one second predetermined sparse arrangement of the one or more reference samples for a chroma block.

35. An apparatus comprising means for: determining one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

36. The apparatus of claim 35, wherein the at least one intra prediction mode comprises a mode for at least one of:chroma prediction, a first mode dependent transform according to intra template matching, a second mode dependent transform according to a matrix weighted intra prediction, a transform process, or a prediction process.

37. The apparatus of claim 35 or 36, wherein the one or more reference samples comprise one or more reconstructed luma samples.

38. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement omits at least one sample that is determined to be duplicative for determining an edge pattern of a block.

39. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement comprises fewer than a set of internal samples of a block.

40. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement omits at least one sample that is determined to have a same value as at least one other sample of a block.

41. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are one sample away from at least one block border.

42. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are a same number of samples away from at least one block border.

43. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a first number of samples away from a first block border, and one or more second samples that are a second number of samples away from a second block border.

44. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a first number of samples away from a first block border, one or more second samples that are a second number of samples away from a second block border, one or more third samples that are a third number of samples away from a third block border, and one or more fourth samples that are a fourth number of samples away from a fourth block border.

45. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples extending symmetrically from a middle of a block towards at least one of a plurality of corners of the block.

46. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples extending symmetrically from a middle of a block towards at least one of a plurality of borders of the block.

47. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a same number of samples away from at least one block border, and one or more second samples extending symmetrically from a middle of a block towards at least one of a plurality of corners of the block.

48. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of: one or more first samples that are a same number of samples away from at least one block border, and one or more second samples extending symmetrically from a middle of a block towards at least one of a plurality of borders of the block.

49. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are not on a border of a block.

50. The apparatus of any of claims 35 through 37, wherein the at least one predetermined sparse arrangement comprises an arrangement of one or more samples that are on at least one block border.

51. The apparatus of claim 35 or 36, wherein the at least one predetermined sparse arrangement comprises at least one first predetermined sparse arrangement of reference samples for a luma block, and at least one second predetermined sparse arrangement of the one or more reference samples for a chroma block.

52. A non-transitory computer-readable medium comprising program instructions stored thereon for performing at least the following:determining one or more reference samples for deriving at least one intra prediction mode based, at least partially, on at least one predetermined sparse arrangement of the one or more reference samples; and determining the at least one intra prediction mode based, at least partially, on the one or more determined reference samples.

Images