Geometrical cross-component filtering
Patent Information
- Application Number
- JP2024057916
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-10-22
- Filing Date
- 2024-03-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-04-13
AI Technical Summary
The existing AV1 video coding format does not consider the directionality of samples in cross-component filtering, which affects its performance.
Applying geometric transformations to filter coefficients based on the orientation of samples to improve the efficiency of cross-component filtering by aligning the directionality of filter coefficients with the detected edges in video data.
Enhances the coding efficiency of video data by considering the orientation of samples during filtering, leading to improved encoding and decoding processes.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001] This application claims the benefit of priority to Provisional Patent Application No. 63 / 015,586, filed in the United States Patent and Trademark Office on April 26, 2020, and Provisional Patent Application No. 17 / 077,564, filed on October 22, 2020, the entire contents of both of which are incorporated herein by reference.
[0002] FIELD OF THE DISCLOSURE The present disclosure relates generally to the field of data processing, and more specifically, to video encoding and decoding. [Background technology]
[0003] AOMedia Video 1 (AV1) is an open video coding format designed for video transmission over the Internet. It was developed as a successor to VP9 by the Alliance for Open Media (AOMedia), a consortium that includes semiconductor companies, video-on-demand providers, video content producers, software development companies, and web browser vendors, founded in 2015. Many parts of the AV1 project grew out of earlier research by the consortium members. Individual contributors started experimental technology platforms several years ago. Xiph / Mozilla's Daala released code in 2010, Google's experimental VP9 evolution project VP10 was announced on September 12, 2014, and Cisco's Thor was released on August 11, 2015. Based on the VP9 code base, AV1h incorporates other technologies, some of which were developed in these experimental formats. The first version 0.1.0 of the AV1 reference codec was released on April 7, 2016. The consortium announced the release of the AV1 bitstream specification, and a reference encoder and decoder based on the software, on March 28, 2018. The specification's validated version 1.0.0 was released on June 25, 2018. The specification's validated version 1.0.0 with errata 1 was released on January 8, 2019. The AV1 bitstream specification includes a reference video codec. Summary of the Invention [Means for solving the problem]
[0004] Embodiments provide methods, systems, and computer-readable media for coding video data. According to one aspect, a method for coding video data is provided, the method including the steps of receiving video data, determining a directionality of sample blocks of the received video data, the directionality corresponding to a descriptor, such as an edge direction or a gradient, applying a geometric transformation based on the determined directionality, and decoding the video data based on the applied geometric transformation.
[0005] According to another aspect, a computer system for coding video data is provided, the computer system may include one or more processors, one or more computer readable memories, one or more computer readable tangible storage devices and program instructions stored in at least one of the one or more storage devices and executed by at least one of the one or more processors via at least one of the one or more memories to enable the computer system to perform a method including receiving video data, determining a directionality of sample blocks of the received video data, the directionality corresponding to a descriptor, such as an edge direction or a gradient, applying a geometric transformation based on the determined directionality, and decoding the video data based on the applied geometric transformation.
[0006] According to yet another aspect, a computer-readable medium for coding video data is provided. The computer-readable medium includes program instructions stored in at least one of one or more computer-readable storage devices and one or more tangible storage devices, the program instructions being executable by a processor. The program instructions are executable by the processor to perform a method, whereby the method includes receiving video data, determining a directionality of sample blocks of the received video data, the directionality corresponding to a descriptor, such as an edge direction or a gradient, applying a geometric transformation based on the determined directionality, and decoding the video data based on the applied geometric transformation. [Brief description of the drawings]
[0007] These and other objects, features and advantages will become apparent from the following detailed description of illustrative embodiments, which is to be read in conjunction with the accompanying drawings, in which various features are not drawn to scale, for clarity of illustration and to facilitate understanding by those skilled in the art in conjunction with the detailed description.
[0008] [Figure 1] 1 illustrates a networked computing environment in accordance with at least one embodiment. [Diagram 2] 1 is a diagram of an example edge direction of video data in accordance with at least one embodiment. [Figure 3A] 1 is an exemplary geometric transformation process according to at least one embodiment. [Figure 3B] 1 is an exemplary geometric transformation process according to at least one embodiment. [Figure 4] 1A-1C are diagrams of example sub-sampling positions of samples corresponding to one or more gradient directions according to at least one embodiment. [Diagram 5]1 is an operational flowchart illustrating steps performed by a program for coding video data based on applying a geometric transformation corresponding to recognized edges of the video data, in accordance with at least one embodiment. [Figure 6] FIG. 2 is a block diagram of an exemplary cloud computing environment including the computer system shown in FIG. 1 according to at least one embodiment. [Figure 7] FIG. 7 is a block diagram of functional layers of the example cloud computing environment of FIG. 6 in accordance with at least one embodiment. [Figure 8] FIG. 2 is a block diagram of internal and external components of the computer and server shown in FIG. 1 according to at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Detailed embodiments of the claimed structures and methods are disclosed herein. However, it should be understood that the disclosed embodiments are merely exemplary of the claimed structures and methods, which may be embodied in various forms. However, these structures and methods may be embodied in many different forms and should not be construed as being limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey its scope to those skilled in the art. The specification may omit details of well-known features and techniques to avoid unnecessarily obscuring the presented embodiments.
[0010] FIELD OF THE DISCLOSURE Embodiments relate generally to the field of data processing, and more specifically to video encoding and decoding. Additionally, exemplary embodiments described below provide systems, methods, and computer programs for encoding and decoding video data based on applying a geometric transformation to edges detected in the video data. Thus, some embodiments are capable of improving the field of computing by enabling improved edge detection in the video data, thereby improving the encoding and decoding of video data.
[0011] As mentioned above, AOMedia Video 1 (AV1) is an open video coding format designed for video transmission over the Internet. AV1 was developed as a successor to VP9 by the Alliance for Open Media (AOMedia), a consortium that includes semiconductor companies, video-on-demand providers, video content producers, software development companies, and web browser vendors, founded in 2015. Many parts of the AV1 project grew out of previous research by the consortium members. Individual contributors started experimental technology platforms several years ago. Xiph / Mozilla's Daala released code in 2010, Google's experimental VP9 evolution project VP10 was announced on September 12, 2014, and Cisco's Thor was released on August 11, 2015. Based on the VP9 code base, AV1h incorporates other technologies, some of which were developed in these experimental formats. The first version 0.1.0 of the AV1 reference codec was released on April 7, 2016. The consortium announced the release of the AV1 bitstream specification, and a software-based reference encoder and decoder on March 28, 2018. Validated version 1.0.0 of the specification was released on June 25, 2018. Validated version 1.0.0 of the specification with errata 1 was released on January 8, 2019. The AV1 bitstream specification includes a reference video codec.
[0012] In AV1, the cross-component filtering method is defined as a filtering process that uses a reconstructed sample of a first color component (e.g., Y or Cb or Cr) as an input, and the output is applied to a second color component as a color component different from the first color component. However, in cross-component filtering (CCF), the directionality of the samples is not considered, which may affect the performance of CCF. In ALF, a geometric transformation of the filter coefficients is applied to handle the different directionality. For CCF, a geometric transformation of the filter coefficients can also be applied, which may help improve the efficiency of coding. Therefore, it may be beneficial to consider the directionality of the samples when training the filter coefficients to perform filtering, determine the directionality of the samples, and apply a geometric transformation to the filter coefficients or samples in the filter support region to improve the efficiency of coding.
[0013] Aspects are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer-readable media according to various embodiments. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.
[0014] Referring now to Figure 1, a functional block diagram of a networked computing environment illustrates a video coding system 100 (hereinafter "system") that encodes and / or decodes video data based on applying a geometric transformation corresponding to recognized edges of the video data. It should be understood that Figure 1 is merely an illustration of one implementation and is not intended to imply any limitations to the environments in which different embodiments may be implemented. Many variations can be made to the depicted environment based on design and implementation requirements.
[0015] The system 100 may include a computer 102 and a server computer 114. The computer 102 may communicate with the server computer 114 via a communications network 110 (hereinafter, "network"). The computer 102 may include a processor 104 and a software program 108, which is stored in a data storage device 106 and enables user interaction and communication with the server computer 114. As described below with reference to FIG. 8, the computer 102 includes internal and external components 800A and 900A, respectively, and the server computer 114 includes internal and external components 800B and 900B, respectively. The computer 102 may be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of executing programs and accessing networks and databases.
[0016] 6 and 7, the server computer 114 may also operate in, for example, a Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS) cloud computing service model. The server computer 114 may reside in, for example, a private cloud, a community cloud, a public cloud, or a hybrid cloud cloud computing deployment model.
[0017] The server computer 114, which can be used to code the video data based on applying a geometric transformation corresponding to the recognized edges of the video data, can execute a video coding program 116 (hereinafter "program") that can interact with the database 112. The video coding program method is described in detail with respect to FIG. 5. In one embodiment, the computer 102 can act as an input device including a user interface, and the program 116 can execute primarily on the server computer 114. In an alternative embodiment, the program 116 can execute primarily on one or more computers 102, and the server computer 114 can process and store data used by the program 116. It is noted that the program 116 can be a stand-alone program or can be integrated into a larger video coding program.
[0018] However, in some embodiments, processing of the program 116 may be shared in any ratio between the computer 102 and the server computer 114. In other embodiments, the program 116 may execute on multiple computers, server computers, or some combination of computers and server computers, e.g., multiple computers 102 communicating with a single server computer 114 over the network 110. In other embodiments, for example, the program 116 may execute on multiple server computers 114, which communicate with multiple client computers over the network 110. Alternatively, the program may execute on a network server, which may communicate with the server and multiple client computers over the network.
[0019] The network 110 may include wired connections, wireless connections, fiber optic connections, or some combination thereof. In general, the network 110 may be any combination of connections and protocols that support communication between the computer 102 and the server computer 114. The network 110 may include various types of networks, such as a local area network (LAN), a wide area network (WAN) such as the Internet, a telecommunications network such as a public switched telephone network (PSTN), a wireless network, a public switched network, a satellite network, a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a metropolitan area network (MAN), a dedicated network, an ad-hoc network, an intranet, a fiber optic based network, etc., and / or a combination of these or other types of networks.
[0020] The number and arrangement of devices and networks shown in Figure 1 are shown as examples. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks in a different arrangement than those shown in Figure 1. Also, two or more devices shown in Figure 1 may be implemented within a single device, or a single device shown in Figure 1 may be implemented as multiple distributed devices. Additionally or alternatively, one group of devices (e.g., one or more devices) of system 100 may perform one or more functions performed by another group of devices of system 100.
[0021] Referring now to FIG. 2, a diagram of an exemplary edge direction 200 is shown. An edge may represent a set of samples with abrupt occurrences of intensity values at consecutive locations in the video and image data. An edge may have a direction that may represent a unit vector along the edge. An edge may have a location that may correspond to a sample location where the edge is located. Thus, edge detection may correspond to a process that can be used to determine an edge direction and an edge location. For example, a gradient along an edge may be determined to perform edge detection. The gradient may correspond to an increase or decrease in sample values along a direction that may be different from the edge direction. A negative gradient value may correspond to a sample intensity that changes to a smaller sample intensity along a particular direction, and a positive gradient value may correspond to a sample intensity that changes to a larger sample intensity along a particular direction. The edge direction 200 may include, for example, a horizontal edge 202, a vertical edge 204, a 45 degree edge 206, and a 135 degree edge 208. It should be understood that if the size of the sample block is large enough, edges of virtually any angle may exist.
[0022] According to one or more embodiments, the directionality of an M×N block can be determined, where M and N are positive integers. Example values of M and N can include, for example, 2, 4, and 8. The input of the process can be an M×N sample block, and the output can be a descriptor that can describe the directionality of the current block. In one embodiment, the edge direction is selected as the descriptor. An edge detection process can be applied to the current block, and the output of the process is the edge direction and edge location. In one embodiment, the edge direction is determined based on a search for the direction of the block with the smallest error value. The direction search works on pixels reconstructed only after the deblocking filter. These pixels are available at the decoder, so the direction does not require signaling. The search operates on 8×8 blocks, which are small enough to handle non-straight edges, but large enough to reliably estimate the direction when applied to a quantized image. Having a constant direction in the 8×8 domain also makes it easier to vectorize the filter. For each block, the direction that best matches the pattern in the block is determined by minimizing the sum of squared differences (SSD) between the quantized block and the closest perfect directional block. A perfect directional block is a block where all pixels along a line in one direction have the same value. In one embodiment, edge detection is performed using a Canny edge detector. In one embodiment, edge detection is performed using a Sobel edge detector.
[0023] In one embodiment, the gradient value is selected as the descriptor. First, a process is applied to the current block to calculate the gradient value along a particular direction. Exemplary gradient directions include, but are not limited to, horizontal, vertical, and diagonal (45 degrees, 135 degrees). In one embodiment, the method proposed above uses M×N blocks of input color components of the CCF to determine the directionality. In one example, if the input color component of the CCF is the Y component, the M×N block is obtained from the Y component. In one example, if the input color component of the CCF is the Cb component, the M×N block is obtained from the Cb component. In one example, if the input color component of the CCF is the Cr component, the M×N block is obtained from the Cr component.
[0024] In one embodiment, the method proposed above uses an M×N block of output color components of the CCF to determine the directionality. In one example, if the input color component of the CCF is a Y component, the M×N block is obtained from the Y component. In one example, if the input color component of the CCF is a Cb component, the M×N block is obtained from the Cb component. In one example, if the input color component of the CCF is a Cr component, the M×N block is obtained from the Cr component.
[0025] Now referring to FIG. 3A, an exemplary geometric transformation process 300A is shown. The geometric transformation process may also include one or more predefined rotations 302A to 302D. According to one or more embodiments, the geometric transformation process 300A can be applied to a filtering process of a CCF. The input to the process is the directionality of the samples in the above example described with reference to the embodiment related to FIG. 2. Based on the directionality, the filter coefficients or reconstructed samples in the filter support region of the CCF are transformed. In one embodiment, when recognizing the directionality of a block by an edge direction, the reconstructed samples in the filter coefficients or filter support region are rotated based on the recognized edge direction. In one embodiment, the target direction includes only a single direction. In one example, the target direction is horizontal. A rotation scheme is shown. If the edge direction of the current block is horizontal, no rotation is performed. If the edge direction of the current block is vertical, rotate the filter coefficients or samples in the filter support region clockwise by 90 degrees. If the edge direction of the current block is 45 degrees, rotate the filter coefficients or samples in the filter support region clockwise by 135 degrees. If the edge direction of the current block is 135 degrees, rotate the filter coefficients or samples in the filter support region 45 degrees clockwise. If the current block is a smooth region, i.e., no edge is detected, no rotation is performed.
[0026] Now referring to FIG. 3B, an exemplary geometric transformation process 300B is shown. The geometric transformation process may also include one or more predefined rotations 304A to 304B. The geometric transformation process 300B may include applying the same transformation to multiple directions. In one example, the target directions are horizontal and vertical. A rotation scheme is shown. If the edge direction of the current block is horizontal, do not perform rotation. If the edge direction of the current block is vertical, do not perform rotation. If the edge direction of the current block is 45 degrees, rotate the filter coefficients or samples in the filter support region 45 degrees clockwise. If the edge direction of the current block is 135 degrees, rotate the filter coefficients or samples in the filter support region 45 degrees clockwise. If the current block is a smooth region, i.e., if no edge is detected, do not perform rotation. In one embodiment, when the CCF scans all samples in the current block, it performs rotation regardless of whether the sample is located on an edge. When it is specified that the CCF scans a sample, it refers to either 1) that the sample has been trained, or 2) the filtering process of the CCF. In one embodiment, the rotation is performed only when the CCF scans a sample located on an edge in the current block. When it is specified that the CCF scans a sample, it means that the sample has been trained or filtered by the CCF.
[0027] In one embodiment, where the gradient value represents the directionality of a block, we transform the filter coefficients or the reconstructed samples in the filter's support region based on the gradient value.
[0028] In one embodiment, the transformation can be performed using a geometric transformation of filter coefficients and clipping values. An Self-Adaptive Loop Filter (ALF) with block-based filter adaptation can be applied. For the luma component, one of 25 filters is selected for each 4x4 block based on local gradient direction and activity. Two diamond filter shapes can be used for the self-adaptive loop filtering. A 7x7 diamond shape can be applied to the luma component and a 5x5 diamond shape can be applied to the chroma component.
[0029] For the luminance component, each 4x4 block can be classified into a class other than the 25 classes. The classification index C is the quantized value of its directionality D and activity.
number
number
number
[0030] To reduce the complexity of block classification, a subsampled 1-D Laplacian computation is applied. The same subsampled positions are used for gradient computation in all directions, as described below with respect to Figure 4. Then, the maximum and minimum values of D for the horizontal and vertical gradients can be set as follows:
number
number
[0031] These values are compared with each other and with two thresholds t1 and t2 to derive the value of the directionality D.
number
number
number
number
number
[0032] The activity value A can be calculated as follows:
number
number
[0033] Before filtering each 4×4 luminance block, geometric transformations such as rotation, diagonal flipping, and vertical flipping are applied to the filter coefficients f(k, l) and the corresponding filter clipping values c(k, l) based on the gradient values calculated for the block. This is equivalent to applying these transformations to the samples in the filter support region. This concept aligns the orientations of the different blocks to which ALF is applied, making these different blocks more similar.
[0034] Three geometric transformations including diagonal flipping, vertical flipping, and rotation can be introduced. Diagonal: f D (k, l) = f(l, k), c D (k, l) = c(l, k) Vertical flipping: f V (k, l) = f(k, K - l - 1), c V (k, l) = c(k, K - l - 1) Rotation: f R (k, l) = f(K - l - 1, k), c R (k, l) = c(K - l - 1, k) Here, K is the size of the filter, and 0 ≤ k, 1 ≤ K - 1 are the coefficient coordinates where the position (0, 0) is at the upper left corner and the position (K - 1, K - 1) is at the lower right corner. Based on the gradient values calculated for the block, the transformation is applied to the filter coefficients f(k, l) and the clipping values c(k, l). If the gradient value gd2 < gd1 and gh < gv, it may not be necessary to apply the transformation. If the gradient value gd2 < gd1 and gv < gh, the diagonal transformation can be applied. If the gradient value gd1 < gd2 and gh < gv, the vertical flipping transformation can be applied. If the gradient value gd1 < gd2 and gv < gh, the rotation transformation can be applied.
[0035] In one embodiment, the directionality of a block is measured by both the edge direction and the gradient, so that the transformation of the filter coefficients of the CCF depends on both the edge direction and the gradient. In one embodiment, the edge direction determines whether the filter coefficients are rotated. In one embodiment, the gradient determines whether the filter coefficients are flipped.
[0036] In one embodiment, a geometric transformation is applied during the training process of the CCF, where the input is the orientation of the samples in the example described above with reference to the embodiment associated with Fig. 2. Based on the orientation, the filter coefficients of the CCF or the reconstructed samples in the filter support region are transformed.
[0037] 4, a diagram 400 of example sub-sampling positions of samples corresponding to one or more gradient directions is shown. The sub-sampling positions may include a position corresponding to a vertical gradient of sample 302A, a position corresponding to a horizontal gradient of sample 302B, a position corresponding to a first diagonal gradient of sample 302C, and a position corresponding to a second diagonal gradient of sample 302D. Sub-sampling positions V, H, D1, and D2 correspond to the vertical gradient, horizontal gradient, first diagonal gradient, and second diagonal gradient, respectively.
[0038] Referring now to Figure 5, an operational flow diagram illustrating steps of a method 500 for coding video data is shown. In some implementations, one or more process blocks of Figure 5 may be performed by computer 102 (Figure 1) and server computer 114 (Figure 1). In some implementations, one or more process blocks of Figure 5 may be performed by another device or group of devices separate from or including computer 102 and server computer 114.
[0039] At 502, the method 500 includes receiving video data.
[0040] At 504, the method 500 includes determining an orientation of the sample blocks of the received video data, the orientation corresponding to the descriptor.
[0041] At 506, the method 500 includes applying a geometric transformation based on the determined orientation.
[0042] At 508, the method 500 includes decoding the video data based on the applied geometric transformation.
[0043] It can be appreciated that Figure 5 is only intended to provide example implementation examples, and is not intended to limit any limitations to the manner in which different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.
[0044] Although this disclosure includes detailed descriptions of cloud computing, it should be understood in advance that implementation of the teachings described herein is not limited to a cloud computing environment. Rather, some embodiments may be implemented in conjunction with other types of computing environments now known or later developed.
[0045] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of deployable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, services, etc.) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. The cloud model includes at least five characteristics, at least three service models, and at least four deployment models.
[0046] Its features are as follows: On-Demand Self-Service: Cloud consumers can unilaterally and automatically provide computing capacity, such as server time or network storage, as needed, without the need for human interaction with the service provider. Broad network access: Capabilities are available over the network and are accessed through standard mechanisms that facilitate use across heterogeneous thin-client and thick-client platforms (eg, mobile phones, laptop computers, and PDAs). Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where various physical and virtual resources are dynamically allocated and reallocated based on demand. Consumers typically have no control or knowledge of the exact location of the provided resources, giving them a sense of location independence, although they can specify location at a higher level of abstraction (e.g. country, state, data center, etc.). Rapid Elasticity: Provide rapid elasticity (sometimes automatically) to expand quickly, release quickly, and contract quickly. To the consumer, the capabilities available often appear unlimited and can be purchased at any time and in any quantity. Metering Services: Cloud systems automatically control and optimize resource usage by utilizing computing capabilities at a level of abstraction appropriate to the type of service (storage, processing, bandwidth, active user accounts, etc.). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being utilized.
[0047] The service model is as follows: Software as a Service (SaaS): The ability offered to the consumer is to use the provider's applications running on a cloud infrastructure. The applications can be accessed from a variety of client devices via a sync-client interface, such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even the respective application functionality, except for limited deployment of user-specific applications. Platform as a Service (PaaS): The capability offered to a consumer is to deploy consumer-created or acquired applications written using provider-supported programming languages and tools onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the placement of the deployed applications and possible application hosting environments. Infrastructure as a Service (IaaS): The capability offered to the consumer is to provide processing, storage, networking, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but may control the operating systems, storage, deployed applications, and may have limited control over the selection of networked components (e.g., host firewalls).
[0048] The deployment model is as follows: Private Cloud: The cloud infrastructure runs solely for the organization. It can be managed by the organization or a third party and can exist on-premise or off-premise. Community Cloud: Cloud infrastructure is shared across multiple organizations to support a specific community with common concerns (e.g. tasks, security requirements, policy and compliance considerations, etc.). It can be managed by the organization or a third party and can exist on-premise or off-premise. Public Cloud: The cloud infrastructure is available for use by the general public or large industry groups and is owned by organizations that sell cloud services. Hybrid Cloud: A cloud infrastructure is a combination of two or more clouds (private, community or public) that remain their own entities but are joined by standardized or proprietary technologies that allow portability of data and applications (e.g., cloud bursting for load balancing between clouds).
[0049] Cloud computing environments are service-oriented and their emphasis is on statelessness, low coupling, modularity and semantic interoperability.The core of cloud computing is an infrastructure that includes a network of interconnected nodes.
[0050] Referring to FIG. 6, an exemplary cloud computing environment 600 is shown. As shown in the figure, the cloud computing environment 600 includes one or more cloud computing nodes 10, through which local computing devices used by cloud consumers can communicate, such as, for example, a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automobile computer system 54N. The cloud computing nodes 10 can communicate with each other. They can be grouped into one or more networks, physically or virtually (not shown), such as, for example, a private cloud, a community cloud, a public cloud, or a hybrid cloud, or a combination thereof, as described above. This allows the cloud computing environment 600 to provide infrastructure as a service, platform as a service, and / or software as a service, without the cloud consumer having to maintain local computing device resources for the infrastructure as a service, platform as a service, and / or software as a service. It should be understood that the types of coding devices 54A to 54N shown in FIG. 6 are exemplary only, and that the cloud computing node 10 and cloud computing environment 600 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., a web browser).
[0051] Referring to Figure 7, a set of functional abstraction layers 700 provided by the cloud computing environment 600 (Figure 6) is shown. It should be understood in advance that the components, layers and functions shown in Figure 7 are for illustrative purposes only and the embodiments are not limited thereto. As shown, the following layers and corresponding functions are provided:
[0052] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include hosts 61, servers based on RISC (Reduced Instruction Set Computer) architecture 62, servers 63, blade servers 64, storage devices 65, and networks and network components 66. In some embodiments, software components include network application server software 67 and database software 68.
[0053] The virtualization layer 70 includes an abstraction layer from which virtual entities such as virtual servers 71, virtual storage devices 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75 can be provided.
[0054] In one example, the management layer 80 can provide the functionality described below. Resource provisioning 81 provides dynamic procurement of coding resources and other resources for executing tasks in the cloud coding environment. Metering and pricing 82 provides cost tracking when utilizing resources in the cloud computing environment and invoices or receipts for the consumption of these resources. In one example, these resources can include application software licenses. Security provides validation for cloud consumers and tasks and protection for data and other resources. User portal 83 provides consumers and system administrators access to the cloud computing environment. Service level management 84 provides allocation and management of cloud computing resources so that required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provides advance provisioning and procurement of cloud computing resources that forecast future demand based on SLAs.
[0055] The workload tier 90 provides examples of functionality that can utilize a cloud computing environment. Examples of workloads and functionality that can be provided from this tier include maps and navigation 91, software development and lifecycle management 92, virtual classroom delivery 93, data analytics processing 94, transaction processing 95, and video coding 96. Video coding 96 can apply geometric transformations that correspond to recognized edges in the video data.
[0056] Figure 8 is a block diagram of internal and external components of the computers 102, 114 shown in Figure 1 in accordance with an exemplary embodiment. It should be appreciated that Figure 8 is merely an illustration of one implementation and is not intended to imply any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.
[0057] The computer 102 (FIG. 1) and the server computer 114 (FIG. 1) may include corresponding sets of internal components 800A, 800B and external components 900A, 900B shown in FIG. 8. Each of the set of internal components 800 includes one or more processors 820 on one or more buses 826, one or more computer-readable RAMs 822 and one or more computer-readable ROMs 824, one or more operating systems 828, and one or more computer-readable tangible storage devices 830.
[0058] The processor 820 may be implemented in hardware, firmware, or a combination of hardware and software. The processor 820 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other type of processing component. In some implementations, the processor 820 includes a processor that is programmed to perform one or more of the functions. The bus 826 includes components that enable communication between the internal components 800A, 800B.
[0059] One or more operating systems 828, software programs 108 (FIG. 1), and video coding programs 116 (FIG. 1) on server computer 114 (FIG. 1) may be stored on one or more of corresponding computer-readable tangible storage devices 830 and executed by one or more of corresponding processors 820 through one or more of corresponding RAMs 822 (which typically include cache memory). In the embodiment shown in FIG. 8, each of computer-readable tangible storage devices 830 is a magnetic disk storage device driven by an internal hard disk. Alternatively, each of computer-readable tangible storage devices 830 is a semiconductor storage device such as ROM 824, EPROM, flash memory, optical disk, magneto-optical disk, solid-state disk, compact disk (CD), digital versatile disk (DVD), floppy disk, magnetic tape cassette, magnetic tape, and / or other types of non-transitory computer-readable tangible storage devices capable of storing computer programs and digital information.
[0060] Each set of internal components 800A, 800B may include a R / W drive or interface 832 for reading from and writing to one or more portable computer-readable tangible storage devices 936, such as, for example, CD-ROMs, DVDs, memory sticks, magnetic tapes, magnetic disks, optical disks, or semiconductor storage devices. Software programs, such as software program 108 (FIG. 1) and video coding program 116 (FIG. 1), may be stored in one or more of the corresponding portable computer-readable tangible storage devices 936 and may be read via the corresponding R / W drive or interface 832 and loaded into the corresponding hard disk driver 830.
[0061] Each set of internal components 800A, 800B includes a network adapter or interface 836, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, or a 3G, 4G, or 5G wireless interface card or other wired or wireless communication link. The software program 108 (FIG. 1) and the video coding program 116 (FIG. 1) on the server computer 114 (FIG. 1) are downloaded to the computer 102 (FIG. 1) and the server computer 114 from an external computer via a network (e.g., the Internet, a local area network, or other wide area network) and a corresponding network adapter or interface 836. The software program 108 and the video coding program 116 on the server computer 114 are loaded from the network adapter or interface 836 to a corresponding hard disk drive 830. The network may include copper wire, optical fiber, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers.
[0062] Each of the sets of external components 900A, 900B may include a computer display monitor 920, a keyboard 930, and a computer mouse 934. The external components 900A, 900B may include touch screens, virtual keyboards, touch pads, pointing devices, and other human interface devices. Each of the sets of internal components 800A, 800B includes a device drive 840 that interfaces with the computer display monitor 920, the keyboard 930, and the computer mouse 934. The device drive 840, the R / W drive or interface 832, and the network adapter or interface 836 include hardware and software (stored in the storage device 830 and / or the ROM 824).
[0063] Some embodiments relate to systems, methods and / or computer-readable media at any possible level of technical detail integration. The computer-readable media may include a computer-readable non-transitory storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform operations.
[0064] A computer-readable storage medium is a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disks read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in slots on which instructions are recorded, and any suitable combination of the above. A computer-readable storage medium as used herein should not be construed as a transient signal per se, but rather as, for example, radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable) or electrical signals transmitted through wires.
[0065] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to a corresponding computing / processing device or to an external computer or storage device, for example, via the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fiber optics, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium in the corresponding computing / processing device for storage.
[0066] The computer readable program code / instructions for performing the operations may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state installation data, configuration data for an integrated circuit system, or source or target code written in any combination of one or more programming languages, including object oriented programming languages such as Smalltalk, C++, and procedural programming languages similar to the "C" programming language or programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, an electronic circuit system, including, for example, a programmable logic circuit system, a field programmable gate array (FPGA), or a programmable logic array (PLA), can utilize state information of the computer readable program instructions to execute computer readable program instructions to personalize the electronic circuit system to perform various aspects or operations.
[0067] These computer readable program instructions may be provided to a general purpose computer processor, a special purpose computer processor, or a processor of other programmable data processing apparatus to create a mechanism such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / operations specified in the blocks of the flowcharts and / or block diagrams. These computer readable program instructions may be stored in a computer readable storage medium that can instruct a computer, programmable data processing apparatus, or other device in a particular manner to function such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture, which contains instructions for implementing various aspects of the functions / operations specified in the blocks of the flowcharts and / or block diagrams.
[0068] The computer readable program instructions can be loaded into a computer, other programmable data processing device, or other device such that a sequence of operational steps performed on the computer, other programmable device, or other device produces a computer-implemented process such that the instructions executed on the computer, other programmable device, or other device implement the functions / acts specified in the flowchart and / or block diagram blocks.
[0069] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer-readable media according to various embodiments. In this aspect, each block in the flowcharts or block diagrams may represent a module, segment, or part of an instruction, which includes one or more executable instructions for implementing a particular logical function. The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks relative to those blocks shown in the figures. In some alternative implementations, the functions shown in the blocks may be generated in the order shown in the figures. For example, two blocks shown in succession may actually be executed simultaneously or substantially simultaneously, or the blocks may be executed in the reverse order depending on the functionality involved. It should be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that executes a particular function or operation or realizes a combination of dedicated hardware and computer instructions.
[0070] It is apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the embodiments. Thus, the operation and behavior of the systems and / or methods are described herein without reference to any specific software code. It is understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.
[0071] No element, act, or instruction used herein should be construed as critical or essential unless expressly described as such. Additionally, the articles "a" and "an" as used herein are intended to include one or more items and can be used interchangeably with "one or more." Additionally, the term "set" as used herein is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and can be used interchangeably with "one or more." As used herein, terms such as "has," "have," "having," and the like are intended to be open-ended terms. Additionally, the phrase "based on" is intended to mean "based at least in part on," unless otherwise specified.
[0072] The description of various aspects and embodiments is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Although combinations of features are described in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features can be combined in ways not specifically described in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with each other claim in the set of claims. Numerous modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein. [Explanation of symbols]
[0073] 100 Video Coding System 102 Computer 104 processors 106 Data storage device 108 Software Programs 110 Communication Network 112 Database 114 Server Computer 116 Programs
Claims
1. 1. A processor-executable video coding method, comprising: receiving video data; determining an orientation of a sample block of received video data, determining a sum of squared differences between the sample block and each of a plurality of target orientation blocks; determining a direction of a target direction block having a minimum sum of squared differences among the plurality of target direction blocks as a direction of the sample block; Output a descriptor indicating the determined directionality of the sample block. determining a directionality of the sample block of the received video data by applying a geometric transformation to transform filter coefficients of the cross-component filter or reconstructed samples in the filter support region based on the determined directionality; decoding the video data based on the applied geometric transformation; The method includes:
2. if the orientation of the sample block is a first direction, rotating the filter coefficients or the reconstructed samples by a first amount; if the orientation of the sample block is in a second orientation different from the first orientation, rotating the filter coefficients or the reconstructed samples by a second amount; The method of claim 1 , wherein the first amount is different from the second amount.
3. if the orientation of the sample block is in a first direction, rotating the filter coefficients or the reconstructed samples by a first amount; if the orientation of the sample block is in a second orientation different from the first orientation, rotating the filter coefficients or the reconstructed samples by a second amount; The method of claim 1 , wherein the first amount is the same as the second amount.
4. The method of claim 1, wherein the descriptor indicates one or more of an edge direction and a gradient value.
5. The method of claim 4, further comprising, if the descriptor indicates an edge direction, determining the directionality based on the edge direction and the edge position.
6. The method of claim 4, wherein if the descriptor indicates a gradient value, the directionality is determined based on a gradient value along a direction corresponding to an edge of the video data.
7. The method of claim 1, further comprising determining the directionality based on input color components of the cross-component filter associated with a sample block of received video data.
8. The method of claim 1, further comprising determining the directionality based on output color components of the cross-component filter associated with a sample block of received video data.
9. The method of claim 1, further comprising recognizing the directionality of the sample block by edge direction.
10. A rotation is not performed based on the edge direction of the current block being horizontal, Rotating the filter coefficients or the reconstructed samples 90 degrees clockwise based on the edge direction of the current block being vertical; Rotating the filter coefficients or the reconstructed samples clockwise by 135 degrees based on the edge direction of the current block being 45 degrees; Rotating the filter coefficients or the reconstructed samples clockwise by 45 degrees based on the edge direction of the current block being 135 degrees; The method of claim 1 , further comprising: not performing rotation based on the current block being a smooth region where no edges are detected.
11. A rotation is not performed based on the edge direction of the current block being horizontal; Do not perform rotation based on the current block edge direction being vertical, Rotating the filter coefficients or the reconstructed samples clockwise by 45 degrees based on the edge direction of the current block being 45 degrees; Rotating the filter coefficients or the reconstructed samples clockwise by 45 degrees based on the edge direction of the current block being 135 degrees; The method of claim 1 , further comprising: not performing rotation based on the current block being a smooth region where no edges are detected.
12. The method of claim 1, wherein the cross-component filter performs rotation as it scans all samples in the current block.
13. The method of claim 1, wherein the rotation is performed when the cross-component filter scans only samples that lie on edges within the current block.
14. The method of claim 1, wherein the directionality of the sample block is recognized by a gradient value.
15. The method described in claim 14, wherein the directionality of the sample block is measured by both edge direction and gradient such that the geometric transformation of the filter coefficients of the cross component filter depends on both edge direction and gradient.
16. The method of claim 15, wherein the edge direction determines whether the filter coefficients are rotated and the gradient determines whether the filter coefficients are inverted.
17. The method of claim 1, wherein the geometric transformation is applied to a training process of the cross-component filter.
18. A computer system for a video codec, comprising: one or more computer readable non-transitory storage media arranged to store computer program code; one or more computer processors arranged to access said computer program code and to act according to the instructions of said computer program code; Including, The computer program code comprises: receiving code configured to cause the one or more computer processors to receive video data; decision code arranged to cause the one or more computer processors to determine an orientation of a sample block of received video data, the decision code comprising: determining a sum of squared differences between the sample block and each of a plurality of target orientation blocks; determining a direction of a target direction block having a minimum sum of square differences among the plurality of target direction blocks as a direction of the sample block; outputting a descriptor indicating the determined directionality of the sample block; a decision code arranged to cause a directionality of said sample block of said received video data to be determined by applying code configured to cause the one or more computer processors to apply a geometric transformation that transforms filter coefficients of a cross-component filter or reconstructed samples in a filter support region based on the determined directionality; decoding code arranged to cause the one or more computer processors to decode the video data based on the applied geometric transformation; 2. A computer system comprising:
19. A non-transitory computer-readable recording medium storing a computer program for a video codec, the computer program being configured to: receiving video data; determining an orientation of a sample block of received video data, determining a sum of squared differences between the sample block and each of a plurality of target orientation blocks; determining a direction of a target direction block having a minimum sum of squared differences among the plurality of target direction blocks as a direction of the sample block; Output a descriptor indicating the determined directionality of the sample block. determining a directionality of the sample block of the received video data by applying a geometric transformation to transform filter coefficients of the cross-component filter or reconstructed samples in the filter support region based on the determined directionality; decoding the video data based on the applied geometric transformation; A non-transitory computer-readable storage medium arranged to cause a program to execute the program.
