Crifford neural layer for multi-vector system modeling

Abstract

Apparatuses, systems, and methods for machine learning (ML) modeling of a system operating on multi-vector objects are generally discussed herein. A method includes receiving, by an ML model, a multi-vector object as an input, the multi-vector object representing a state of a multi-vector system. The method includes operating, by an ML model, a multi-vector input using a Crifford layer to generate a multi-vector output representing a state of a multi-vector system responsive to the multi-vector input, the Crifford layer including neurons implementing a multi-vector kernel.

Description

Background Art

[0001] Current machine learning (ML) models operate on standard vectors or input scalars. A vector consists of multiple scalar entries. Summary of the invention

[0002] Provided is a device, system, method and computer-readable medium, which are configured to implement an ML model that operates on multi-vector inputs. The ML model can implement a multi-vector valued neural network (NN) that operates on multi-vector inputs. Clifford algebra is used to define how to operate on multi-vector objects and between multi-vector objects. The output can be multi-vectors, classifications, vectors, etc. One or more of a Clifford convolutional layer, a Clifford Fourier transform layer and an equivariant Clifford layer can be used to operate on multi-vector inputs. Examples of systems using multi-vector modeling include mechanical engineering, civil engineering, geophysics, or meteorology using fluid dynamics (e.g., two-dimensional (2D) Navier-Stokes equations), electrical technology, optical technology, or radio technology that relies on solving three-dimensional (3D) Maxwell equations, and modeling of other general physical systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 A diagram of an embodiment of an ML system 100 for operating on a multi-vector object input 102 is shown by way of example.

[0004] Figure 2 Spatial primitives for multi-vector objects and diagrams of example multi-vector objects are shown by way of example.

[0005] Figure 3 A first vector field representing some physical phenomenon is shown.

[0006] Figure 4 By way of example, a vector field generated by decomposing the vector field into corresponding x and y Fourier transforms and then attempting to reconstruct the vector field from the Fourier transforms is shown.

[0007] Figure 5 A vector field generated by processing a vector field into a multi-vector object, performing a Clifford Fourier transform on the multi-vector object using a Clifford Fourier layer, and then reconstructing the vector field from the Clifford Fourier transform is shown by way of example.

[0008] Figure 6 A diagram showing an embodiment of a Clifford convolutional layer is shown by way of example.

[0009] Figure 7 A diagram showing an embodiment of a Clifford Fourier Neural Operator (CFNO) layer is shown by way of example.

[0010] Figure 8 Example rollouts of scalar and vector field components obtained by surrogate of the Clifford Fourier PDE are shown for qualitative comparison.

[0011] Fig. 9 Graphs of one-step loss and rolling loss for various NN PDE modeling architectures implementing the Navier-Stokes equations are shown by way of example.

[0012] Fig.10 The MSE losses for one-step and rolling are shown by way of example for various NN architectures implementing the Navier-Stokes equations.

[0013] Fig.11 By way of example, a plot of MSE versus the number of training trajectories for FNO and CFNO NN architectures for Maxwell equations simulation is shown.

[0014] Fig.12 A diagram of an embodiment of a method for modeling a system of multiple vector objects is shown by way of example.

[0015] Fig.13 is a block diagram of an example of an environment including a system for neural network training.

[0016] Fig.14 A block diagram of an embodiment of a machine (eg, a computer system) that implements one or more embodiments is shown by way of example. DETAILED DESCRIPTION

[0017] In the following description, reference is made to the accompanying drawings which form a part thereof, and specific embodiments that can be implemented are shown by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice these embodiments. It should be understood that other embodiments can be utilized, and structural, logical and / or electrical changes can be made without departing from the scope of the embodiments. Therefore, the following description of the embodiments should not be understood in a limiting sense, and the scope of the embodiments is defined by the appended claims.

[0018] Some systems are classically modeled using standard vectors and scalars. These systems can benefit from the modeling using multivector objects. The elements of Clifford algebra are called multivectors or multivector objects. Multivector objects contain elements of subspaces (e.g., scalars, vectors, bivectors, trivectors, etc.). Geometric product is the following operation, which combines multivector objects so that the result is another multivector object. Embodiments can use multivector kernels to define convolutions on multivector object inputs. Embodiments can use geometric products in Fourier space to define Clifford Fourier layers on multivector objects.

[0019] The value of a given element of a multi-vector object depends on one or more values ​​of other elements of the multi-vector object. No single current ML model takes into account such a variety of input elements or dependencies between elements. Embodiments operate using neural network (NN) elements that operate using Clifford algebra. This is in contrast to normal NN elements that operate on wx+b using an activation function, where w is a learned weight, x is a vector or scalar input value (but not a combination thereof), and b is a bias.

[0020] One application of the Clifford neural layer is modeling by solving partial differential equations (PDEs). PDEs have important uses in science and engineering. PDEs simulate physical processes as scalar and vector fields that interact and evolve together over time. Due to the computationally expensive nature of standard PDE solving methods, neural PDE substitution has become an active research topic to accelerate these simulations. However, current methods do not explicitly consider the relationship between different fields and their internal components (usually correlated). Modeling the time evolution of such related fields using multi-vector fields takes into account the relationship between different fields and their internal components. Multi-vector fields can include scalars, vectors, and higher-order components, such as bi-vectors and tri-vectors. The algebraic properties of multi-vector fields, such as multiplication, addition, and other arithmetic operations can be described by Clifford algebra. The embodiment presents the use of such multi-vector representations together with Clifford convolution and Clifford Fourier transform for the first time in the context of deep learning (DL). The resulting Clifford neural layer is generally applicable and can be used in the fields of fluid dynamics, weather forecasting, and general physical system modeling. We explore an empirical evaluation of the benefits of Clifford neural layers by replacing convolution and Fourier operations in common neural PDE alternatives with their Clifford counterparts. The PDE alternatives analyzed include two-dimensional Navier-Stokes and weather modeling tasks, as well as three-dimensional Maxwell equations. Clifford neural layers consistently improve the generalization capabilities of the tested neural PDE alternatives.

[0021] Most scientific phenomena are described by the evolution and interaction of physical quantities in space and time. The concept of field is a widely used structure for parameterizing these physical quantities continuously on selected coordinates. Prominent examples include (i) fluid mechanics, which has applications in fields ranging from mechanical and civil engineering to geophysics and meteorology, and (ii) electromagnetism, which provides mathematical models for electrical, optical or radio technology. The underlying equations of these examples are described in various forms of Navier-Stokes equations and Maxwell equations. For most of these equations, the solutions are analytically intractable, and obtaining accurate predictions requires falling back on numerical approximation schemes, which often come at prohibitive computational costs. The success of DL in overcoming the curse of dimensionality in many fields has led to a surge in interest in scientific applications, especially in using neural networks to enhance and replace numerical solution schemes in fluid dynamics.

[0022] Weather simulations are used to provide a basis for further description. In weather simulations, two different kinds of fields appear: scalar fields such as temperature or humidity, and vector fields such as wind speed or pressure gradient. Current DL-based methods treat different vector field components equally with scalar fields and stack all scalar fields along the channel dimension, thus ignoring the geometric relationships between different components within the vector field and between different components between individual vector fields and scalar fields. This practice misses important inductive bias information present in the input data. For example, wind speed is strongly correlated in the x-direction and the y-direction, making them form a vector field. In addition, the wind vector field and the scalar pressure field are related because the gradient of the pressure field causes air movement and subsequently affects the wind component. In this study, the following neural PDE alternatives are established, which not only model local and global relationships like current methods, but also model the relationship between different fields (e.g., wind and pressure fields) and the relationship between field components (e.g., the x-component and y-component of wind speed).

[0023]

[0024] Quaternion base elements .

[0025] Figure 1A diagram of an embodiment of an ML system 100 for operating on a multi-vector object input 102 is shown by way of example. The ML system 100 shown includes an ML model 104, which receives and operates on the multi-vector object input 102. The ML model 104 operates on the multi-vector object input 102 using one or more Clifford neural layers 108 to generate an output 106. The output 106 is a representation of the state of a system modeled by the ML model 104, the state of which responds to the stimulus represented by the multi-vector object input 102. The multi-vector object input 102 includes a combination of entries, which include two or more entries in scalars, vectors, double vectors, triple vectors, or other pseudo-scalars. That is, the entries of the multi-vector object input 102 include two or more entries in scalars, vectors, double vectors, triple vectors, or other higher dimensional vectors.

[0026] The ML model 104 models a system (e.g., a physical system) including a stimulus represented by a multi-vector object input 102. Examples of such physical systems include 3D electromagnetic fields, dynamic fluids, and weather forecasts, among others. The ML model 104 includes a Clifford neural layer 108 that operates on the input 102. The Clifford neural layer 108 may include a Clifford convolutional layer, a Clifford Fourier layer, a Clifford equivariant layer, or a combination thereof, such as a Clifford Fourier neural operator (FNO) that includes a Clifford convolutional layer and a Clifford Fourier layer. The Clifford neural layer 108 operates on the multi-vector input 102 using Clifford algebra. The output 106 is based on the result of the operation performed by the Clifford neural layer 108. More details about each of these layers can be found in the Appendix.

[0027] The output 106 may include scalars, categories, multi-vector objects, vectors, combinations thereof, etc. The output 106 generally represents the state of the system modeled by the ML model 104 in response to the input 102 .

[0028] Clifford algebras are at the core intersection of geometry and algebra, being introduced to simplify the spatial and geometric relationships between many mathematical concepts. For example, Clifford algebras naturally unify real numbers, vectors, complex numbers, quaternions, exterior algebra, and more. Most notably, in contrast to standard vector analysis where the primitives are scalars and vectors, Clifford algebras have additional spatial primitives for representing planes and volume segments. An illustrative example is the cross product of two vectors in 3 dimensions, which naturally transforms into the plane segment spanned by the two vectors. This cross product is often represented as a vector because it has three independent components, but under reflection it flips signs, whereas a real vector does not. In Clifford algebra, different spatial primitives can be generalized into objects called multivectors.

[0029] Figure 2 The spatial primitives of a multi-vector object and a diagram of an example multi-vector object are shown by way of example. Figure 2 As shown, space primitives include scalars (which are one-dimensional primitives), vectors (which are points with directions), bivectors (which are combinations of two vectors), and trivectors (which are combinations of three vectors). There are higher dimensional primitives. A vector is a multivector of order 1, a bivector is a multivector of order 2, and so on, and there are multivectors of order 4, and so on. A multivector includes a corresponding entry corresponding to each of the scalars, vectors, bivectors, and trivectors. Note that a given entry may be zero if the corresponding primitive does not exist. Multivector

[0030] In an embodiment, operations on feature fields in the DL architecture are replaced by their Clifford algebraic counterparts, which operate on multi-vector feature fields. For example, a convolution kernel is assigned multi-vector components so that the kernel can perform convolution on a multi-vector feature map. The introduction of the Clifford neural layer produces at least two advantages: (i) scalar components and vector components can be naturally grouped together into a single multi-vector object using operations on multi-vectors and mappings between multi-vectors defined by Clifford algebra; and (ii) higher-order geometric information such as bivectors and trivectors emerges, which match the above-mentioned higher-order objects. For example, the curl of a vector field can be described via bivectors, and therefore it is beneficial to have bivector filter components that match those bivectors.

[0031] More specifically, embodiments define Clifford convolution and Clifford Fourier transform on multi-vector fields, establishing connections between Clifford algebras, complex numbers, and quaternions. Based on these definitions, Clifford layers are implemented for different Clifford algebras to replace convolution and Fourier operations in DL architectures. An evaluation of the impact of using Clifford-based architectures on 2-dimensional problems of fluid dynamics and weather modeling, as well as on 3-dimensional Maxwell equations, is provided to demonstrate that Clifford neural layers consistently improve the generalization capabilities of the tested architectures.

[0032] Figures 3 to 5 Corresponding vector fields are shown by way of example to help explain the difference between existing field processing and field processing using multi-vector objects. Figure 3 A first vector field 300 representing some physical phenomenon is shown. The vector field 300 includes a number of sinks and sources. Sinks are points where vectors tend to converge (points where vectors point to). Sinks are considered stable points. Sources are points where vectors tend to point away from or "originate from". Sources are considered unstable points. Figure 3 In , the batch size is 1, there are two channels (x and y), and there are 32 samples for each of x and y in the batch.

[0033] Figure 4 By way of example, vector field 400 is shown which is generated by decomposing vector field 300 into its corresponding x and y Fourier transforms and then attempting to reconstruct vector field 300 from the Fourier transforms. As can be seen, most, if not all, of the point source and sink structure from vector field 300 has been lost in vector field 400. Figure 4 In , the batch size is 1, there are two channels (x and y), and there are 32 samples for each of x and y in the batch.

[0034] Figure 5 By way of example, vector field 500 is shown, which is generated by processing vector field 300 into a multi-vector object, performing a Clifford Fourier transform on the multi-vector object using a Clifford Fourier layer, and then reconstructing vector field 300 from the Clifford Fourier transform. As can be seen, most, if not all, of the point source and sink structure from vector field 300 is preserved in vector field 500. Figure 5 In , the batch size is 1, there is a single complex-valued channel, and there are 32 samples per entry in the single channel in the batch. The Fourier transform is performed on a single channel rather than on multiple channels as is usually done.

[0035] Channel reduction is one way in which the Clifford neural layer of an embodiment provides improvement. Compared to decomposing the problem into more channels, operating on these channels separately, and then recombining the outputs of the operations on each channel separately, channel reduction simplifies processing and preserves more structure.

[0036] The reason that vector field 400 does not preserve structure is because the x and y components are processed independently and the correlation between the x and y components is ignored or assumed to be negligible. However, in vector field 500, by treating the vector field as a multi-vector object, the correlation between the x and y components is preserved.

[0037] The following is a description of the Clifford algebra that forms the basis for the treatment of vector fields as multivector objects.

[0038] We introduce important mathematical concepts and discuss three prominent Clifford algebras, Cl 2,0 (R),Cl 0,2 (R),Cl 3,0 (R), which are directly related to the Clifford neural layer. A more detailed introduction and connections to complex numbers and quaternions are provided elsewhere. Consider a vector space R with the standard Euclidean product 〈.,.〉 n , where n = p + q, and p and q are non-negative integers. Real Clifford algebra Cl p,q (R) is generated by the vector space Rn The p+q standard orthogonal basis elements e1,…,e p+q The resulting associative algebra makes the following quadratic relations true:

[0039] For 1≤i≤p, e 2 =+1(1)

[0040] For p<j≤p+q, e 2 =-1(2)

[0041] For i≠j,e i e j =-e j e i (3)

[0042] The pair (p,q) is called the signature and is related to Cl p,q The vector space G of (R) p+q The basis elements of (R) together define the Clifford algebra Cl p,q (R). A vector space of a Clifford algebra has scalar elements and vector elements, but can also have vector spaces generated by R. n Elements of multiple basis elements (which can be interpreted as plane and volume segments). Exemplary low-level Wickliford algebras are: (i) Cl 0,0 (R), which is a one-dimensional algebra spanned by the basis elements {1}, and is therefore isomorphic to R (the field of real numbers); (ii) Cl 0,1 (R), whose two-dimensional algebra has a vector space G spanned by {1, e1} 1 , where the square of the basis vector e1 is 1, and is therefore isomorphic to C (the field of complex numbers); (iii) Cl 0,2 (R), which is a 4-dimensional algebra with a vector space G spanned by {1,e1,e2,e1e2} 2 , where the squares of {e1,e2,e1e2} are all 1 and anti-commute. Therefore, Cl 0,2 (R) is isomorphic to the quaternion H.

[0043] The grade of a Clifford algebra is the dimension of the subspace it represents. For example, the Clifford algebra Cl 2,0 The vector space G of (R) 2The basis elements {1,e1,e2,e1e2} of Clifford algebra have order {0,1,1,2}. Using the concept of order, the Clifford algebra can be divided into linear subspaces consisting of elements of each order. The order subspace with the smallest dimension is M0, which is the subspace of all scalars (elements of the generated vector space with 0 basis vectors). The elements of M1 are called vectors, the elements of M2 are bivectors, and so on. In general, the Clifford algebra Cl p,q The vector space G of (R) p+q The direct sum of all these subspaces can be written as: G p+q =M0⊕M1⊕…⊕M p+q The elements of a Clifford algebra are called multivectors, which contain elements of the subspace, i.e., scalars, vectors, bivectors, ..., k-vectors. The basis elements with the highest order are called pseudoscalars, which are 2 corresponds to the double vector e1e2 in R 3 Corresponding to the three vectors e1e2e3.

[0044] The dual a of a multivector * is defined as a * =ai p+q , where i p+q represents the corresponding pseudoscalar of the Clifford algebra. This definition allows different multivectors to be related to each other, which is a useful property when defining the Clifford Fourier transform. For example, for R 2 In the Clifford algebra, the dual of a scalar is a bivector, and in R 3 In this case, the dual of a scalar is a three-vector.

[0045] The geometric product is a bilinear operation on multi-vectors. For any multi-vector a, b, c ∈ G p+q , and the scalar λ, the geometric product has the following properties: (i) closed, that is, abG p+q , (ii) associativity, i.e., (c=a); (iii) scalar multiplication commutativity, i.e., λa=aλ; (iv) additive distributiveness, i.e., a=ab+ac. The geometric product is generally non-commutative, i.e., ab≠ba. Note that Equations 1, 2, and 3 specifically describe the geometric product between the basis elements of the generated vector space.

[0046] Now let's discuss Clifford algebra Cl 2,0 (R) and Cl 0,2 (R). The 4-dimensional vector space of these Clifford algebras has basis vectors {1,e1,e2,e1e2}, where the squares of e1,e2,e1e2 are 2,0 (R) is +1, for Cl 0,2 (R) is -1. 2,0(R), two multivectors a=a0+a1e1+a2e2+a 12 e1e e and b = b0 + b1e1 + b2e2 + b 12 e1e e The geometric product of is given by:

[0047] ab=(a0b0+a1b1+a2b2-a 12 b 12 )1+(a0b1+a1b0-a2b 12 +a 12 b2)e1+(a0b2+a1b 12 +α2b0-a 12 b1)e2+(a0b 12 +a1b2-a2b1+a 12 b0)e1e2(4)

[0048] It can be derived by collecting terms multiplied by the same basis elements. A vector x = (x1, x2) ∈ R with a standard Euclidean product <.,.< 2 Can be used with Correlation. Two vectors x, The Clifford multiplication yields the geometric product xy:

[0049]

[0050] where Λ is the outer product or wedge product. The asymmetry xΛy = yΛx is associated with a bivector, which can be interpreted as an oriented plane segment.

[0051] The unit bivector i2 formed by the (orthonormal) basis vectors e1 and e2 is determined by the following product:

[0052]

[0053] If this is squared, it yields So i2 represents the geometry From Formula 6, it can be concluded that e2=e1i2=-i2e1 and e1=i2e2=-e2i2.

[0054] Using the pseudoscalar i2, the dual of a scalar is a bivector and the dual of a vector is also a vector. The dual pair of basis vectors is and These dual pairs allow arbitrary multivectors to be represented as

[0055]

[0056] It can be considered as two complex-valued parts: a spinor part, which can be interchanged with the basis element 1, i.e., 1i2=i21, and a vector part, which can be anti-interchangeable with the corresponding basis element e1, i.e., e1i2=e1e1e2=-e1e2e1=-i2e1. This decomposition will be the basis of the Clifford Fourier transform layer, which naturally acts on dual pairs.

[0057] Clifford Algebra Cl 0,2 (R) is isomorphic to the quaternion H, which is an extension of the complex numbers and is usually written in the literature as Quaternions are also formed by Zhang Cheng's 4-dimensional algebra, where The squares are all -1. Since the squares of e1, e2, e1e2 are all -1 and are anti-commutative, it is easy to verify that Cl 0,2 (R) algebraic isomorphism. The basis element 1 is often called the scalar part, and the basis element The vector part is called a quaternion. Quaternions have practical uses in applied mathematics, especially for calculations involving three-dimensional rotations, which is used to define the rotational Clifford convolution layer elsewhere.

[0058] Now let's discuss Clifford algebra Cl 3,0 (R). Clifford Algebra Cl 3,0 The 8-dimensional vector space G of (R) 3 It has basis vectors {1,e1,e2,e3,e1e2,e3e1,e2e3,e1e2e3}, that is, it consists of a scalar, three vectors {e1,e2,e3}, three double vectors {e1e2,e3e1,e2e3}, and a three-vector e1e2e3. This three-dimensional vector is an algebraic pseudo-scalar i3. The definition of the geometric product of two multivectors is similar to Cl 2,0 (R) is the geometric product definition. 3,0 The dual pair of (R) is: and

[0059] When the expression of the electromagnetic field F is written in the form of the electric vector field E and the magnetic vector field B so that F = E + Bi3, Cl 3,0 An interesting example of the duality of the multivector of (R) arises, where E = E x e1+E y e2+E z e3 and B=B x e1+B y e2+B z In this way, the electromagnetic field F is decomposed into an electric vector part and a magnetic bivector part via the pseudo-scalar i3. For example, for the basis component B of B xe1, with B x e1i3=B x e1e1e2e3=B x e2e3, which is a double vector and is related to the basis component E of E x e1 is dual. Therefore, the multivector representing F consists of three vectors (the electric field components) and three bivectors (the magnetic field components multiplied by i3). This view gives the Clifford neural layer a natural advantage over its scalar DL counterpart.

[0060] We now describe the Clifford Convolutional Layer and the Clifford Fourier Transform Layer for a 2D problem. Extensions to 3 dimensions and a discussion of appropriate normalization and initialization procedures are provided elsewhere.

[0061] From regular CNN layer to Clifford CNN layer. Regular convolutional neural network (CNN) layers use feature maps as input and compare them with a set of c out Filters Perform convolution, where

[0062]

[0063] Equation 8 can be interpreted as 2 At , the inner product of the input feature map and the corresponding filter is calculated. By applying c out The filter output feature map can be interpreted as 2 c out dimensional feature vector. We hope to expand the convolutional layer so that the element-wise product of the scalar f j (y)w i,j (yx) is the geometric product of the multi-vector input and the multi-vector filter f j (y)w i,j (yx) is substituted. Therefore, the feature map From the multi-vector feature map f: Replace and with a group c out Multi-vector filter convolution:

[0064]

[0065] Note that by i∈{1,...,c out} and j∈{1,...,c in Each (geometric) product of the indices now produces a new multivector instead of a scalar. So, the output of the layer is c outThe 2D Clifford CNN layer can be implemented using Equation 4, where corresponds to 4 different kernels representing a 2D multi-vector kernel, i.e. 4 different convolutional layers, and Corresponding to the scalar part, vector part and double vector part of the input multi-vector field. The channels of different layers represent different stacks of scalars, vectors and double vectors. Similarly, the 3D Clifford CNN layer can be implemented using Equation 45. Figure 6 A schematic diagram of the Clifford convolutional layer is shown in FIG.

[0066] Figure 6 A diagram of an embodiment of a Clifford neural layer 600 is shown by way of example. The Clifford neural layer 600 convolves a multi-vector input 660 with a multi-vector kernel 662. The convolution may be performed using a geometric product 664. The result is an output including a multi-vector output field 666. The geometric product 664 is described elsewhere. Figure 6 In a standard convolution implementation consistent with the convolution of , there will be 8 separate channels and 32 samples per channel. Figure 6 In contrast, there are only two channels, each with four entry inputs (eg, the scalar portion, the vector portion, the double vector portion, etc. of a multivector). The geometric product multiplies the multivector kernel by the entries, preserving dependencies on the entries of the multivector inputs.

[0067] The following is pseudocode for a 2D Clifford kernel and a Clifford convolutional layer:

[0068]

[0069] The alternative parameterization of the Clifford CNN layer introduced in Equation 9 is called the rotated Clifford CNN layer and can be obtained by using the Clifford algebra Cl 0,2 (R) is isomorphic to quaternions. Note that quaternion rotation can be implemented through matrix multiplication. Using this isomorphism, the feature map f j and filter w i,j Represented as a quaternion: Now we can design the feature map f j With w i,j More precisely, consider the composite operation that produces a scalar quantity and a quaternion rotation, where the quaternion rotation acts on the quaternion f j The vector portion of the quaternion is rotated w and only produces a non-zero expansion factor for the vector portion of the quaternion output. i,j f j (w i,j ) -1 Acting on f jThe vector part of and can be manipulated algebraically into vector-matrix operations R i,j f j , where R i,j :H→H is from w i,j In other words, it can be established by i,j Scalar and vector parts of ∈H Constructed rotation matrix R i,j , transform f j The vector part of ∈H In summary, the rotating multi-vector filter By acting on the multi-vector feature map f:Z 2 (G 2 ) c Rotation transformation of the vector and double vector parts of in and the additional scalar response of the multi-vector filter to the feature map f j Action:

[0070]

[0071] in is the scalar output of Equation 4.

[0072] The rotational multi-vector filter R is outlined elsewhere. i,j Detailed description of (yx). Although in principle the Clifford CNN layer in Equation 9 and the rotated Clifford CNN layer in Equation 10 are equally flexible, the experiments described below show that the rotated Clifford CNN layer produces better performance.

[0073] Here is the pseudocode for a 2D rotated Clifford kernel and a rotated Clifford convolutional layer:

[0074] layer:

[0075]

[0076]

[0077] Clifford convolution satisfies equivariance under translation of multi-vector inputs. Similar to the theorem presented below, translation equivariance can also be shown for rotation of Clifford CNN layers. However, the current definition of Clifford convolution is not equivariant under multi-vector rotation or reflection. A general kernel constraint for a multi-vector kernel is provided below, which depicts a path to a general Clifford convolution that is equivariant with respect to rotation or reflection.

[0078] From Fourier Neural Operator (FNO) to Clifford Fourier Layer. In any dimension n, the n-dimensional complex signal f(x) = f(x1,...,xn ):R n →C in M1×...×M n The grid points are defined as:

[0079]

[0080] Among them, (ξ1,...,ξ n )∈Z M 1...×...Z M In the Fourier Neural Operator (FNO), the discrete Fourier transform on a real-valued input field and the corresponding inverse transform (implemented as a fast Fourier transform (RFFT) on real-valued input) are combined with a shape c in ×c out The weights of the complex weight matrix for each mode are multiplied and interleaved, which produces the form A complex-valued weight tensor of , where the cutoff frequency is exceeded The Fourier modes of are set to zero. In addition, residual connections are applied, which are usually implemented as convolutional layers with kernel size 1.

[0081] Figure 7 A diagram of an embodiment of a Clifford Fourier Neural Operator (CFNO) 700 is shown by way of example. In the CFNO 700, a real-valued Fast Fourier Transform (RFFT) on a real-valued scalar input field f(x) is replaced by a complex Fast Fourier Transform (FFT) on the complex-valued dual parts v(x) 770 and s(x) 772 of a multi-vector field f(x) 774. Point-wise multiplication in Fourier space via a complex weight tensor W is replaced by a geometric product 778 in Clifford Fourier space via a multi-vector weight tensor W 776. In addition, the convolution path is replaced by a Clifford convolution using a multi-vector kernel 780w.

[0082] For Figure 7 In a standard Fourier transform implementation consistent with the convolution of , there will be 8 separate channels and each channel will have 32×32 samples. Therefore, 8 separate Fourier transforms will be performed and the results will be combined. Figure 7 In contrast, there are only two channels, each with four entry inputs (eg, the scalar portion, the vector portion, the bi-vector portion of a multi-vector, etc.). Four Fourier transforms are performed, two for each multi-vector input.

[0083] For multi-vector valued functions f(x):R 2 →G 2 and the vector x,ξ∈R 2 The 2-Wickliford Fourier transform and the corresponding inverse transform are defined as:

[0084]

[0085] The condition is that the integral exists. In contrast to the standard Fourier transform, f(x)774 and denote the multi-vector field in the spatial domain and the frequency domain, respectively. In addition, i2=e1e2 is used in the index. Inserting the definition of the multi-vector field, Equation 12 becomes:

[0086]

[0087] The Clifford Fourier transform is obtained by applying two standard Fourier transforms to these dual pairs f0 = f0(x) + f 12 (x)i2 and f1=f1(x)+f2(x)i2, both of which can be regarded as complex-valued signals f0,f1:R 2 →C. Therefore, f(x)774 can be understood as C 2 The 2D Clifford Fourier transform is a linear combination of two classical Fourier transforms. Similar to Equation 11, the discrete version of Equation 14 is obtained. Similar to FNO, the multi-vector weight tensor 776 is applied where again, exceeding the cutoff frequency The Fourier modes of are set to zero. Therefore, the Clifford Fourier modes are modified pointwise via geometric products as follows:

[0088]

[0089] When the spin and vector parts of Eq. 14 are combined, the Clifford Fourier model is obtained. Finally, the residual connection of the previous FNO is replaced by the Clifford convolution using the multi-vector kernel 780k, as shown by Cl 2,0 (R) Convolutional layer 600 is replaced. Figure 7 A schematic diagram of the Clifford Fourier layer is shown in . The 3D Clifford Fourier transform follows a similar elegant construction, where the four separate Fourier transforms of the complex field are as in Eq. 16:

[0090] f0(x)=f0(x)+f 123 (x)i3,f1(x)=f1(x)+f 23 (x)i3

[0091] f2(x)=f2(x)+f 31 (x)i3,f3(x)=f3(x)+f 12 (x)i3

[0092] In equation 16, the scalar / tri-vector and vector / bi-vector components are combined into a complex field and then Fourier transformed. More details about 2D and 3D Fourier transforms and their characteristics can be found elsewhere. The spatial representation of v(x) 770 and s(x) 772 is recovered by an inverse FFT, which provides v*(x) 782 and s*(x) 784 that are combined to form f*(x) 786. The result 788 of the Clifford convolution is combined with f*(x) 786 to form the output 790 of the Clifford FNO 700.

[0093] The following is pseudocode for implementing the Clifford spectral convolution and the Clifford Fourier layer:

[0094]

[0095] An evaluation of Clifford neural layers is performed on different architectures in three experimental settings: the incompressible Navier-Stokes equations, the shallow water equations for weather modeling, and the 3D Maxwell equations in media. Ablation experiments are also performed on different dataset sizes. All datasets share the common feature of containing multiple input fields and output fields. More precisely, a scalar and a 2D vector field in the case of Navier-Stokes and shallow water equations, and a 3D (electric) vector field and its dual (magnetic) bivector field in the case of Maxwell equations. The ResNet and Fourier Neural Operator (FNO) based architectures are replaced by their corresponding Clifford algebra neural layer counterparts. In doing so, each convolution and each Fourier transform is changed to the corresponding Clifford neural layer operation. In addition, normalization techniques and activation function replacements are performed with appropriate counterparts. Different training set sizes are evaluated, and the losses for scalar and vector fields are reported. The input to the neural PDE replacement is the corresponding field at the previous t time steps, where t varies between different PDEs. The one-step loss is the sum of the mean squared errors over the field at the next time step. The rolling loss is the mean squared errors summed over the field and time dimensions after applying the neural PDE substitution 5 times. More information on the dataset, implementation details of the tested architectures, loss functions, and more detailed results can be found elsewhere. Figure 8 Example scrolling of scalar and vector field components obtained by Clifford Fourier PDE substitution is shown for qualitative comparison.

[0096] The summed MSE (SMSE) loss used in these experiments is defined as:

[0097]

[0098] where u is the target, is the model output, N fieldsincludes scalar fields as well as individual vector field components, and N y is the total number of spatial points. SMSE The equation is used for training, where N t =1, and there are four metrics:

[0099] One-step loss, where N t =1 and N fields Includes all scalar and vector components.

[0100] Vector loss, where N t =1 and N fields Only vector components are included.

[0101] Scalar loss, where N t =1 and N fields Only scalar fields are included.

[0102] Rolling loss, where N t =5 and N fields Includes all scalar and vector components.

[0103] For Maxwell's equations, electric and magnetic losses are defined similarly to the vector and scalar losses for Navier-Stokes and shallow water experiments.

[0104] Experiments were performed using two families of architectures: ResNet models and Fourier Neural Operators (FNO). All baseline models were fine-tuned for all individual experiments in terms of number of blocks, number of channels, number of modes (FNO), learning rates, normalization and initialization procedures, and activation functions. The best model is reported, and for reported Clifford results, each convolutional layer is replaced by a Clifford convolution, each Fourier layer is replaced by a Clifford Fourier layer, each normalization is replaced by a Clifford normalization, and each nonlinearity is replaced by a Clifford nonlinearity. The Clifford nonlinearity in this context is the application of the corresponding default linearity to the different multivector components.

[0105] For Navier-Stokes and shallow water experiments, a ResNet architecture with 8 residual blocks was used, each of which consisted of two convolutional layers with 3×3 kernels, shortcut connections, group normalization, and Gaussian error linear unit (GeLU) activation functions. Two embedding layers and two output layers were used, i.e., the overall architecture can be classified as a Res-20 network. Compared with the standard residual network for image classification, no down-projection technique was used. Down-projection is a convolutional layer with a stride greater than 1 or via a pooling layer. In contrast, the spatial resolution remains constant throughout the network. The same number of hidden channels, i.e., 128 channels per layer, was used throughout the network. Overall, this results in approximately 1.6 million parameters. Increasing the number of residual blocks or the number of channels did not significantly improve performance.

[0106] For each ResNet-based experiment, the fine-tuned ResNet architecture was replaced by its two Clifford counterparts: each CNN layer was replaced by (i) a Clifford CNN layer, and (ii) a rotated Clifford CNN layer. To keep the number of weights similar, the resulting architecture had 64 multivector channels instead of 128, which again resulted in about 1.6 million floating point parameters. In addition, for both architectures, the GeLU activation function was replaced by the Clifford GeLU activation function, and the group normalization was replaced by the Clifford group normalization. Using the Clifford initialization technique did not improve the results.

[0107] For Navier-Stokes and shallow water experiments, a 2D Fourier Neural Operator (FNO) consisting of 8 FNO blocks, two embedding layers, and two output layers was used. Each FNO block includes a convolution path with a 1×1 kernel and an FFT path. 16 Fourier modes (for x and y components) for point-wise weight multiplications are used, and 128 hidden channels are used overall. The GeLU activation function is used. Additional shortcut connections or normalization techniques such as batch normalization or group normalization are used. Note that normalization did not improve performance, a larger number of hidden channels did not improve performance, and more FNO blocks did not improve performance. Overall, this results in approximately 140 million parameters for the FNO-based architecture.

[0108] For the 3D Maxwell experiments, a 3D Fourier Neural Operator (FNO) consisting of 4 FNO blocks, two embedding layers, and two output layers was used. Each FNO block includes a 3D convolution path with a 1×1 kernel and an FFT path. Six Fourier modes (for x, y, and z components) were used for point-by-point weight multiplication, and 96 hidden channels were used overall. Interestingly, using more layers or more Fourier modes degrades performance. Similar to the 2D experiments, the GeLU activation function was used, and no shortcut connections or normalization techniques such as batch normalization or group normalization were applied. Overall, this results in approximately 65 million floating-point parameters for the FNO-based architecture.

[0109] For each FNO-based experiment, the fine-tuned FNO architecture was replaced by its corresponding Clifford counterpart: Each FNO layer was replaced by its Clifford counterpart. To keep the number of weights similar, the resulting architecture had 48 multivector channels instead of 128, resulting in roughly the same number of parameters. In addition, the GeLU activation function was replaced by the Clifford GeLU activation function. Using the Clifford initialization technique did not improve the results.

[0110] For 3-dimensional Maxwell experiments, each 3D Fourier transform layer is replaced by a 3D Clifford Fourier layer, and each 3D convolution is replaced by a corresponding Clifford convolution. Used 6 Fourier modes (for x, y, and z components) for point-by-point weight multiplication, and generally used 32 hidden multivector channels, which resulted in roughly the same number of parameters (55 million). Compared with 2-dimensional implementations, the Clifford initialization technique is shown to be important for 3-dimensional architectures. Most notably, the excessive weight initialization value of the Clifford convolutional layer hinders the gradient flow through the Clifford Fourier operation.

[0111] The Adam optimizer is used with a learning rate of [10 -4 ,2·10 -4 ,5 10 -4 ] Learn for 50 epochs to optimize the model and minimize L SMSEThe sum of mean squared errors (SMSE) outlined in the equation. Cosine annealing was used as the learning rate scheduler with linear warm-up. For the baseline ResNet model, the number of layers, channels, and normalization procedures were optimized. Different activation functions were tested. For the baseline FNO model, the number of layers, channels, and Fourier modes were optimized. A larger number of layers or channels did not improve the performance of the ResNet model and the FNO model. For the corresponding Clifford counterparts, the convolutional layers and Fourier layers were replaced by Clifford convolutional layers and Clifford Fourier layers, respectively. The Clifford normalization scheme was used. The number of layers was adjusted to obtain a similar number of parameters. For example, the Clifford architecture can be slightly optimized by using a different number of hidden layers compared to the baseline model. However, this would (i) slightly violate the idea of ​​having a "plug-and-play" replacement layer, and (ii) add considerable computational overhead.

[0112] All FNO and CFNO experiments used 4x16GB NVIDIA V100 machines for training. All ResNet and Clifford ResNet experiments used 8x32 GB NVIDIA V100 machines. Average training time varied between 3h and 48h, depending on the number of tasks and trajectories. Clifford runs took on average twice as long to train for the same architecture and epochs.

[0113] The incompressible Navier-Stokes equations are based on the conservation of momentum and mass of the fluid expressed for the velocity field v:

[0114]

[0115] in It is convection, Viscosity Internal force and f external force. In addition to the velocity field v(x), a scalar field s(x) can be introduced, which represents a scalar, such as smoke, transported via the velocity field. The scalar field is advected by the vector field, and as the vector field changes, the scalar field is transported with it, while the scalar field only affects the vector field via the external force term f. This is called weak coupling between the vector field and the scalar field. The 2D Navier-Stokes equations can be modeled using ΦFlow, and data is obtained on a grid with a spatial resolution of 128×128 (Δx=0.25, Δy=0.25) and a temporal resolution of Δt=1.5s. Fig. 9 The results of one-step loss and rolling loss on the test set are shown in .

[0116] Convection is the rate of change of a vector field along a vector field (in this case along itself), viscosity is the diffusion of the vector field, i.e. the net movement from areas of higher values ​​to areas of lower concentration, and μ is the viscosity coefficient. The incompressibility constraint is obtained by This results in conservation of mass. For example, v can represent the velocity of the air in a room, and s can represent the density of smoke. Similar to convection, advection is the transport of a scalar field along a vector field.

[0117] The 2D Navier-Stokes equations can be implemented using ΦFlow. The solution is carried forward, where the pressure field can be solved and then its spatial gradient subtracted. Semi-Lagrangian advection (convection) is used for v, and McCormack advection is used for s. In addition, the external buoyancy force f in Eq. 17 is expressed as a force acting on a scalar field. The solution is obtained using the Boussinesq approximation, which ignores density differences unless they appear in a way that is multiplied by the acceleration due to gravity. The key point of the Boussinesq approximation is that the difference in inertia is negligible, but the gravity is strong enough that there is a significant difference in the specific gravity of the two fluids.

[0118] Fig. 9 By way of example, plots of one-step loss and rolling loss are shown for various NN PDE modeling architectures implementing the Navier-Stokes equations. For the ResNet-like architectures, both CResNet (Clifford ResNet or ResNet with layers replaced by Clifford neural layers) and CResNetrot (Rotated Clifford ResNet or ResNet with layers replaced by rotated Clifford neural layers) are observed to improve over the ResNet baseline. Furthermore, it is observed that the rolling loss is also lower for both Clifford-based architectures, which can be attributed to better and more stable models that do not overfit to one-step predictions as easily. Finally, while in principle based on CResNet and CResNet rot The architecture is equally flexible, but in this scenario, CResNet rot Overall, it performs better than CResNet. For FNO and the corresponding Clifford Fourier-based (CFNO) architecture, the loss is generally much lower than that of the ResNet-based architecture. For all dataset sizes, and for both one-step loss and rolling loss, the CFNO architecture improves over the FNO architecture.

[0119]

[0120]

[0121] Table 1: Model comparison on four different metrics for neural PDE surrogates trained on Navier-Stokes training datasets of different sizes. Error bars are obtained by running experiments with three different initial seeds.

[0122] The shallow water equations describe a thin layer of fluid with constant density in a state of hydrostatic balance, bounded below by topography and on top by a free surface. For example, the deep-water propagation of a tsunami can be described by the shallow water equations. For simplified weather modeling, the shallow water equations express the velocity in the x-direction, or zonal velocity vx, the velocity in the y-direction, or meridional velocity vy, the acceleration due to gravity g, and the vertical displacement of the free surface η(x,y), which is then used to derive the pressure field:

[0123]

[0124] The earth's topography is modeled by h(x,y). The relationship between the vector components and the scalar components is relatively strong (strong coupling between the vector and scalar fields due to the 3-coupled PDE of Eq. 18). The implementation in SpeedyWeather.jl can be modified to obtain data on a grid with a spatial resolution of 192×96 (Δx=1.875°, Δy=3.75°) and a temporal resolution of Δt=6h. The results of the one-step loss and rolling loss on the test set are shown in Fig.10 The equations are solved on a closed domain with periodic boundary conditions. The simulation is advanced for 20 days and samples are taken every 6 h. Here, 20 days is of course not the actual simulation time, but the time of the simulation. The trajectory contains the scalar pressure and wind vector fields at 84 different time points.

[0125] SpeedyWeather.jl combines the shallow water equations with spherical harmonics (for linear terms) and Gaussian grids (for nonlinear terms), with appropriate spectral transformations. It internally uses a skip-time scheme using Robert and Willia's filters to suppress computational modes and achieve third-order accuracy. SpeedyWeather.jl is based on the atmospheric general circulation model SPEEDY in Fortran.

[0126] Fig.10 The MSE losses for one-step and rolling are shown by way of example for various NN architectures implementing the Navier-Stokes equations. Similar performance characteristics are observed as for the previously discussed Navier-Stokes experiments. However, the performance difference between the baseline and the Clifford architecture is even more significant, which can be attributed to the stronger coupling of the scalar and vector fields. For the ResNet-like architectures, CResNet and CResNet rotImproved over the ResNet baseline, with much lower rolling loss for both Clifford-based architectures and CResNet-based rot The CFNO architectures perform better overall than the CResNet-based architectures. For the Fourier-based architectures, the losses are generally much lower than the ResNet-based architectures (a training set size of 56 trajectories yields similar (C)FNO test set performance compared to a training set size of 896 trajectories for the ResNet-based architectures). The CFNO architecture improves over the FNO architecture for all dataset sizes and for both one-step and rolling losses, which is particularly noticeable for low numbers of training trajectories.

[0127]

[0128]

[0129] Table 2: Model comparison of neural PDE surrogates trained on shallow water equation training datasets of different sizes on four different metrics. Results are obtained by using two time step history inputs. Error bars are obtained by running experiments with three different initial seeds.

[0130]

[0131] Table 3: Model comparison of neural PDE surrogates trained on shallow water equation training datasets of different sizes on four different metrics. Results are obtained by using a four-time step history input. Error bars are obtained by running experiments with three different initial seeds.

[0132] Electromagnetic simulations play a key role in understanding light-matter interactions and designing optical components. Neural networks have been successfully applied to inversely design photonic structures.

[0133] Maxwell's equations for matter are:

[0134] Gauss's law

[0135] Gauss's law for magnetism

[0136] Faraday's Law of Induction

[0137] Ampere's circuit law

[0138] In an isotropic medium, the Maxwell equations in matter propagate solutions of the displacement field D and the magnetization field H, where the displacement field D is transmitted via D = ∈ 0 ∈ r E is related to the electric field, where ∈0 is the dielectric constant of free space, and ∈ ris the dielectric constant of the medium, and the magnetic field H is via H = μ0μ r B is related to the magnetic field B, where μ0 is the permeability of free space, μ r is the magnetic permeability of the medium. Finally, j is the current density and ρ is the total charge density.

[0139] The electromagnetic field F has the interesting property that the electric field E and the magnetic field B are a dual pair, so that F = E + Bi3. There is a strong coupling between the electric field and its dual (two-vector) magnetic field. This duality also applies to D and H. Solutions to Maxwell's equations in matter can be propagated using the finite-difference time-domain method, where the discretized Maxwell equations are solved in a step-by-step manner. First, the electric field vector components in a volume of space are solved at a given time. Second, the magnetic field vector components in the same volume of space are solved at the next time.

[0140] For this experiment, 18 different light sources (6 in the xy plane, 6 in the xz plane, and 6 in the yz plane) were randomly placed outside the cube, emitting light with different amplitudes and different phase shifts, causing the generated D and H fields to interfere with each other. -7 m) and a time resolution of Δt=50 s. The FNO-based architecture and the corresponding Clifford correspondence form (CFNO) were tested. Note that Maxwell's equations are generally relatively simple due to the wave-like propagation of electric and magnetic fields. However, it is a good testing ground for demonstrating the inductive biasing advantages of the Clifford-based architecture, which exploits the vector-two-vector nature of the electric and magnetic field components. Fig.11 The results of one-step loss and rolling loss on the test set are shown in Figure 2.

[0141] The two fields are strongly coupled, for example, time variations in the electric field produce magnetic fields and vice versa. Perhaps the most illustrative co-occurrence of electric and magnetic fields is when describing the propagation of light. In standard vector algebra, E is a vector and B is a pseudovector, i.e., the two fields are distinguished by a sign difference under spatial inversion. F = E + Bi3 naturally decomposes the electromagnetic field into its vector and bivector parts via the pseudoscalar i3. For example, for the basis components B of B x e1, with B x e1i3=B x e1e1e2e3=B x e2e3,B xe2e3 is the dual of the bivector sum and the basis component e1 of E. Geometric algebra reveals that a pseudovector is nothing more than a bivector represented by its dual, so the magnetic field B in F = E + Bi3 is fully represented by the complete bivector Bi3, not just B. Therefore, the multivector representing F consists of three vectors (the electric field components) and three bivectors e1i3 = e2e3, e2i3 = e3e1, e3i3 = e1e2 (the magnetic field components multiplied by i3).

[0142] Fig.11 By way of example, plots of MSE versus number of training trajectories are shown for FNO and CFNONN architectures for Maxwell equations simulations. Note that the CFNO architecture improves upon the FNO architecture, especially at low trajectory numbers. The results suggest that Clifford-based 3D Fourier layers have a stronger inductive bias and that they are more generalizable to 3D problems, which are structurally more interesting than 2D problems.

[0143]

[0144] Table 4: Model comparison of neural PDE surrogates trained on Maxwell equations training sets of different sizes on four different metrics. Results are obtained by using two time step history inputs. Error bars are obtained by running experiments with three different initial seeds.

[0145] The embodiments introduce Clifford neural layers. Convolutional Clifford layers, Fourier Clifford layers, rotational Clifford convolutional layers, and equivariant Clifford layers are provided. The Clifford neural layer handles various scalars (e.g., charge density), vectors (e.g., electric fields), bivectors (magnetic fields), and higher-order fields, organized as multivectors as appropriate geometric objects. This geometric algebraic perspective provides natural generalizable convolutions and Fourier transforms to their Clifford counterparts. The embodiments provide sophisticated and natural rules for designing new neural network layers. The multivector view of the Clifford neural layer leads to a better representation of the relationships between different fields and the relationships between their individual components, thereby significantly outperforming the corresponding standard architectures for neural PDE alternatives. The performance gap increases with decreasing data availability on all settings, showing the superior inductive bias of the Clifford neural layer.

[0146] One limitation is the current speed of Fast Fourier Transform (FFT) operations on machine learning accelerators such as Graphics Processing Units (GPUs). Although an active research area, the currently available version of the cuFFT kernel packaged in PyTorch is still not highly optimized, especially in terms of gradient propagation. In addition, the FFT of real-valued signals is Hermitian symmetric, and the output contains only positive frequencies below the Nyquist frequency. Such real-valued FFTs are used in Fourier Neural Operator (FNO) layers. On the other hand, the Clifford Fourier layer uses complex-valued FFT operations, where the reverse propagation is about two times slower. For similar parameter counts, the inference time of FNO and CFNO networks is similar. The speed of geometric convolution layers has been studied. On the one hand, Clifford convolutions are more parameter efficient than standard convolutions because they share parameters between filters, and on the other hand, for the same amount of parameters, the net number of operations is larger, resulting in an increase in training time by about 2 times. Finally, from the point of view of PDEs, the methods provided for obtaining PDE alternatives are limited because the NN must be retrained for different equation parameters or, for example, different Δt. Custom multi-vector GPU cores can overcome many of the speed issues because the Clifford operation is much more computationally intensive, which is better suited for hardware accelerators.

[0147] In addition to PDE modeling, weather prediction, electromagnetic field interactions, and fluid dynamics, applications of multi-vector representations and Clifford layers can be found in neural implicit representations.

[0148] Now provides a more detailed Cl 2,0 (R), Cl 0,2 (R), and Cl 3,0 (R). A vector space over a field F is a set V, together with two binary operations that satisfy the axioms for vector addition and scalar multiplication. The addition axiom ensures that if two elements of V are added together, another element of V is provided as the result. These elements of F are called scalars. Examples of a field F are the real numbers R and the complex numbers C. Although it is conventional to refer to the elements of a general vector space V as vectors, to avoid confusion, this discussion reserves the use of this term for R. n As shown below, a general vector space can consist of more complex and higher-order objects than scalars, vectors, or matrices.

[0149] The algebra over the field consists of a vector space V over the field F together with an additional bilinear combination law V×V→V for the elements of the vector space, that is, if a and b are any two elements of V, then ab:V×V→V is an element of V that satisfies a pair of distributive laws: a(λ1b+λ2c)=λ1ab+λ2ac and (λ1a+λ2b)c=λ1ac+λ2bc, for λ1,λ2∈F and a,b,c∈V. Note that general vector spaces do not have bilinear operations defined on their elements.

[0150] We now focus on Clifford algebras on R. Real Clifford algebras consist of n-dimensional vector spaces R n By the vector space R n The base elements of e1,...,e n Represents R n The basis elements of , and without loss of generality, these basis elements are chosen to be mutually orthogonal. Take two non-negative integers p and q, such that p + q = n, and then generate the real Clifford algebra Cl with the "trace shape" (p, q) by the following relation p,q (R), these relations define how the algebraic bilinear product affects R n The basis elements are operated on:

[0151] For 1≤i≤p, (19)

[0152]

[0153] e i e j =-e j e i For i≠j.(21)

[0154] From these relations, we can generate a basis for the vector space of the Clifford algebra, denoted by G. Equations 19 and 20 show that the product between two vectors produces a scalar. According to the above definition of the field algebra, the Clifford algebra with a vector space G is equipped with a bilinear product G×G→G, which combines two elements from the vector space G and produces another element of the same space G. Therefore, scalars and vectors are both elements of the vector space G. Equation 21 shows that in addition to the scalar and vector elements, two basic elements (such as e i e j and e j e i ) are also part of the vector space G. By combining equations 19, 20, and 21, even higher-order elements can be created, such as e i e j e k(for i≠j≠k) or e1e2...e p+q , all of which must be part of the vector space G.

[0155] In order to determine the Cl p,q (R) is the basis element of the vector space G. Note that, depending on the sign of the permutation σ, the element e σ(1) e σ(2) ...e σ(k) and e1e2...e k are related by a simple scalar multiplication factor of plus or minus one. Therefore, consider R n An unordered combination of the basis elements of suffices: the basis of the vector space G consists of {1,e1,e2,...,e p+q ,e1e2,...,e p+q- 1e p+q ,...,e1e2...e p+q} given.

[0156] In summary, two different vector spaces are introduced. The first is the vector space R that generates the Clifford algebra n , the second is the vector space G, which is represented by the Clifford algebra Cl p,q The convention is to denote the vector space of a real Clifford algebra by the superscript n of the dimension of the generated vector space, so that for the generated vector space R n Generate G n . It should be noted that the vector space G n The dimension is 2 n =2 p+q .

[0157] Common examples of low Wickliford algebras are: (i) Cl 0,0 (R), which is a one-dimensional algebra spanned by the vectors {1}, and is therefore isomorphic to R (the field of real numbers); (ii) Cl 0,1 (R), which is a vector space G spanned by {1,e1} 1 The two-dimensional algebra of , where the square of the basis vector e1 is -1, and is therefore isomorphic to C (the field of complex numbers); (iii) Cl 0,2 (R), which is a 1, e2,e1e2} spanned by the vector space G 2 4-dimensional algebra, where the squares of e1, e2, e1e2 are all -1 and satisfy the anti-commutative relation. Therefore, Cl 0,2 (R) is isomorphic to the quaternion H.

[0158] The order of a Clifford algebra is the dimension of the subspace it represents. For example, the Clifford algebra Cl 0,2(R) and Cl 2,0 The basis elements {1, e1, e2, e1e2} of (R) have order {0, 1, 1, 2}. Using the concept of order, the vector space of the Clifford algebra can be divided into linear subspaces consisting of elements of each order. The order subspace of minimum dimension is M0, which is the subspace of all scalars (elements with 0 basis vectors). The elements of M1 are called vectors, the elements of M2 are bivectors, and so on. In general, the Clifford algebra Cl p,q The vector space G p+q can be written as a straightforward sum over all these subspaces:

[0159]

[0160] Elements of the Clifford algebra are called multivectors, which contain elements of the subspace (e.g., scalars, vectors, bivectors, trivectors, etc.). The highest-order basis elements are called pseudoscalars, which are 2 corresponds to the double vector e1e2 in R 3 corresponds to the three vectors e1e2e3. Pseudoscalars are usually symbolized by i p+q express.

[0161] Using Equations 19, 20, and 21, we can see how to use the basis elements of the generated vector space V to form the vector space G of the Clifford algebra. p+q Note that G p+q How are the elements of G combined? In other words, how are multivectors bilinearly operated on? The geometric product is a bilinear operation on multivectors in Clifford algebra. For any multivector a, b, c ∈ G p+q , and the scalar λ, the geometric product has the following properties:

[0162] ab∈G p+q Closure (23)

[0163] (ab)c=a(bc) binding (24)

[0164] λa=aλ Scalar multiplication commutativity (25)

[0165] a(b+c)=ab+ac Additive distributive property (26)

[0166] The geometric product is generally non-commutative, i.e. ab ≠ ba. The geometric product consists of two things: the inner product (which captures similarities) and the outer product (wedge product) which captures differences.

[0167] The dual a* of a multivector a is defined as:

[0168] a*=ai p+q (27),

[0169] where i p+q The corresponding pseudoscalar representing the Clifford algebra.

[0170] This definition allows different multivectors to be related to each other, which is a useful property when defining the Clifford Fourier transform. For example, for R 2 The dual of a scalar is a bivector, and for the Clifford algebra R 3 , the dual of a scalar is a three-vector.

[0171] Clifford Algebra Cl 0,1 (R) is a vector space G spanned by {1,e1} 1 The two-dimensional algebra of , and the square of the basis vector e1 is -1. Therefore, Cl 0,1 (R) is algebraically isomorphic to the field of complex numbers C. This becomes more apparent by recognizing that the basis element with the highest order (i.e., e1) is the pseudoscalar i1, which is the imaginary part of a complex number. Therefore, the geometric product between two multivectors a = a0 + a1e1 and b = b0 + b1e1 is also isomorphic to the product of two complex numbers:

[0172] ab=a0b0+a0b1e1+a1b0e1-a1b1e1e1=(a0b0-a1b1)+(a0+b1+a1b0)e1 (28)

[0173] Clifford Algebra Cl 2,0 (R) is a 4-dimensional algebra with a vector space G spanned by the basis vectors {1,e1,e2,e1e2} 2 , where the squares of e1, e2, e1e2 are all +1. Two multivectors a=a0+a1e1+a2e2+a 12 e1e2 and b=b0+b1e1+b2e2+b 12 The geometric product of e1e2 is defined as: ab = a0b0 + a0b1e1 + a0b2e2 + a0b 12 e1e2

[0174] +a1b0e1+a1b1e1e1+a1b2e1e2+a1b 12 e1e1e2

[0175] +a2b0e2+a2b1e2e1+a2b2e2e2+a2b 12 e2e1e2

[0176] +a 12 b0e1e2+a 12 b1e1e2e1+a 12 b2e1e2e2+a 12 b12 e1e2e1e2. (29)

[0177] Using the relationship e1e1=1, e2e2=1, and e i e j =-e j e i (for i≠j∈{e1,e2}), it follows that e1e2e1e2=-1, and:

[0178] ab=a0b0+a1b1+a2b2-a 12 b 12

[0179] +(a0b1+a1b0-a2b 12 +a 12 b2)e1

[0180] +(a0b2+a1b 12 +α2b0-a 12 b1)e2

[0181] +(a0b 12 +a1b2-a2b1+α 12 b0)e1e2. (30)

[0182] vector by Logo. Clifford product of two vectors

[0183] This produces the geometric product xy:

[0184]

[0185] The asymmetry xΛy=-yΛx is associated with a double vector which can be interpreted as an oriented plane segment.

[0186] Equation 31 can be rewritten as a geometric product expressing the (symmetric) inner product and the (antisymmetric) outer product:

[0187] x∧y=1 / 2(xy-yx) (32)

[0188] <x,y> =1 / 2(x+yx) (33)

[0189] From Clifford Algebra Cl 2,0 The vector space G of (R) 2 The basis vectors of are {1,e1,e2,e1e2}, with perhaps the most interesting being e1e2. A closer look is provided by the unit twin vector i2=e1e2, which is the plane spanned by e1 and e2 and determined by the geometric product:

[0190]

[0191] Among them, the inner product<e1,e2> is zero due to the orthogonality of these basis vectors. The bivector i2 (if squared) yields And therefore i2 represents the true geometric meaning From equation 34, we can get:

[0192] e2=e1i2=-i2e1

[0193] e1=i2e2=-e2i2.(35)

[0194] Multi-vector a∈G 2 The dual of is defined as i2a via this bivector. Thus, the dual of a scalar is a bivector, and the dual of a vector is still a vector. The dual pair of basis vectors is and These dual pairs allow any multivector a to be written as

[0195] a=a0+a1e1+a2e2+a 12 e 12 ,

[0196]

[0197] It can be thought of as two complex-valued parts: a spin part that commutes with i2 and a vector part that anti-commutes with i2.

[0198] Clifford Algebra Cl 0,2 (R) is a 4-dimensional algebra, where the vector space G 2 It is composed of the basis vectors {1, e1, e2, e1e2}, where the squares of e1, e2, e1e2 are all -1. Clifford algebra Cl 0,2 (R) is an algebra isomorphic to the quaternion H, which is usually written as The (imaginary part) basis element The following relations are satisfied:

[0199]

[0200] Quaternions are also formed by Zhang Cheng's 4-dimensional algebra, where The square of is -1. The basis element 1 is often called the scalar part, and the basis element is called the vector part of the quaternion. In equations (29) and (30) it is shown that Cl 0,2 The geometric product of two multivectors in (R).

[0201] Clifford algebra is an 8-dimensional algebra with a vector space G spanned by the basis vectors {1,e1,e2,e3,e1e2,e1e3,e2e3,e1e2e3} 3 , i.e., a scalar, three vectors {e1, e2, e3}, three double vectors {e1e2, e1e3, e2e3}, and a three-vector e1e2e3. This three-vector is an algebraic pseudo-scalar i3. Similar to Cl 2,0 The geometric product of (R) defines the geometric product of two multivectors, following multivector association and bilinear multiplication: For i = 1, 2, 3 (38)

[0202] e i e j =-e i e j For i, j = 1, 2, 3, i ≠ j (39)

[0203] Using Definition 2, the dual pair of Cl3,0 is:

[0204]

[0205]

[0206]

[0207]

[0208] The geometric product of Cl3,0(R) is defined similarly to the geometric product of Cl2,0(R) in the following way:

[0209] ab=a0b0+a0b1e1+a0b2e2+a0b3e3

[0210] +a0b 12 e1e2+a0b 13 e1e3+a0b 23 e2e3+a0b 123 e1e2e3

[0211] +a1b0e1+a1b1e1e1+a1b2e1e2+a1b3e1e3

[0212] +a1b 12 e1e1e2+a1b 13 e1e1e3+a1b 23 e1e2e3+a1b 123 e1e1e2e3

[0213] +a2b0e2+a2b1e2e1+a2b2e2e2+a2b3e2e3<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0214] <h2 style=";text-align:left;direction:ltr"> +a2b<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> e2e1e2+a2b<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> e1e3e2+a2b<h2 style=";text-align:left;direction:ltr"> 23 <h2 style=";text-align:left;direction:ltr"> e2e2e3-a2b<h2 style=";text-align:left;direction:ltr"> 123 <h2 style=";text-align:left;direction:ltr"> e2e2e1e3<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0215] <h2 style=";text-align:left;direction:ltr"> +a3b<h2 style=";text-align:left;direction:ltr"> 03 <h2 style=";text-align:left;direction:ltr"> +a3b1e3e1+a3b2e3e2+a3b3e3e3<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0216] <h2 style=";text-align:left;direction:ltr"> +a3b<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> e1e3e2-a3b<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> e1e3e3-a3b<h2 style=";text-align:left;direction:ltr"> 23 <h2 style=";text-align:left;direction:ltr"> e2e3e3+a3b<h2 style=";text-align:left;direction:ltr"> 123 <h2 style=";text-align:left;direction:ltr"> a1e2e3e3<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0217] <h2 style=";text-align:left;direction:ltr"> +a<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> b0e1e2-a<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> b1e2e1e1+a<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> b2e1e2e2+a<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> b3e1e2e3<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0218] <h2 style=";text-align:left;direction:ltr"> +a<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> b<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> e1e2e1e2-a<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> b<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> e1e1e2e3+a<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> b<h2 style=";text-align:left;direction:ltr"> 23 <h2 style=";text-align:left;direction:ltr"> e2e2e1e3+a<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> b<h2 style=";text-align:left;direction:ltr"> 123 <h2 style=";text-align:left;direction:ltr"> e1e2e1e2e3<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0219] <h2 style=";text-align:left;direction:ltr"> +a<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> b0e1e3-a<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> b1e3e1e1+a<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> b2e1e3e2+a<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> b3e1e3e3<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0220] <h2 style=";text-align:left;direction:ltr"> -a<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> b<h2 style=";text-align:left;direction:ltr"> 12 <h2 style=";text-align:left;direction:ltr"> e1e1e3e2+a<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> b<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> e1e3e1e3-a<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> b<h2 style=";text-align:left;direction:ltr"> 23 <h2 style=";text-align:left;direction:ltr"> e1e2e3e3+a<h2 style=";text-align:left;direction:ltr"> 13 <h2 style=";text-align:left;direction:ltr"> b<h2 style=";text-align:left;direction:ltr"> 123 <h2 style=";text-align:left;direction:ltr"> e1e3e1e2e3<h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0221] <h2 style=";text-align:left;direction:ltr"> +a<h2 style=";text-align:left;direction:ltr"> 23 <h2 style=";text-align:left;direction:ltr"> b0e2e3+a<h2 style=";text-align:left;direction:ltr"> 23 <h2 style=";text-align:left;direction:ltr"> b1e1e3e2+a<h2 style=";text-align:left;direction:ltr"> 23 <h2 style=";text-align:left;direction:ltr"> b2e2e3e2+a<h2 style=";text-align:left;direction:ltr"> 23b3e2e3e3

[0222] +a 23 b 12 e2e3e1e2-a 23 b 13 e2e1e3e3+a 23 b 23 e2e3e2e3+a 23 b 123 e2e3e1e2e3

[0223] +a 123 b0e1e2e3+a 123 b1e1e2e3e1-a 123 b2e1e3e1e2+a 123 b3e1e2e3e3

[0224] +a 123 b 12 e1e2e2e1e2+a 123 b 13 e1e2e3e1e3+a 123 b 23 e1e2e3e2e3+a 123 b 123 e1e2e3e1e2e3 (44)

[0226] The negative sign appears due to the reordering of the basis elements. Equation 44 simplifies to:

[0227] ab=a0b0+a1b1+a2b2+a3b3-α12b12-a13b13-a23b23-a123b123+(a0b1+a1b0-a2b12-a3b13+a12b2+a13b3-a23b123-a123b23)e1

[0228] +(a0b2+a1b12+a2b0-a3b23-a12b1+a13b123+a23b3+a123b13)e2

[0229] +(a0b3+a1b13+a2b23+a3b0-a12b123-a13b1-a23b2-a123b12)e3

[0230] +(a0b12+a1b2-a2b1+a3b123+a12b0-a13b23+a23b13+a123b3)e1e2

[0231] +(a0b13+a1b3-a2b123-a3b1+a12b23+a13b0-a23b12-a123b2)e1e3

[0232] +(a0b23+a1b123+a2b3-a3b2-a12b13-a13b12+a23b0+a123b1)e2e3

[0233] +(a0b 123 +a1b 23 -a2b 13 +a3b 12 +a 12 b3-a 13 b2+a 23 b1+a 123 b0)e1e2e3. (45)

[0234] Now available for G 2 The multi-vectors in (by 2-dimensional vector space R 2 The derivation of the translation equivariant Clifford convolutional layer for the multi-vector generated by the Clifford algebra. For details of the Clifford convolutional layer, refer to equations (8)-(10). Then describe the 3D vector space R 3 Extension of the generated Clifford algebra.

[0235] f:Z 2 →(G 2 ) c in is a multi-vector feature map, and w:Z 2 →(G 2 ) Cin is a multi-vector kernel, then [[L t f]*w](x)=[L t [f*w]](x).

[0236] prove.

[0237]

[0238] Now let's discuss Cl in more detail. 2,0 (R) Layer implementation. The 2D Clifford CNN layer can be implemented using Equation 30, where corresponds to 4 different kernels representing a 2D multi-vector kernel, i.e. 4 different convolutional layers, and Corresponding to the scalar, vector, and bivector parts of the input multivector field. Channels at different layers represent different stacks of scalars, vectors, and bivectors. All kernels have the same number of input and output channels (the number of input and output multivectors), so channel mixing appears separately for different terms of equations 30 and 45. Finally, usually not all parts of the multivector are present in the input vector field. This can be easily solved by omitting only the corresponding parts of equations 30 and 45. Similar reasoning applies to the output vector field.

[0239] By using the Clifford algebra Cl 0,2 (R) is isomorphic to quaternions to achieve an alternative parameterization of the Clifford CNN layer introduced in Equation 9. This alternative parameterization exploits the fact that quaternion rotations can be achieved through matrix multiplication. Using the isomorphism, the feature map f j and filter w i,j Represented as a quaternion: Using this quaternion representation, the feature map f can be realized j With w i,j More precisely, consider a composite operation that produces a scalar quantity and a quaternion rotation, where the quaternion rotation acts on the quaternion f j The vector part of the quaternion output is the vector part of the quaternion, and a non-zero expansion factor is only generated for the vector part of the quaternion output. i , j f j (w i,j ) -1 Acting on f j The vector part of and can be manipulated algebraically into vector-matrix operations R i,j f j , where R i,j :H→H is from w i,j In other words, we can i,j Scalar and vector parts of ∈H Constructed rotation matrix R i,j , to transform f j The vector part of ∈H In summary, the rotating multi-vector filter Z 2 →(G 2 ) acts on the feature map f through the following terms j : Rotation transformation This rotation transformation acts on the multi-vector feature map f:Z 2 →(G 2 ) c in; and the additional scalar response of the multi-vector filter:

[0240]

[0241] in This is the scalar output of Equation 30. The rotation matrix R i,j The written form of (yx) is:

[0242]

[0243] in, is a normalized filter, where For clarity, in the rotation matrix R i,j Internal dependencies (yx) are omitted.

[0244] Similar to the 2D case, the 3D Clifford CNN layer can be implemented using Equation 45, where {b0,b1,b2,b 12 ,b 13 ,b 23 ,b 123} corresponds to 8 different kernels representing a 3D multi-vector kernel, i.e., 8 different convolutional layers, and {a0,a1,a2,a 12 ,a 13 ,a 23 ,a 123} corresponds to the scalar part, vector part, two-vector part and three-vector part of the input multi-vector field.

[0245] Different normalization schemes have been proposed to stabilize and accelerate training DNNs. Their standard formulations only work for real values. Simply translating and scaling multivectors so that their mean is 0 and their variance is 1 is not sufficient, as this does not ensure equal variance across all components.

[0246] The batch normalization formula can be applied to complex values. The same principle can be used to provide an appropriate batch normalization scheme for multi-vectors. For the form a = a0 + a1e1 + a2e2 + a 12 The 2D multi-vector of e1e2 can be formulated as a whitening 4D vector problem:

[0247]

[0248] The covariance matrix V is

[0249]

[0250] The shift parameter β is a multi-vector parameter with 4 learnable components, and the scaling parameter γ is a 4×4 positive semi-definite matrix. Multi-vector batch normalization is defined as:

[0251] BN(a)=γa+β(48)

[0252] When the batch size is small, it may be more appropriate to use group normalization or layer normalization. These can be derived along the appropriate tensor dimensions by appropriate application of Eq. 47.

[0253] Similar to Clifford normalization, the quaternion initialization scheme can be adapted to the Clifford layer. In practice, tighter bounds are required for the uniform distribution from which the Clifford weights are sampled. However, despite intensive investigation, no performance gain was observed for 2D experiments compared to the default PyTorch initialization scheme. However, 3D implementations require much smaller initialization values ​​(e.g., about 1 / 8th the size).

[0254] As shown in the figure, Clifford convolution satisfies equivariance under translation of multivector input. However, the current definition of Clifford convolution is not equivariant under multivector rotation or reflection. Here, a general kernel constraint is derived that allows the establishment of generalized Clifford convolutions that are equivariant under rotation or reflection of multivectors. That is, if the multivector kernel multivector filter The equivariance of the Clifford layer under rotation and reflection is proven if the following constraints are satisfied:

[0255] w i,j (Tx) = Tw i,j (x),

[0256] For 0≤j <c in , first define the orthogonal transformation on multi-vectors by:

[0257]

[0258] where u and f are multivectors multiplied using the geometric product. The negative sign is obtained from the reflection rather than by rotation, i.e., depends on the parity of the transformation. This construction is called the "versor" product. The above construction makes it immediately clear that T(fg) = (Tf)(Tg). Tx means an orthogonal transformation of Euclidean vectors (which can also be defined using rotators in principle). To demonstrate equivariance, for the multivector f:Z 2 →(G) c in and a group of c out Multi-vector filter prove:

[0259] f′(Tx)=Tf(x), (50)

[0260] and

[0261] w i (Tx) = Tw i (x), (51)

[0262] Equations 50 and 51 produce:

[0263]

[0264] That is, if the input multivector field is transformed into a multivector, and the kernel satisfies the equivariance constraints described, then the output multivector field is also appropriately transformed into a multivector. Note that T may act differently on each component (scalar, vector, pseudoscalar, pseudovector) under rotation and / or reflection.

[0265] Now,

[0266]

[0267]

[0268] where in the fourth line the variables are transformed as y→y′, in the fifth line the invariance of the sum “measure” under T is used, and in the sixth line the transformation properties of f and w are used i The equivariance of , the multivector property is used in the seventh row, and the linearity of T is used in the eighth row.

[0269] Now we deduce that for G 2 and G 3 The multi-vectors in (i.e., the 2-dimensional vector space R 2 and the 3D vector space R 3 The Clifford Fourier layer is an implementation of the multivector of the Clifford algebra generated by the Clifford algebra. In any dimension n, for a continuous n-dimensional complex-valued signal f(x) = f(x1,...,x n ):R n →Fourier transform of C is defined as:

[0270]

[0271] The condition is that the integral exists where x and ξ are n-dimensional vectors and<x,ξ> is the contraction of x and ξ. Usually,<x,ξ> is the inner product, and ξ is the dual vector space R n＊ The inverse transform theorem states the reverse transformation from the frequency domain to the spatial domain:

[0272]

[0273] The Fourier transform of Equation 53 can be rewritten in coordinates:

[0274]

[0275] The discrete form of Equation 53 is obtained by converting M1×...×M n The n-dimensional complex signal at the grid point f(x)=f(x1,...,x n ):R n →C is converted to its complex Fourier mode:

[0276]

[0277] where (ξ1,...,ξ n )∈Z M 1......Z M n. The Fast Fourier Transform (FFT) greatly speeds up the computation of the transform of Eq. 56 by factoring the discrete Fourier transform matrix into a product of sparse (mostly zero) factors.

[0278] Similar to Equation 53, for multi-vector valued function f(x): R 2 →G 2 and vector x,ξ∈R 2 The Clifford Fourier transform and the corresponding inverse transform are defined as:

[0279]

[0280] The condition is that the integral exists. The difference between equations 53 and 54 is that f(x) and denote the multi-vector field in the spatial domain and the frequency domain respectively, and use the pseudo-scalar i2=e1e2 in the exponent. Inserting the definition of the multi-vector field, Equation 57 can be rewritten as Equation 14.

[0281] Spin / Vector Section The discretized version of is similar to Equation 56 and reads:

[0282]

[0283] In the same way, (ξ1,ξ2)∈Z M 1×Z M n. Similar to the Fourier Neural Operator (FNO), where the weight tensor is applied point by point in Fourier space, a multi-vector weight tensor can be applied point by point Exceeding the cut-off frequency The Fourier patterns of are set to zero. In doing so, the Clifford Fourier patterns are modified via the geometric product

[0284]

[0285] When the spin and vector parts of Eq. 59 are combined, Clifford Fourier modes naturally result. Similar to FNO, higher order modes are truncated. Finally, the residual connections used in the FNO layer are replaced by a multi-vector weight matrix implemented as a Clifford convolution, ideally Cl 2,0 (R) Convolutional layer.

[0286] Below is a proof of the 2D Clifford convolution theorem for a multivector-valued filter applied from the right such that the filter operation is consistent with a Clifford convolution layer. First, the commutation of the Clifford kernel with the spin and the anti-commutation with the vector part of the multivector are shown. For each scalar s∈R and multivector a∈G 2 Will accumulate ae i 2s is written as

[0287]

[0288] For the basis of the spin part, note that 1i2=i21, and for the basis of the vector part, e1i2=e1e1e2=-e1e2e1=-i2e1. Therefore, the Fourier kernel Commutes with the spin part, and anti-commutes with the vector part of a. Here is a proof of the convolution theorem for the commutative spin part and anti-commutative vector part of a.

[0289] Let field f:R 2 →G 2 is a multi-vector value, filter k s :R 2 →G 2 is the spin value, filter k v :R 2 →G 2 is a vector value, and let If exists,

[0290]

[0291] in

[0292] prove:

[0293]

[0294] Similar to Equation 53, for multi-vector valued function f:R 3 →G 3 and vector x,ξ∈R 3 The Clifford Fourier transform and the corresponding inverse transform are defined as:

[0295]

[0296] The condition is that the integral exists. Multi-vector valued function f:R 3 →G 3 ,

[0297] f=f0+f1e1+f2e2+f3e3+f 12 e 12 +f 13 e 13 +f 23 e 23 +f 123 e 123 (67)

[0298] It can be expressed as follows via the pseudo-scalar i3=e1e2e3:

[0299] f=(f0+f 123 i3)1

[0300] +(f1+f 23 i3)e1

[0301] +(f2+f 31 i3)e2

[0302] +(f3+f 12 i3)e3, (68)

[0303] By considering the four dual pairs f0=f0(x)+f 123 (x)i3, f1=f1(x)+f 23 (x)i3, f2=f2(x)+f 31 (x)i3 and f3=f3(x)+f 12 (x)i3 applies four standard Fourier transforms to obtain 3Wickliford Fourier transforms, all of which can be viewed as complex-valued signals f0,f1,f2,f3:R 3 →C. Therefore, f(x) can be understood as C 4 The 3D Clifford Fourier transform is a linear combination of the four classical Fourier transforms:

[0304]

[0305] Similar to the 2D Clifford Fourier transform, a multi-vector weight tensor can be applied point-wise Exceeding the cut-off frequency The Fourier patterns of are set to zero. In doing so, the Clifford Fourier patterns are modified via the geometric product

[0306]

[0307] When the four parts of Eq. 69 are combined, the Clifford Fourier model is naturally obtained. Finally, the residual connection used in the FNO layer is replaced by a multi-vector weight matrix implemented as a Clifford convolution, such as Cl 3,0 (R) Convolutional layer.

[0308] First, we show that the Clifford kernel commutes with spin and decommutes with the vector part of the multivector. For every scalar s∈R and multivector a∈G 3 , accumulation of i 3s is written as

[0309]

[0310] Compared with the 2-Wickliford Fourier transform, now all four parts of the multivector of equation 68 commute with i3:

[0311] li3=i31

[0312] e1i3=e1e1e2e3=-e1e2e1e3=e1e2e3e1=i3e1

[0313] e2i3=e2e1e2e3=-e1e2e2e3=e1e2e3e2=i3e2

[0314] e3i3=e3e1e2e3=-e1e3e2e3=e1e2e3e3=i3e3. (72)

[0315] Let field f:R 3 →G 3 is a multi-vector value, filter k a :R 3 →G 3 is multi-vector-valued, and let Existence, then in

[0316] prove:

[0317]

[0318] The 2D Clifford Fourier layer can be implemented by applying two standard Fourier transforms on the dual pairs of Eq. 14. These dual pairs can be viewed as complex-valued inputs. Similarly, by applying, for example, 3,0 The dual pair of (Equation 40-Equation 43) uses four standard Fourier transforms to realize the 3D Clifford Fourier layer. Since the Clifford convolution theorem is respectively for l 2,0 The vector part and the spin part of the 3,0The four dual pairs of hold, so the modes in Fourier space are multiplied using the geometric product. Finally, the inverse Fourier transform is applied and the multivector is reconstructed in the spatial domain.

[0319] Fig.12 A diagram of an embodiment of a method 1200 for machine learning (ML) modeling of a multi-vector system operating on a multi-vector object is shown by way of example. The illustrated method 1200 includes: at operation 1220, receiving a multi-vector object as an input by the ML model, the multi-vector object representing a state of the multi-vector system; and at operation 1222, operating the multi-vector input by the ML model using a Clifford layer to generate a multi-vector output, the multi-vector output representing the state of the multi-vector system in response to the multi-vector input, the Clifford layer including neurons implementing a multi-vector kernel. The multi-vector input represents a change in state (e.g., a stimulus) in the multi-vector system.

[0320] The method 1200 may also include, wherein the Clifford layer is a Clifford Fourier layer, a Clifford convolution layer, or an equivariant Clifford convolution layer. The method 1200 may also include, wherein the Clifford layer is a Clifford Fourier neural operator (FNO), which includes a Clifford Fourier layer and a Clifford convolution layer. The method 1200 may also include, wherein the multi-vector system is an electromagnetic field or a dynamic fluid. The method 1200 may further include, wherein the Clifford layer is a Clifford convolution, and the method further includes constraining the kernel for the Clifford convolution layer so that the Clifford convolution layer is equivariant.

[0321] The method 1200 may also include, wherein the multi-vector object includes two or more of a scalar, a vector, a double vector, or a triple vector. The method 1200 may also include, wherein the multi-vector object includes a vector. The method 1200 may also include, wherein the multi-vector object includes a double vector. The method 1200 may also include, wherein the multi-vector object includes a triple vector.

[0322] Artificial intelligence (AI) is a field that involves developing decision-making systems to perform cognitive tasks that traditionally require the participation of living beings (e.g., humans). A neural network (NN) is a computing structure loosely modeled on biological neurons. In general, a NN encodes information (e.g., data or decision making) via weighted connections (e.g., synapses) between nodes (e.g., neurons). Modern NNs are fundamental to many AI applications, such as object or condition recognition, device behavior modeling, etc. The ML model 102 or a portion thereof, such as the Clifford neural layer 108, may include one or more NNs or be implemented using one or more NNs.

[0323] Many NNs are represented as matrices of weights (sometimes called parameters) corresponding to the modeled connections. NNs operate by accepting data into a set of input neurons that typically have many outgoing connections to other neurons. At each traversal between neurons, the corresponding weights modify the input and are tested against a threshold at the destination neuron. If the weighted value exceeds the threshold, the value is weighted again, or transformed by a nonlinear function, and sent to another neuron further down the NN graph—if the threshold is not exceeded, the value is typically not sent to the next graph neuron, and the synaptic connection remains inactive. The process of weighting and testing continues until an output neuron is reached; the pattern and value of the output neuron constitutes the result of the NN processing.

[0324] Most NNs rely on accurate weights for optimal operation. However, NN designers typically do not know which weights will work for a given application. NN designers typically choose multiple layers of neurons or specific connections between layers, including ring connections. A training process can be used to determine appropriate weights by selecting initial weights.

[0325] In some examples, the initial weights may be randomly selected. The training data is fed into the NN and the results are compared to an objective function that provides an error indication. The error indication is a measure of the degree of error in the results of the NN compared to the expected results. The error is then used to correct the weights. Over multiple iterations, the weights will converge together to encode the actual application data (operational data) into the NN. This process can be called optimization of an objective function (e.g., a cost or loss function), whereby the cost or loss is minimized.

[0326] Gradient descent techniques are commonly used to perform objective function optimization. Gradients (e.g., partial derivatives) are calculated with respect to layer parameters (e.g., weights) to provide the direction of correction and possible degree of correction, but this does not result in a single correction that sets the weight to the "correct" value. That is, over several iterations, the weight will move toward a "correct" or practically useful value. In some implementations, the amount or step size of the move is fixed (e.g., the same for each iteration). Small step sizes tend to take a long time to converge, while large step sizes may oscillate around the correct value or exhibit other undesirable behaviors. Variable step sizes can be tried to provide faster convergence while avoiding the disadvantages of large step sizes.

[0327] Backpropagation is a technique whereby training data is fed forward through a NN, where "forward" means that the data starts at the input neuron and follows the directed graph of neuron connections until the output neuron is reached, and an objective function is applied back through the NN to correct the synaptic weights. At each step in the backpropagation process, the result of the previous step is used to correct the weights. Thus, the result of the output neuron correction is applied to the neurons connected to the output neuron, and so on, until the input neuron is reached. Backpropagation has become a popular technique for training various NNs. Any known optimization algorithm for backpropagation can be used, such as stochastic gradient descent (SGD), Adam, etc.

[0328] Fig.13 is a block diagram of an example of an environment including a system for neural network training. One or more of the Clifford neural layer and other parts of the ML model discussed herein may be used, at least in part, Fig.13 The system is implemented by using an artificial NN (ANN) 1305 trained using a processing node 1310. The processing node 1310 can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or other processing circuit. In an example, multiple processing nodes can be used to train different layers of the ANN 1305, or even different nodes 1307 within a layer. Thus, a group of processing nodes 1310 are arranged to perform the training of the ANN 1305.

[0329] The processing node set 1310 is arranged to receive a training set 1315 for the ANN 1305. The ANN 1305 includes a set of nodes 1307 (illustrated as rows of nodes 1307) arranged in a hierarchy and a set of inter-node weights 1308 (e.g., parameters) between the nodes in the set of nodes. In an example, the training set 1315 is a subset of the complete training set. Here, the subset can enable processing nodes with limited storage resources to participate in training the ANN 1305.

[0330] The training data may include a plurality of values ​​representing a domain, such as image features, etc. After the ANN 1305 is trained, each value of the training or input 1317 to be classified is provided to a corresponding node 1307 in the first layer or input layer of the ANN 1305. These values ​​are propagated through the layers and are changed by the objective function.

[0331] As noted, the group of processing nodes is arranged to train neural networks to create trained neural networks. After ANN is trained, the data input into ANN will produce, for example, effective classification 1320 (e.g., input data 1317 will be assigned to categories). The training performed by the group of processing nodes 1307 is iterative. In an example, each iteration of training ANN 1305 is performed independently between the layers of ANN 1305. Therefore, two different layers can be processed in parallel by different members of the group of processing nodes. In an example, the different layers of ANN 1305 are trained on different hardware. The members of different members of the group of processing nodes can be located in different packages, shells, computers, cloud-based resources, etc. In an example, each iteration of training is performed independently between the nodes in the node set. This example is additional parallelization, thereby independently training each node 1307 (e.g., neuron). In an example, nodes are trained on different hardware.

[0332] Fig.14 A block diagram of an embodiment of a machine 1400 (e.g., a computer system) that implements one or more embodiments is shown by way of example. ML model 102, Clifford neural layer 108, Clifford convolutional layer 600, Clifford FNO layer 700, method 1200, Fig.13 The training system, or one or more of its components or operations, may be implemented at least in part using components of machine 1400. Example machine 1400 (in the form of a computer) may include a processing unit 1402, a memory 1403, a removable memory 1410, and a non-removable memory 1412. Although an exemplary computing device is illustrated and described as machine 1400, in different embodiments, the computing device may be in different forms. For example, the computing device may be a smart phone, a tablet, a smart watch, or a computer including a computer system such as the one described above. Fig.14 Other computing devices with the same or similar elements shown and described. Furthermore, although various data storage elements are shown as part of the machine 1400, the storage additionally or alternatively includes cloud-based storage accessible via a network, such as the Internet.

[0333] The memory 1403 may include volatile memory 1414 and non-volatile memory 1408. The machine 1400 may include or have access to a computing environment that includes various computer-readable media, such as the volatile memory 1414 and non-volatile memory 1408, removable storage 1410 and non-removable storage 1412. Computer memory includes random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices capable of storing computer-readable instructions for execution to perform the functions described herein.

[0334] The machine 1400 may include or be able to access a computing environment including input 1406, output 1404, and communication connection 1416. The output 1404 may include a display device that may also act as an input component, such as a touch screen. The input 1406 may include one or more of a touch screen, a touchpad, a mouse, a keyboard, a camera, one or more device-specific buttons, one or more sensors integrated into the machine 1400 or coupled to the machine 1400 via a wired or wireless data connection, and other input components. The computer may operate in a networked environment using a communication connection to connect to one or more remote computers, such as a database server, including cloud-based servers and storage. The remote computer may include a personal computer (PC), a server, a router, a network PC, a peer device or other public network node, etc. The communication connection may include a local area network (LAN), a wide area network (WAN), a cellular, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi), Bluetooth, or other networks.

[0335] Computer-readable instructions stored on a computer-readable storage device may be executed by a processing unit 1402 (sometimes referred to as a processing circuit) of the machine 1400. Hard drives, CD-ROMs, and RAM are some examples of items that include non-transitory computer-readable media (such as storage devices). For example, the computer program 1418 may be used to cause the processing unit 1402 to perform one or more methods or algorithms described herein.

[0336] In some embodiments, the operations, functions or algorithms described herein can be implemented in software. Software may include computer executable instructions stored on a computer or other machine-readable medium or storage device, such as one or more non-transient memories (e.g., non-transient machine-readable media) or other types of hardware-based storage devices (whether local or network). In addition, such functions may correspond to a subsystem, which may be software, hardware, firmware or a combination thereof. Multiple functions may be performed in one or more subsystems as needed, and the described embodiments are only examples. Software may be executed on a processing circuit, such as may include a digital signal processor, an ASIC, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or other types of processors operating on a computer system (such as a personal computer, a server, or other computer systems), thereby turning such a computer system into a specially programmed machine. Additionally or alternatively, the processing circuit may include electrical and / or electronic components (e.g., one or more transistors, resistors, capacitors, inductors, amplifiers, modulators, demodulators, antennas, radios, regulators, diodes, oscillators, multiplexers, logic gates, buffers, caches, memories, GPUs, CPUs, field programmable gate arrays (FPGAs), etc.). The terms computer-readable medium, machine-readable medium, and storage device do not include carrier waves or signals as long as the carrier waves and signals are considered too transient.

[0337] Additional Notes and Examples

[0338] Example 1 includes a method for machine learning (ML) modeling of a multi-vector system operating on a multi-vector object, the method comprising: receiving a multi-vector object as input by the ML model, the multi-vector object representing a state of the multi-vector system; and operating the multi-vector input by the ML model using a Clifford layer to generate a multi-vector output, the multi-vector output representing the state of the multi-vector system in response to the multi-vector input, the Clifford layer including neurons that implement a multi-vector kernel.

[0339] In Example 2, Example 1 also includes, wherein the Clifford layer is a Clifford Fourier layer, a Clifford convolutional layer, or an equivariant Clifford convolutional layer.

[0340] In Example 3, at least one of Examples 1-2 also includes, wherein the Clifford layer is a Clifford Fourier Neural Operator (FNO), and the Clifford Fourier Neural Operator includes a Clifford Fourier layer and a Clifford convolutional layer.

[0341] In Example 4, at least one of Examples 1-3 also includes wherein the multi-vector system is an electromagnetic field or a dynamic fluid.

[0342] In Example 5, at least one of Examples 1-4 also includes wherein the multi-vector object includes two or more of a scalar, a vector, a bivector, or a trivector.

[0343] In Example 6, Example 5 also includes, wherein the multi-vector object includes a vector.

[0344] In Example 7, at least one of Examples 5-6 further includes, wherein the multi-vector object comprises a dual vector.

[0345] In Example 8, at least one of Examples 5-7 also includes, wherein the multi-vector object includes a three-vector.

[0346] In Example 9, at least one of Examples 1-8 also includes, wherein the Clifford layer is a Clifford convolution, and the method further includes constraining a kernel for the Clifford convolution layer such that the Clifford convolution layer is equivariant.

[0347] Example 10 includes a system comprising a processing circuit and one or more memories, the one or more memories comprising instructions and parameters for neurons, the neurons being trained for machine learning (ML) modeling of a multi-vector system operating on multi-vector objects, and the instructions, when executed by the processing circuit, cause the processing circuit to perform operations for implementing the ML model using a method as one of Examples 1-9.

[0348] Example 11 includes a computer-readable medium including instructions that, when executed by a machine, cause the machine to perform the method of any one of claims 1-9.

[0349] Although several embodiments have been described in detail above, other modifications are possible. For example, the logic flows depicted in the accompanying figures do not require the order or sequence order shown to achieve the desired results. Desirability of the embodiments may include that users have confidence in the state of their data, settings, controls, and confidentiality before, during, and after migrating to a new version of an application. Using multiple factors to check the state, integrity, existence, or absence of data before and after migration can increase confidence. Other steps may be provided, or certain steps may be eliminated from the described processes, and other components may be added to or removed from the described systems. Other embodiments may be within the scope of the following claims.

Claims

1. A method for machine learning (ML) modeling of a multi-vector system operating on a multi-vector object, the method comprising: receiving, by an ML model, the multi-vector object as input, the multi-vector object representing a state of the multi-vector system; as well as The multi-vector input is operated on by the ML model using a Clifford layer to generate a multi-vector output, the multi-vector output representing the state of the multi-vector system in response to the multi-vector input, the Clifford layer comprising neurons that implement a multi-vector kernel.

2. The method of claim 1, wherein the Clifford layer is a Clifford Fourier layer, a Clifford convolution layer, or an equivariant Clifford convolution layer.

3. The method according to claim 1, wherein the Clifford layer is a Clifford Fourier Neural Operator (FNO), and the Clifford Fourier Neural Operator includes a Clifford Fourier layer and a Clifford convolutional layer. The method of claim 1 , wherein the multi-vector system is an electromagnetic field or a dynamic fluid. The method of claim 1 , wherein the multi-vector object comprises two or more of a scalar, a vector, a bivector, or a trivector. The method of claim 5 , wherein the multi-vector object comprises the vector. The method of claim 5 , wherein the multi-vector object comprises the dual vector. The method of claim 5 , wherein the multi-vector object comprises the three vectors.

9. The method of claim 1, wherein the Clifford layer is a Clifford convolution, and the method further comprises constraining a kernel for the Clifford convolution layer such that the Clifford convolution layer is equivariant.

10. A system comprising: Processing circuit; as well as one or more memories, the one or more memories comprising instructions and parameters for neurons trained for machine learning (ML) modeling of a multi-vector system operating on a multi-vector object, and the instructions, when executed by the processing circuit, cause the processing circuit to perform operations for implementing the ML model, the operations comprising: receiving, by the ML model, the multi-vector object as input, the multi-vector object representing a state of the multi-vector system; as well as The multi-vector input is operated on by a Clifford layer of the ML model to generate a multi-vector output, the multi-vector output representing the state of the multi-vector system in response to the multi-vector input, the Clifford layer comprising neurons implementing a multi-vector kernel.

11. The system of claim 10, wherein the Clifford layer is a Clifford Fourier layer, a Clifford convolution layer, or an equivariant Clifford convolution layer.

12. The system of claim 10, wherein the Clifford layer is a Clifford Fourier Neural Operator (FNO), the Clifford Fourier Neural Operator comprising a Clifford Fourier layer and a Clifford Convolutional layer.

13. The system of claim 10, wherein the multi-vector system is an electromagnetic field or a dynamic fluid.

14. The system of claim 10, wherein the multi-vector object comprises two or more of a scalar, a vector, a bivector, or a trivector.

15. A computer-readable medium comprising instructions that, when executed by a machine, cause the machine to perform a method for implementing a machine learning (ML) model for modeling a multi-vector system, the multi-vector system operating on a multi-vector object, the method comprising a method according to one of claims 1-9.

Images