Neuromorphic image deformation sensing method and system based on learnable activation operator

Abstract

The invention provides a neural morphological image deformation sensing method and system based on a learnable activation operator, and relates to the technical field of image tracking, and the method comprises the steps: extracting a feature map of an image pair before and after deformation through a feature encoder, and decomposing the feature map of the image pair into a plurality of sub-regions; obtaining related body feature representations of a plurality of pairs of image sub-regions before and after deformation, and standardizing the last dimension through a softmax function to obtain related body feature representations of the sub-regions; constructing two layers of learnable neuron modules as nerve activation operators, converting related body feature representation of each sub-region into x-direction displacement information and y-direction displacement information, and splicing to obtain a preliminarily predicted displacement field; and splicing all the sub-region correlation body feature representations into a complete correlation body feature representation, splicing the complete correlation body feature representation and the preliminarily predicted displacement field in a channel dimension to obtain a feature map, and carrying out multiple times of down-sampling and up-sampling to obtain a final displacement field prediction result of the image pair. According to the method, a dense and high-precision displacement field prediction result is obtained through a small number of parameters.

Description

Technical Field

[0001] The present invention relates to the field of sensitive image tracking and deep learning technology, and in particular to a neuromorphic image deformation perception method and system based on a learnable activation operator. Background Art

[0002] Image motion and deformation estimation is a fundamental task in computer vision. It infers the motion or spatial deformation of objects in a scene by analyzing pixel changes between adjacent frames in a single-view or multi-view image sequence. It is crucial for applications such as visual object tracking, action recognition, autonomous driving, visual navigation, and structural behavior monitoring. Typical methods for image motion and deformation estimation, including optical flow and digital image correlation, rely on assumptions of image grayscale conservation and spatial smoothness (e.g., the Lucas-Kanade iterative matching algorithm). However, these assumptions often fail to adapt well to complex environments (e.g., illumination variations, dimming, image degradation, and occlusion). In recent years, deep learning-based neural network models using optical flow prediction have effectively achieved relevant results, such as FlowNet, RAFT, GMFlow, and their variants. However, the high-precision models employ attention mechanisms and computation of relevant feature volumes, making these methods computationally intensive and demanding on hardware resources, especially for real-time applications. Optical flow models are often trained on the assumption of brightness constancy and minimal high-order deformation in the displacement field, based on which motion estimation is performed. However, in deformation problems, this assumption may not hold true because the displacement field is more complex and there is noise and illumination, which makes the displacement field prediction face additional challenges.

[0003] While existing deep learning image motion and displacement field estimation models have achieved significant progress in displacement field prediction accuracy for small deformation problems, they face challenges in practical deployment, especially when running on resource-constrained development boards. Specifically, these models suffer from large parameter counts, high computational complexity, and high energy consumption. These limitations significantly restrict their deployment on edge devices. Summary of the Invention

[0004] To solve the above problems, the present invention provides a neuromorphic image deformation perception method and system based on a learnable activation operator. The method constructs a lightweight deep neural network model for image motion and displacement field estimation based on a trainable neural activation operator, and obtains dense and highly accurate displacement field prediction results with a small number of parameters.

[0005] To achieve the above objectives, the present invention provides the following technical solutions.

[0006] The present invention provides a neuromorphic image deformation perception method based on a learnable activation operator, comprising the following steps:

[0007] Extracting feature maps of the image pair before and after deformation through a feature encoder, and decomposing the feature maps of the image pair into a plurality of sub-regions; wherein the deformed image sub-region in the image pair has edges of adjacent sub-regions added to the size of the pre-deformed image sub-region;

[0008] Obtain the relevant volume feature representations of multiple pairs of image sub-regions before and after deformation, and normalize the last dimension through the softmax function to obtain the relevant volume feature representations of each sub-region;

[0009] Two single-layer learnable neuron modules are used as neural activation operators to convert the feature representations of the relevant volume of each sub-region into x-direction displacement information and y-direction displacement information respectively, and then splice them to obtain a preliminary predicted displacement field. Among them, the B-spline function is used as the one-dimensional activation function between each input node and output node in the neural activation operator;

[0010] The correlation volume feature representations of all sub-regions are spliced ​​into a complete correlation volume feature representation, and then the complete correlation volume feature representation and the preliminary predicted displacement field are spliced ​​in the channel dimension to obtain a feature map; the spliced ​​feature map is downsampled, and the predicted displacement field is obtained based on the neural activation operator prediction;

[0011] After splicing the predicted displacement field with the current feature map, downsampling and displacement field prediction based on the neural activation operator are performed in sequence; after multiple executions, upsampling and displacement field prediction based on the neural activation operator are performed on the spliced ​​feature map in sequence; after multiple executions, a displacement field of the same size as the complete correlation volume feature representation is obtained; the displacement field of the same size as the complete correlation volume feature representation is pixel-expanded by upsampling to obtain the displacement field prediction result of the final image pair.

[0012] Preferably, the acquisition of the sub-region related volume feature representation comprises the following steps:

[0013] The calculation of the correlation volume feature C of a pair of sub-regions M0 and M1 at the corresponding positions of the feature map of the image pair before and after deformation is shown in the following formula:

[0014]

[0015] Where D is the channel dimension of the sub-regions M0 and M1 extracted from the feature map;

[0016] The matching distribution C' is obtained by normalizing the last dimension of the relevant body feature representation C through the softmax function:

[0017] C = softmax(C)

[0018] The information represented by the last dimension of the sub-region correlation feature C' is the correlation information between one element in M0 and all elements in M1.

[0019] Preferably, the single-layer neural activation operator includes multiple input nodes and 1 output node, forming a linear mapping; there is an independent one-dimensional activation function between each input node and the output node; wherein the one-dimensional activation function is a B-spline function.

[0020] Preferably, downsampling the spliced ​​feature map and obtaining a predicted displacement field based on a neural activation operator prediction comprises the following steps:

[0021] The concatenated feature map is convolved and downsampled to obtain the downsampled feature map;

[0022] The shape of the downsampled feature map is changed to match the shape required by the neural activation operator input dimension, input into the neural activation operator, and the displacement field prediction value output by the neural activation operator is obtained.

[0023] Preferably, the step of sequentially performing upsampling and displacement field prediction based on a neural activation operator on the spliced ​​feature map comprises the following steps:

[0024] The concatenated feature map is upsampled by transposed convolution to obtain the upsampled feature map;

[0025] The shape of the upsampled feature map is changed to match the shape required by the neural activation operator input dimension, input into the neural activation operator, and the displacement field prediction value output by the neural activation operator is obtained.

[0026] Preferably, the pixel expansion of the displacement field having the same size as the complete correlation volume feature representation by upsampling comprises the following steps:

[0027] Construct a weight generation network, including sequentially connected convolutional layers, ReLU activation functions, and convolutional layers;

[0028] The feature map obtained by the last upsampling and the displacement field predicted by the neural activation operator are input into the weight generation network to obtain the weight;

[0029] Each original pixel point of the displacement field and the surrounding 8 neighboring pixels are weighted based on the weight to obtain each extended pixel point of the displacement field and obtain the displacement field prediction result.

[0030] Preferably, the method further includes training with the expanded optical flow dataset, comprising the following steps:

[0031] Construct an optical flow dataset, which is divided into reference images and deformation images;

[0032] Perform local or global deformation on the deformation map to obtain displacement fields and deformation maps of complex deformations;

[0033] The complex deformed displacement field is superimposed on the original displacement field to obtain a new displacement field; the new displacement field and deformation map are added to the used data set to form an expanded optical flow data set.

[0034] Preferably, the method further includes constructing a loss function based on displacement field loss and displacement field gradient loss for training, including the following steps:

[0035] The displacement field loss is:

[0036]

[0037] The displacement field gradient loss is:

[0038]

[0039] Among them, f represents the predicted displacement field, g represents the real displacement field; m and n are the displacements in two directions; λ is the weight.

[0040] Preferably, the acquisition of the displacement field of the complex deformation includes the following steps:

[0041] A 2D displacement field is defined for each deformation map by combining random rigid body translation, rotation, stretching, compression, and shear with local deformation represented by a 2D Gaussian function:

[0042]

[0043] Wherein, the subscripts x and y represent the horizontal and vertical directions; u and v represent the displacement in the x and y directions at the coordinate (i, j); t represents the translation; k represents the degree of tensile and compressive deformation; γ represents the degree of shear deformation; θ represents the rotational deformation angle; a represents the Gaussian deformation amplitude; (i0, j0), (i1, j1) represent the center position of the Gaussian deformation, and c represents the center position of the Gaussian deformation. x0 ,c y0 ,c x1 ,c y1 represents the standard deviation of the Gaussian deformation.

[0044] A neuromorphic image deformation perception system based on a learnable activation operator, the system comprising:

[0045] Feature encoder, used to extract feature maps of image pairs before and after deformation;

[0046] A correlation calculation module is used to decompose the feature maps of the image pair into multiple sub-regions. The deformed image sub-regions in the image pair are increased by the edges of adjacent sub-regions based on the size of the pre-deformed image sub-regions. The module is also used to obtain the correlation volume feature representations of multiple pairs of pre-deformed and post-deformed image sub-regions, and normalize the last dimension using the softmax function to obtain the correlation volume feature representations of each sub-region.

[0047] The related feature activation module is used to use two single-layer learnable neuron modules as neural activation operators to convert the related volume feature representations of each sub-region into x-direction displacement information and y-direction displacement information respectively, and then splice them to obtain a preliminary predicted displacement field. The B-spline function is used as the one-dimensional activation function between each input node and output node in the neural activation operator;

[0048] The displacement field optimization module is used to splice the feature representations of all sub-region correlation volumes into a complete correlation volume feature representation, and then splice the complete correlation volume feature representation and the preliminary predicted displacement field in the channel dimension to obtain a feature map; downsample the spliced ​​feature map and obtain a predicted displacement field based on the neural activation operator prediction; and splice the predicted displacement field with the current feature map, and then sequentially perform downsampling and displacement field prediction based on the neural activation operator; after multiple executions, the spliced ​​feature map is sequentially upsampled and displacement field prediction based on the neural activation operator is performed; after multiple executions, the displacement field of the same size as the complete correlation volume feature representation is obtained;

[0049] The upsampling module is used to expand the displacement field of the same size as the complete correlation volume feature representation by upsampling to obtain the displacement field prediction result of the final image pair.

[0050] Beneficial effects of the present invention:

[0051] This paper proposes a neuromorphic image deformation perception method and system based on learnable activation operators. This method constructs a lightweight deep neural network model for image motion and displacement field estimation based on the activation function of the neural operator. Combined with multiple downsampling and upsampling, this method achieves dense and highly accurate displacement field prediction results with a small number of parameters. Furthermore, the displacement fields of the model training dataset, including complex high-order deformations, can effectively learn displacement field prediction for small deformation problems. This overcomes the limitation of deep learning optical flow models deployed on low-power computing platforms, enabling the displacement field prediction model to be deployed on a wider range of embedded platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of a neuromorphic image deformation perception method based on a learnable activation operator according to an embodiment of the present invention;

[0053] Figure 2This is a diagram of the architecture of a lightweight deep neural network model for image motion and displacement field estimation based on a trainable neural activation operator according to an embodiment of the present invention;

[0054] Figure 3 3 is a comparison diagram of the displacement field predicted by the model and the actual displacement field in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0056] Example

[0057] The process of the neuromorphic image deformation perception method based on learnable activation operators proposed in this invention is as follows: Figure 1 As shown, the following steps are included:

[0058] Construct a lightweight deep neural network model for image motion and displacement field estimation based on trainable neural activation operators, including feature encoder, correlation calculation module, related feature activation module, displacement field optimization module and upsampling module. The model architecture is shown in the figure below. Figure 2 shown.

[0059] S1: Extract feature maps of the image pair before and after deformation through a feature encoder, and decompose the feature maps of the image pair into multiple sub-regions; among which, the edges of adjacent sub-regions are added to the size of the sub-region of the image before deformation in the image pair after deformation.

[0060] S2: Through the correlation calculation module, the relevant body feature representations of multiple pairs of image sub-regions before and after deformation are obtained respectively, and the last dimension is normalized through the softmax function to obtain the relevant body feature representations of each sub-region.

[0061] S3: Through the relevant feature activation module, two single-layer learnable neuron modules are used as neural activation operators to convert the relevant body feature representations of each sub-region into x-direction displacement information and y-direction displacement information respectively, and then spliced ​​together to obtain a preliminary predicted displacement field; among them, the B-spline function is used as the one-dimensional activation function between each input node and output node in the neural activation operator.

[0062] S4: Through the displacement field optimization module, all sub-region correlation volume feature representations are spliced ​​into a complete correlation volume feature representation, and then the complete correlation volume feature representation and the preliminary predicted displacement field are spliced ​​in the channel dimension to obtain a feature map; the spliced ​​feature map is downsampled, and the predicted displacement field is obtained based on the neural activation operator prediction.

[0063] S5: After splicing the predicted displacement field with the current feature map, downsampling and displacement field prediction based on the neural activation operator are performed in sequence; after multiple executions, upsampling and displacement field prediction based on the neural activation operator are performed on the spliced ​​feature map in sequence; after multiple executions, a displacement field with the same size as the complete correlation volume feature representation is obtained.

[0064] S6: Through the upsampling module, the displacement field with the same size as the complete correlation volume feature representation is expanded by upsampling to obtain the displacement field prediction result of the final image pair.

[0065] Specifically, in this embodiment:

[0066] The feature encoder includes a multi-layer convolutional neural network. The input images I0 and I1 are input into the encoder respectively. After the encoder extracts the features, the feature maps F0 and F1 of I0 and I1 are obtained, with the shape of [B, C, H / 4, W / 4].

[0067] The correlation calculation module decomposes F0 into multiple subregions of the shape [B, C, 4, 4]. The resulting subregions are in the shape [B, C, H*W / 256, 4, 4]. F1 also needs to be decomposed into multiple subregions of the shape [B, C, 6, 6]. A 6*6 subregion is a 4*4 subregion plus the edges of its adjacent subregions. The resulting subregions are in the shape [B, C, H*W / 256, 6, 6].

[0068] The correlation volume feature representation is calculated for a pair of subregions M0 and M1 at the corresponding positions of F0 and F1, as shown in the following formula. The shape of M0 is [B, C, 4*4], the shape of M1 is [B, C, 6*6], and the shape of C is [B, 16, 36].

[0069]

[0070] The calculated correlation volume feature representation is normalized by the softmax function on the last dimension of C to obtain the matching distribution C'. As shown in the following formula:

[0071] C'=softmax(C)

[0072] The information represented by the last dimension of C' is the correlation information between a certain element in M0 and all elements in M1.

[0073] The relevant feature activation module needs to convert the feature matching information of the feature body C' into displacement information. This step is achieved through a specific, trainable neural activation operator, which has n-dimensional input and 1-dimensional output. The neural activation operator is a fully connected network structure designed based on the Kolmogorov-Arnold representation theorem. Unlike traditional multi-layer perceptrons (MLPs), neural activation operators use learnable activation functions on the weights of the network. These activation functions are usually parameterized univariate functions, such as B-spline functions. Such a design allows deep neural network blocks to achieve higher expressive power with fewer parameters and good interpretability. This design enables the system to learn the mapping relationship between feature matching information and displacement information, making it more flexible and accurate when processing new data.

[0074] A single-layer neural activation operator consists of 36 input nodes and one output node, forming a (36,1) linear mapping. Each input node has an independent one-dimensional activation function connected to the output node, rather than fixed traditional weights. By adjusting the parameters of these one-dimensional activation functions, the neural activation operator module can adaptively learn the complex nonlinear mapping from n-dimensional input features to displacement information. This learning process allows the module to capture subtle patterns in the input data and generate accurate displacement information.

[0075] In this example, a B-spline function is used as the activation function. B-spline functions are a type of piecewise polynomial function that provides a high degree of flexibility while maintaining continuity and smoothness. The coefficients of each polynomial segment can be optimized through a training process to minimize prediction error. The spline order is 3, and the grid size is 10. The activation function is shown below, where x is the total number of elements in the last dimension of C'.

[0076] φ=w*spline(x)

[0077] The complete matching feature activation module can be expressed as follows:

[0078]

[0079] The feature representations C' of all sub-region correlation bodies are input into two different correlation feature activation modules respectively. The two modules are used to convert feature-related information into x-direction displacement information and y-direction displacement information respectively. After splicing, the preliminary predicted displacement field u0 can be obtained, and the shape of u0 is [B, 2, H / 4, W / 4].

[0080] Then, all sub-region correlation volume feature representations are spliced ​​into a complete correlation volume feature representation C1. The shape of the complete correlation volume feature representation is [B, 36, H / 4, W / 4]. The complete correlation volume feature representation C1 and the initial predicted displacement field are spliced ​​in the channel dimension to obtain the feature map F2, which is input into the displacement field optimization module.

[0081] The first-layer input F2 of the displacement field optimization module is downsampled through a 3x3 convolutional kernel downsampling module to obtain a feature map F3. The shape of feature map F3 is [B, 256, H / 8, W / 8]. F3 is then fed into the displacement field prediction module to obtain the displacement field u1. The displacement field input module first reshapes the input feature map to [B*H / 8*W / 8, 256]. After that, the displacement field is reshaped back to [B, 2, H / 8, W / 8] by passing it through a neural activation operator module with an input dimension of 256 and an output dimension of 2.

[0082] The feature map F3 and the displacement field u1 are then concatenated in the channel dimension to generate the feature map F4. The second-layer input of the displacement field optimization module, F4, is downsampled through a 3x3 convolutional kernel downsampling module to generate the feature map F5. The shape of feature map F5 is [B, 512, H / 16, W / 16]. This is then fed into the displacement field prediction module to generate the displacement field u2. The displacement field input module first reshapes the input feature map to [B*H / 16*W / 16, 512]. This is then passed through a KANLinear module with an input dimension of 256 and an output dimension of 2, reshaping the displacement field back to [B, 2, H / 16, W / 16].

[0083] Feature map F5 and displacement field u2 are then concatenated in the channel dimension to generate feature map F6. The third-layer input of the displacement field optimization module, F6, is upsampled through a 3x3 convolution kernel transposed convolution upsampling module to generate feature map F7. Feature map F7 has a shape of [B, 256, H / 8, W / 8]. This shape is then fed into the displacement field prediction module to generate displacement field u3. The displacement field input module first reshapes the input feature map to [B*H / 8*W / 8, 256]. This is then passed through a neural activation operator module with an input dimension of 256 and an output dimension of 2, reshaping the displacement field back to [B, 2, H / 8, W / 8].

[0084] Feature map F7 and displacement field u3 are then concatenated in the channel dimension to produce feature map F8. The fourth-layer input of the displacement field optimization module, F8, is upsampled through a 3x3 convolution kernel transposed convolution upsampling module to produce feature map F9. Feature map F9 has a shape of [B, 36, H / 4, W / 4]. This is then fed into the displacement field prediction module to produce displacement field u4. The displacement field input module first reshapes the input feature map to [B*H / 4*W / 4, 256]. This is then passed through a neural activation operator module with an input dimension of 256 and an output dimension of 2, reshaping the displacement field back to [B, 2, H / 4, W / 4]. The module that directly predicts displacement using the neural activation operator significantly improves prediction accuracy.

[0085] Then, the shape of the displacement field u4 is transformed into [B, 2, H, W] through the upsampling module, which is the calculated displacement field u.

[0086] The upsampling module performs a 4x upsampling process. Each pixel is expanded to 8x8 pixels. Specifically, each expanded pixel is weighted by the original pixel and its eight neighboring pixels (a total of 9 pixels). The weights are generated by the network. The network inputs for weight generation include F9 and u4. Through a 3x3 convolutional layer, a ReLU activation function, and a 3x3 convolutional layer, the resulting weights are in the shape of [B, 4x4x9, H / 4, W / 4].

[0087] In this embodiment, the training of a lightweight deep neural network model for image motion and displacement field estimation based on a trainable neural activation operator includes the following steps:

[0088] A dataset for training the model is constructed. The input of the model is an image pair and a displacement field between the images. In this embodiment, the FlyingChairs dataset can be selected.

[0089] The images in the optical flow dataset are divided into reference images and deformed images. This embodiment requires further complex deformation of the deformed image. The deformation can be applied locally or globally. The displacement field of the applied complex deformation is then superimposed on the original displacement field. The resulting new displacement field and new deformed image are added to the dataset used.

[0090] Generate displacement fields for additional deformations: A 2D displacement field is defined for each sample image by combining random rigid body translation, rotation, stretching / compression, shearing, and local deformation represented by a 2D Gaussian function.

[0091]

[0092] Wherein, the subscripts x and y represent the horizontal and vertical directions; u and v represent the displacement in the x and y directions at the coordinate (i, j); t represents the translation; k represents the degree of tensile and compressive deformation; γ represents the degree of shear deformation; θ represents the rotational deformation angle; a represents the Gaussian deformation amplitude; (i0, j0), (i1, j1) represent the center position of the Gaussian deformation, and c represents the center position of the Gaussian deformation. x0 ,c y0 ,c x1 ,c y1 represents the standard deviation of the Gaussian deformation.

[0093] The loss function used in model training comprehensively considers the accuracy and smoothness of the displacement field. The loss function L is divided into displacement field loss and displacement field gradient loss.

[0094] L=L u +L grad

[0095] At the same time, the displacement difference and displacement gradient between each pixel in the reference image and the corresponding pixel in the deformed image are calculated pixel by pixel. The pixel and its eight adjacent points form a subset for calculating the difference. The center point and its adjacent points have different weights, and the difference between adjacent points should be multiplied by λ when adding it to the loss.

[0096] The displacement field loss is:

[0097]

[0098] The displacement field gradient loss is:

[0099]

[0100] Among them, f represents the predicted displacement field, g represents the real displacement field; m and n are the displacements in two directions.

[0101] The comparison between the displacement field predicted by the model and the actual displacement field is as follows: Figure 3 As shown in the figure, it can be seen that the prediction results can achieve sub-pixel accuracy, which proves the role of learnable neural activation operators in predicting displacement fields.

[0102] The above is a neuromorphic image deformation perception method based on a learnable activation operator provided by one embodiment of this embodiment. Based on the same idea, this embodiment also provides a corresponding neuromorphic image deformation perception system based on a learnable activation operator. For the specific definition of the neuromorphic image deformation perception system based on a learnable activation operator, please refer to the definition of the neuromorphic image deformation perception method based on a learnable activation operator above, which will not be repeated here. Each module in the above-mentioned neuromorphic image deformation perception system based on a learnable activation operator can be fully or partially implemented by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0103] This embodiment also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 A neuromorphic image deformation perception method based on learnable activation operators is provided.

[0104] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0105] The construction of the small deformation displacement field prediction system includes the following steps:

[0106] The monocular camera is connected to a power supply and a low-power computing platform via a cable, capturing image data simultaneously and transmitting it to the computing platform. The computing platform then sequentially calculates the incoming image data using a lightweight, small-deformation displacement field prediction model, generating a continuous predicted displacement field that is then transferred to a storage medium. Because the model is lightweight, it ensures instantaneous computational speed and enables instantaneous displacement field prediction for continuous input image data. Users can then extract the required displacement field data from the storage medium or combine multiple sets of displacement field data to obtain long-term displacement field predictions.

[0107] The trained model has a total of 13.9M parameters. Furthermore, since it avoids using an attention mechanism and iterative optical flow updates, the model has low memory requirements.

[0108] Among the computing platforms, you can choose: Jetson Nano from the NVIDIA Jetson series, with 4GB or 2GB LPDDR4 RAM; Jetson Xavier NX from the NVIDIA Jetson series, with 8GB 128-bit LPDDR4x RAM, providing more powerful computing power; Raspberry Pi 4, with standard configuration of 4GB, 8GB or 16GB LPDDR4 SDRAM; ualcomm Snapdragon 820 / 845 / 865 development boards, these Qualcomm processor-based development boards provide RAM ranging from 4GB to 16GB, and they have built-in Hexagon DSP and Adreno GPU.

[0109] All of the above development boards can run ONNX-type lightweight deep neural network models for image motion and displacement field estimation based on trainable neural activation operators.

[0110] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A neuromorphic image deformation perception method based on a learnable activation operator, characterized in that: The following steps are involved: Extracting feature maps of the image pair before and after deformation through a feature encoder, and decomposing the feature maps of the image pair into a plurality of sub-regions; wherein the deformed image sub-region in the image pair adds edges of adjacent sub-regions based on the size of the pre-deformed image sub-region; Obtain the relevant volume feature representations of multiple pairs of image sub-regions before and after deformation respectively, and normalize the last dimension through the softmax function to obtain the relevant volume feature representations of each sub-region; A two-layer learnable neuron module is constructed as a neural activation operator, and the feature representation of each sub-region is converted into x-direction displacement information and y-direction displacement information respectively, and then the initial predicted displacement field is obtained after splicing; wherein, the B-spline function is used as a one-dimensional activation function between each input node and output node in the neural activation operator; The correlation volume feature representations of all sub-regions are spliced ​​into a complete correlation volume feature representation, and then the complete correlation volume feature representation and the preliminary predicted displacement field are spliced ​​in the channel dimension to obtain a feature map; the spliced ​​feature map is downsampled, and the predicted displacement field is obtained based on the neural activation operator prediction; After splicing the predicted displacement field with the current feature map, downsampling and displacement field prediction based on the neural activation operator are performed in sequence; after multiple executions, upsampling and displacement field prediction based on the neural activation operator are performed on the spliced ​​feature map in sequence; after multiple executions, a displacement field of the same size as the complete correlation volume feature representation is obtained; the displacement field of the same size as the complete correlation volume feature representation is pixel-expanded by upsampling to obtain the displacement field prediction result of the final image pair.

2. The neuromorphic image deformation perception method based on a learnable activation operator according to claim 1, characterized in that: The acquisition of the sub-region related body feature representation comprises the following steps: The calculation of the correlation volume feature representation C of a pair of sub-regions M0 and M1 at the corresponding positions of the feature map of the image pair before and after deformation is shown in the following formula: Where D is the channel dimension of sub-regions M0 and M1 extracted from the feature map; The matching distribution C' is obtained by normalizing the last dimension of the relevant body feature representation C through the softmax function: C'=softmax(C) The sub-region correlation body feature indicates that the information represented by the last dimension of C' is the correlation information between one element in M0 and all elements in M1.

3. The neuromorphic image deformation perception method based on a learnable activation operator according to claim 1, characterized in that: The single-layer neural activation operator includes multiple input nodes and one output node, forming a linear mapping; there is an independent one-dimensional activation function between each input node and the output node; wherein the one-dimensional activation function is a B-spline function.

4. The neuromorphic image deformation perception method based on a learnable activation operator according to claim 1, characterized in that: The step of downsampling the concatenated feature map and obtaining a predicted displacement field based on a neural activation operator prediction includes the following steps: The concatenated feature map is convolved and downsampled to obtain a downsampled feature map; The shape of the downsampled feature map is changed to match the shape required by the neural activation operator input dimension, input into the neural activation operator, and the displacement field prediction value output by the neural activation operator module is obtained.

5. The neuromorphic image deformation perception method based on learnable activation operators according to claim 1, characterized in that: The step of sequentially performing upsampling and displacement field prediction based on a neural activation operator on the spliced ​​feature map comprises the following steps: The concatenated feature map is upsampled by transposed convolution to obtain the upsampled feature map; The shape of the upsampled feature map is changed to match the shape required by the input dimension of the neural activation operator, input into the neural activation operator, and the displacement field prediction value output by the neural activation operator is obtained.

6. The neuromorphic image deformation perception method based on learnable activation operators according to claim 1, characterized in that: The pixel expansion of the displacement field having the same size as the complete correlation volume feature representation by upsampling comprises the following steps: Construct a weight generation network, including sequentially connected convolutional layers, Relu activation functions, and convolutional layers; The feature map obtained by the last upsampling and the displacement field predicted by the displacement field based on the neural activation operator are input into the weight generation network to obtain the weight; Each original pixel point of the displacement field and the surrounding 8 neighboring pixels are weighted based on the weight to obtain each extended pixel point of the displacement field, and obtain the displacement field prediction result.

7. The neuromorphic image deformation perception method based on a learnable activation operator according to claim 1, characterized in that: It also includes training with the expanded optical flow dataset, including the following steps: Construct an optical flow dataset, which is divided into reference images and deformation images; Perform local or global deformation on the deformation map to obtain the displacement field and deformation map of complex deformation; The complex deformed displacement field is superimposed on the original displacement field to obtain a new displacement field; the new displacement field and the deformation map are added to the used data set to form an expanded optical flow data set.

8. The neuromorphic image deformation perception method based on learnable activation operators according to claim 1, characterized in that: It also includes constructing a loss function based on displacement field loss and displacement field gradient loss for training, including the following steps: The displacement field loss is: The displacement field gradient loss is: Among them, f represents the predicted displacement field, g represents the real displacement field; m and n are the displacements in two directions; λ is the weight.

9. The neuromorphic image deformation perception method based on learnable activation operators according to claim 7, characterized in that: The acquisition of the displacement field of the complex deformation includes the following steps: A 2D displacement field is defined for each deformation map by combining random rigid body translation, rotation, stretching, compression, and shear with local deformation represented by a 2D Gaussian function: Wherein, the subscripts x and y represent the horizontal and vertical directions; u and v represent the displacement in the x and y directions at the coordinate (i, j); t represents the translation; k represents the degree of tensile and compressive deformation; γ represents the degree of shear deformation; θ represents the rotational deformation angle; a represents the Gaussian deformation amplitude; (i0, j0), (i1, j1) represent the center position of the Gaussian deformation, and c represents the center position of the Gaussian deformation. x0 ,c y0 ,c x1 ,c y1 Represents the standard deviation of the Gaussian deformation.

10. A neuromorphic image deformation perception system based on a learnable activation operator, characterized in that: The system comprises: A feature encoder, used to extract feature maps of image pairs before and after deformation; A correlation calculation module is used to decompose the feature map of the image pair into multiple sub-regions respectively; wherein the deformed image sub-region in the image pair increases the edge of the adjacent sub-region on the basis of the size of the pre-deformed image sub-region; and is also used to obtain the correlation volume feature representation of multiple pairs of pre-deformed and post-deformed image sub-regions respectively, and normalize the last dimension through the softmax function to obtain the correlation volume feature representation of each sub-region; The related feature activation module is used to use two single-layer learnable neuron modules as neural activation operators to convert the related body feature representations of each sub-region into x-direction displacement information and y-direction displacement information respectively, and then splice them to obtain a preliminary predicted displacement field; wherein the B-spline function is used as a one-dimensional activation function between each input node and output node in the neural activation operator; The displacement field optimization module is used to splice all sub-region correlation volume feature representations into a complete correlation volume feature representation, and then splice the complete correlation volume feature representation and the preliminary predicted displacement field in the channel dimension to obtain a feature map; downsample the spliced ​​feature map, and obtain the predicted displacement field based on the neural activation operator prediction; and splice the predicted displacement field with the current feature map, and sequentially perform downsampling and displacement field prediction based on the neural activation operator; after multiple executions, the spliced ​​feature map is sequentially upsampled and the displacement field prediction based on the neural activation operator is performed; after multiple executions, the displacement field of the same size as the complete correlation volume feature representation is obtained; The upsampling module is used to expand the displacement field of the same size as the complete correlation volume feature representation by upsampling the pixels to obtain the displacement field prediction result of the final image pair.

Images