Neuromorphic image morphing perception method and system based on learnable activation operator

Abstract

This invention provides a neuromorphic image deformation perception method and system based on learnable activation operators, relating to the field of image tracking technology. The method includes: extracting feature maps of image pairs before and after deformation using a feature encoder; decomposing the feature maps of the image pairs into multiple sub-regions; calculating the relevant volumetric feature representations of multiple pairs of image sub-regions before and after deformation, and standardizing the last dimension using a softmax function to obtain the relevant volumetric feature representations of each sub-region; constructing a two-layer learnable neuron module as a neural activation operator to convert the relevant volumetric feature representations of each sub-region into x-direction displacement information and y-direction displacement information, respectively, and then concatenating them to obtain a preliminary predicted displacement field; concatenating all the relevant volumetric feature representations of the sub-regions into a complete relevant volumetric feature representation; and then concatenating the complete relevant volumetric feature representation and the preliminary predicted displacement field along the channel dimension to obtain a feature map. Through multiple downsampling and upsampling operations, the final displacement field prediction result for the image pair is obtained. This method obtains dense and highly accurate displacement field prediction results with a small number of parameters.

Description

Technical Field

[0001] This invention relates to the fields of sensitive image tracking and deep learning technology, specifically to a method and system for perceiving neuromorphic image deformation based on learnable activation operators. Background Technology

[0002] Image motion and deformation estimation is a fundamental task in computer vision. It infers the motion or spatial deformation information of target points in a scene by analyzing pixel information changes between adjacent frames in a single-view or multi-view image sequence. This is crucial for applications such as visual target tracking, action recognition, autonomous driving, visual navigation, and structural behavior monitoring. Typical image motion and deformation estimation methods include optical flow methods and digital image correlation methods, which rely heavily on assumptions of image grayscale conservation and spatial smoothness (e.g., the Lucas-Kanade iterative matching algorithm). However, these assumptions often fail to adequately address complex real-world environments (e.g., lighting variations, low light, image degradation, and occlusion). In recent years, deep learning-based methods, such as FlowNet, RAFT, GMFlow, and their variants, have effectively implemented related work using neural network models that utilize optical flow prediction. Nevertheless, the high accuracy of these models, employing attention mechanisms and calculating relevant feature volumes, typically results in computationally intensive methods with high hardware resource requirements, especially in real-time applications. The training of optical flow models often assumes constant brightness and minimal high-order deformation in the displacement field, using this as a basis for motion estimation. However, in deformation problems, this assumption may not hold true due to the greater complexity of the displacement field and the presence of noise and illumination, which poses additional challenges to displacement field prediction.

[0003] While existing deep learning-based image motion and displacement field estimation models have achieved significant progress in accuracy for displacement field prediction in small deformation problems, they face several challenges in practical deployment, especially when running on resource-constrained development boards. Specifically, these models suffer from large parameter counts, high computational complexity, and high energy consumption, which significantly limits their deployment on edge devices. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a neuromorphic image deformation perception method and system based on learnable activation operators. This method constructs a lightweight deep neural network model for image motion and displacement field estimation based on trainable neural activation operators, obtaining dense and highly accurate displacement field prediction results with a small number of parameters.

[0005] To achieve the above objectives, the present invention provides the following technical solution.

[0006] This invention provides a neuromorphic image deformation perception method based on learnable activation operators, comprising the following steps:

[0007] The feature maps of the image pair before and after deformation are extracted by the feature encoder, and the feature maps of the image pair are decomposed into multiple sub-regions; wherein, the deformed image sub-region in the image pair is based on the size of the image sub-region before deformation by adding the edges of the adjacent sub-regions.

[0008] The relevant volumetric feature representations of multiple pairs of image sub-regions before and after deformation are obtained respectively, and the last dimension is standardized by the softmax function to obtain the relevant volumetric feature representations of each sub-region;

[0009] Two single-layer learnable neuron modules are used as neural activation operators. The volumetric feature representations of each sub-region are converted into displacement information in the x-direction and y-direction, respectively, and then concatenated to obtain the preliminary predicted displacement field. Among them, the B-spline function is used as the one-dimensional activation function between each input node and the output node in the neural activation operator.

[0010] All subregion-related volume feature representations are concatenated into a complete related volume feature representation. Then, the complete related volume feature representation and the initially predicted displacement field are concatenated along the channel dimension to obtain a feature map. The concatenated feature map is downsampled and the predicted displacement field is obtained based on the neural activation operator.

[0011] After concatenating the predicted displacement field with the current feature map, downsampling and displacement field prediction based on neural activation operators are performed sequentially. After multiple executions, upsampling and displacement field prediction based on neural activation operators are performed sequentially on the concatenated feature map. After multiple executions, a displacement field of the same size as the complete related volume feature representation is obtained. The displacement field of the same size as the complete related volume feature representation is expanded by pixels through upsampling to obtain the final displacement field prediction result of the image pair.

[0012] Preferably, the acquisition of the sub-region related volume feature representation includes the following steps:

[0013] The calculation of the volumetric feature representation C of a pair of sub-regions M0 and M1 at corresponding positions in the feature maps of the images before and after deformation is shown in the following formula:

[0014]

[0015] Where D represents the channel dimension of sub-regions M0 and M1 extracted from the feature map;

[0016] The matching distribution C' is obtained by standardizing the last dimension of the relevant volume feature representation C using the softmax function:

[0017] C = softmax(C)

[0018] Among them, the information represented by the last dimension of the sub-region correlation feature is the correlation information between one element in M0 and all elements in M1.

[0019] Preferably, the single-layer neural activation operator includes multiple input nodes and one output node, forming a linear mapping; each input node has an independent one-dimensional activation function between it and the output node; wherein the one-dimensional activation function is a B-spline function.

[0020] Preferably, the step of downsampling the spliced ​​feature map and predicting the predicted displacement field based on a neural activation operator includes the following steps:

[0021] The concatenated feature map is then subjected to convolutional downsampling to obtain the downsampled feature map.

[0022] The shape of the downsampled feature map is transformed to match the input dimension requirement of the neural activation operator, which is then input into the neural activation operator, and the displacement field prediction value is output by the neural activation operator.

[0023] Preferably, the step of sequentially performing upsampling and displacement field prediction based on neural activation operators on the stitched feature maps includes the following steps:

[0024] The concatenated feature map is upsampled by transpose convolution to obtain the upsampled feature map;

[0025] The shape of the upsampled feature map is transformed to match the input dimension requirement of the neural activation operator. The neural activation operator is then input, and the displacement field prediction value is output by the neural activation operator.

[0026] Preferably, the step of upsampling the displacement field of the same size as the complete correlated volume feature representation includes the following steps:

[0027] Construct a weight generation network, consisting of sequentially connected convolutional layers, ReLU activation functions, and convolutional layers;

[0028] The feature map obtained by the final upsampling and the displacement field predicted based on the neural activation operator are input into the weight generation network to obtain the weights;

[0029] For each original pixel in the displacement field and its 8 surrounding neighboring pixels, each extended pixel in the displacement field is obtained by weighting the original pixel and the surrounding 8 neighboring pixels, thus obtaining the displacement field prediction result.

[0030] Preferably, the method further includes training using an augmented optical flow dataset, comprising the following steps:

[0031] An optical flow dataset is constructed, consisting of a reference image and a deformable image.

[0032] Perform local or global deformation on the deformation diagram to obtain the displacement field and deformation diagram of complex deformation;

[0033] The new displacement field is obtained by superimposing the complex deformation displacement field with the original displacement field; the new displacement field and deformation map are added to the dataset used to form the expanded optical flow dataset.

[0034] Preferably, the method further includes constructing a loss function based on displacement field loss and displacement field gradient loss for training, including the following steps:

[0035] The displacement field loss is:

[0036]

[0037] The displacement field gradient loss is:

[0038]

[0039] Where f represents the predicted displacement field, g represents the actual displacement field, m and n are the displacements in two directions, and λ is the weight.

[0040] Preferably, obtaining the displacement field of the complex deformation includes the following steps:

[0041] By combining random rigid body translation, rotation, stretching, compression, and shearing, and using local deformation represented by a 2D Gaussian function, a 2D displacement field is defined for each deformation diagram:

[0042]

[0043] In the formula, the subscripts x and y represent the horizontal and vertical directions; u and v represent the displacements in the x and y directions at coordinate (i,j); t represents the translation; k represents the degree of tensile and compressive deformation; γ represents the degree of shear deformation; θ represents the rotational deformation angle; a represents the Gaussian deformation amplitude; (i0, j0) and (i1, j1) represent the Gaussian deformation center positions; c x0 ,c y0 ,c x1 ,c y1 This represents the standard deviation of Gaussian deformation.

[0044] A neuromorphic image deformation perception system based on learnable activation operators, the system comprising:

[0045] A feature encoder is used to extract feature maps of image pairs before and after deformation.

[0046] The correlation calculation module is used to decompose the feature maps of image pairs into multiple sub-regions. The deformed image sub-region in the image pair is based on the size of the undeformed image sub-region by adding the edges of the adjacent sub-regions. It is also used to calculate the correlation volume feature representations of multiple pairs of image sub-regions before and after deformation, and to standardize the last dimension by the softmax function to obtain the correlation volume feature representations of each sub-region.

[0047] The relevant feature activation module is used to use two single-layer learnable neuron modules as neural activation operators to convert the relevant volume feature representations of each sub-region into displacement information in the x-direction and y-direction, respectively, and then concatenate them to obtain the preliminary predicted displacement field; wherein, the B-spline function is used as the one-dimensional activation function between each input node and the output node in the neural activation operator;

[0048] The displacement field optimization module is used to concatenate the relevant volume feature representations of all sub-regions into a complete relevant volume feature representation, and then concatenate the complete relevant volume feature representation and the initially predicted displacement field along the channel dimension to obtain a feature map. The concatenated feature map is downsampled, and a predicted displacement field is obtained based on a neural activation operator. It is also used to concatenate the predicted displacement field with the current feature map, and then sequentially perform downsampling and displacement field prediction based on a neural activation operator. After multiple executions, the concatenated feature map is sequentially upsampled and subjected to displacement field prediction based on a neural activation operator. After multiple executions, a displacement field of the same size as the complete relevant volume feature representation is obtained.

[0049] The upsampling module is used to expand the displacement field of the same size as the complete related volume feature representation by pixels through upsampling to obtain the displacement field prediction result of the final image pair.

[0050] The beneficial effects of this invention are:

[0051] This invention proposes a neuromorphic image deformation perception method and system based on learnable activation operators. This method constructs a lightweight deep neural network model for image motion and displacement field estimation based on activation functions of neural operators. By combining multiple downsampling and upsampling operations, it achieves dense and high-accuracy displacement field prediction results with a small number of parameters. Furthermore, the displacement fields in the model training dataset, including complex high-order deformations, can effectively learn displacement field predictions for small deformation problems. This overcomes the limitation of deploying deep learning optical flow models on low-power computing platforms, enabling the displacement field prediction model to be deployed on a wider range of embedded platforms. Attached Figure Description

[0052] Figure 1 This is a flowchart of the neuromorphic image deformation perception method based on learnable activation operators according to an embodiment of the present invention;

[0053] Figure 2This is a diagram of a lightweight deep neural network model architecture for image motion and displacement field estimation based on trainable neural activation operators, according to an embodiment of the present invention.

[0054] Figure 3 This is a comparison diagram of the model-predicted displacement field and the actual displacement field in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0056] Example

[0057] The flowchart of the neuromorphic image deformation perception method based on learnable activation operators proposed in this invention is as follows: Figure 1 As shown, it includes the following steps:

[0058] A lightweight deep neural network model for image motion and displacement field estimation based on trainable neural activation operators is constructed, including a feature encoder, a correlation calculation module, a correlation feature activation module, a displacement field optimization module, and an upsampling module. The model architecture diagram is shown below. Figure 2 As shown.

[0059] S1: Extract feature maps of the image pair before and after deformation through a feature encoder, and decompose the feature maps of the image pair into multiple sub-regions; wherein, the deformed image sub-region in the image pair is based on the size of the image sub-region before deformation by adding the edges of the adjacent sub-regions.

[0060] S2: Through the correlation calculation module, the correlation volume feature representations of multiple pairs of image sub-regions before and after deformation are obtained respectively, and the last dimension is standardized by the softmax function to obtain the correlation volume feature representations of each sub-region.

[0061] S3: Through the relevant feature activation module, two single-layer learnable neuron modules are used as neural activation operators to convert the relevant volume feature representations of each sub-region into x-direction displacement information and y-direction displacement information, respectively, and then concatenate them to obtain the preliminary predicted displacement field; among them, the B-spline function is used as the one-dimensional activation function between each input node and the output node in the neural activation operator.

[0062] S4: Through the displacement field optimization module, the relevant volume feature representations of all sub-regions are spliced ​​into a complete relevant volume feature representation. Then, the complete relevant volume feature representation and the preliminary predicted displacement field are spliced ​​together in the channel dimension to obtain a feature map. The spliced ​​feature map is downsampled and the predicted displacement field is obtained based on the neural activation operator.

[0063] S5: After concatenating the predicted displacement field with the current feature map, perform downsampling and displacement field prediction based on neural activation operators in sequence; after multiple executions, perform upsampling and displacement field prediction based on neural activation operators on the concatenated feature map in sequence; after multiple executions, obtain a displacement field with the same size as the complete related volume feature representation.

[0064] S6: Through the upsampling module, the displacement fields of the same size as the complete related volume feature representations are expanded by pixels through upsampling to obtain the displacement field prediction results of the final image pair.

[0065] Specifically, in this embodiment:

[0066] The feature encoder consists of multiple layers of convolutional neural networks. The input images I0 and I1 are input into the encoder, and the encoder extracts features to obtain feature maps F0 and F1 of I0 and I1, with shapes [B, C, H / 4, W / 4].

[0067] The correlation calculation module decomposes F0 into multiple sub-regions of shape [B,C,4,4]. The combined shape of the sub-regions after decomposition of F0 is [B,C,H*W / 256,4,4]. Simultaneously, F1 needs to be decomposed into multiple sub-regions of shape [B,C,6,6]. A 6*6 sub-region is a 4*4 sub-region plus the edges of its adjacent sub-regions. The combined shape of the sub-regions after decomposition of F1 is [B,C,H*W / 256,6,6].

[0068] For each pair of sub-regions M0 and M1 at corresponding positions of F0 and F1, calculate the relevant volumetric feature representation as shown in the following formula. The shape of M0 is [B, C, 4*4], and the shape of M1 is [B, C, 6*6]. The shape of C is [B, 16, 36].

[0069]

[0070] The calculated relevant volume feature representations are standardized using the softmax function on the last dimension of C to obtain the matching distribution C', as shown in the following equation:

[0071] C' = softmax(C)

[0072] The last dimension of C' represents the correlation information between a certain element in M0 and all elements in M1.

[0073] The relevant feature activation module needs to transform the feature matching information of feature volume C' into displacement information. This step is achieved through a specific, trainable neural activation operator with n-dimensional input and 1-dimensional output. This neural activation operator is a fully connected network structure designed based on the Kolmogorov-Arnold representation theorem. Unlike traditional multilayer perceptrons (MLPs), the neural activation operator uses learnable activation functions on the network weights. These activation functions are typically parameterized univariate functions, such as B-spline functions. This design allows deep neural network blocks to achieve higher expressive power with fewer parameters and possesses good interpretability. This design enables the system to learn the mapping relationship between feature matching information and displacement information, thus making it more flexible and accurate in processing new data.

[0074] The single-layer neural activation operator consists of 36 input nodes and 1 output node, forming a linear mapping of (36,1). Each input node has an independent one-dimensional activation function, rather than fixed traditional weights, connecting it to the output node. By adjusting the parameters of these one-dimensional activation functions, the neural activation operator module can adaptively learn a complex nonlinear mapping from n-dimensional input features to displacement information. This learning process allows the module to capture subtle patterns in the input data, thereby generating accurate displacement information.

[0075] In this embodiment, a B-spline function is chosen as the activation function. B-spline functions are a class of piecewise polynomial functions that offer high flexibility while maintaining continuity and smoothness. The coefficients of each polynomial segment can be optimized through the training process to minimize prediction error. The spline order is 3, and the grid size is 10. The activation function is shown in the following equation, where x represents all elements of the last dimension of C'.

[0076] φ = w*spline(x)

[0077] A complete matching feature activation module can be represented by the following formula:

[0078]

[0079] All sub-region related volume feature representations C' are input into two different related feature activation modules. The two modules are used to convert feature-related information into displacement information in the x-direction and displacement information in the y-direction, respectively. After concatenation, the preliminary predicted displacement field u0 can be obtained. The shape of u0 is [B,2,H / 4,W / 4].

[0080] Then, all sub-region related volume feature representations are concatenated into a complete related volume feature representation C1. The shape of the complete related volume feature representation is [B,36,H / 4,W / 4]. The complete related volume feature representation C1 is concatenated with the initial predicted displacement field in the channel dimension to obtain feature map F2. F2 is then input into the displacement field optimization module.

[0081] The first layer input F2 of the displacement field optimization module is downsampled by a 3*3 convolutional downsampling module to obtain feature map F3, with the shape of feature map F3 being [B, 256, H / 8, W / 8]. Simultaneously, input F3 enters the displacement field prediction module to obtain displacement field u1. The displacement field input module first transforms the shape of the input feature map into [B*H / 8*W / 8, 256], and then passes it through a neural activation operator module with an input dimension of 256 and an output dimension of 2, which then transforms the shape of the displacement field back to [B, 2, H / 8, W / 8].

[0082] Then, feature map F3 and displacement field u1 are concatenated along the channel dimension to obtain feature map F4. The second layer input F4 of the displacement field optimization module is downsampled through a 3*3 convolutional downsampling module to obtain feature map F5. The shape of feature map F5 is [B, 512, H / 16, W / 16]. At the same time, input F5 enters the displacement field prediction module to obtain displacement field u2. The displacement field input module first changes the shape of the input feature map to [B*H / 16*W / 16, 512], and then passes it through a KANLinear module with an input dimension of 256 and an output dimension of 2, and then changes the shape of the displacement field back to [B, 2, H / 16, W / 16].

[0083] Then, feature map F5 and displacement field u2 are concatenated along the channel dimension to obtain feature map F6. The third layer input F6 of the displacement field optimization module is upsampled through the transposed convolution upsampling module with a 3*3 convolution kernel to obtain feature map F7. The shape of feature map F7 is [B, 256, H / 8, W / 8]. At the same time, input F7 enters the displacement field prediction module to obtain displacement field u3. The displacement field input module first changes the shape of the input feature map to [B*H / 8*W / 8, 256]. Then, it passes through the neural activation operator module with an input dimension of 256 and an output dimension of 2, and then changes the shape of the displacement field back to [B, 2, H / 8, W / 8].

[0084] The feature map F7 and displacement field u3 are then concatenated along the channel dimension to obtain feature map F8. The fourth layer input F8 of the displacement field optimization module is upsampled through a transposed convolution upsampling module with a 3x3 convolution kernel to obtain feature map F9. The shape of feature map F9 is [B, 36, H / 4, W / 4]. Simultaneously, input F9 enters the displacement field prediction module to obtain displacement field u4. The displacement field input module first transforms the shape of the input feature map to [B*H / 4*W / 4, 256], then passes it through a neural activation operator module with an input dimension of 256 and an output dimension of 2, finally transforming the displacement field shape back to [B, 2, H / 4, W / 4]. The module using neural activation operators for direct displacement prediction significantly improves prediction accuracy.

[0085] Then, the shape of the displacement field u4 is transformed into [B,2,H,W] through the upsampling module, which is the calculated displacement field u.

[0086] The upsampling module performs a 4x upsampling. The process can be simply described as follows: each pixel is expanded into 8x8 pixels. Specifically, each expanded pixel is obtained by weighting the original pixel and its 8 neighboring pixels (a total of 9 pixels). The weights are generated by the network. The network input for weight generation includes F9 and u4, which are passed through a 3x3 convolutional layer, a ReLU activation function, and another 3x3 convolutional layer to obtain weights with the shape [B, 4x4x9, H / 4, W / 4].

[0087] In this embodiment, the training of a lightweight deep neural network model for image motion and displacement field estimation based on trainable neural activation operators includes the following steps:

[0088] Construct a dataset for training the model. The input to the model is a pair of images and the displacement field between the images. In this embodiment, the FlyingChairs dataset can be selected.

[0089] The images in the optical flow dataset are divided into reference images and deformed images. In this embodiment, the deformed image needs to be subjected to more complex deformation. The deformation can be applied locally or globally. Then, the displacement field of the applied complex deformation is superimposed with the original displacement field, and the new displacement field and the new deformed image are added to the dataset used.

[0090] Generate additional deformation displacement fields: Define a 2D displacement field for each sample image by combining random rigid body translation, rotation, stretching / compression, shearing, and local deformation represented by a 2D Gaussian function.

[0091]

[0092] In the formula, the subscripts x and y represent the horizontal and vertical directions; u and v represent the displacements in the x and y directions at coordinate (i,j); t represents the translation; k represents the degree of tensile and compressive deformation; γ represents the degree of shear deformation; θ represents the rotational deformation angle; a represents the Gaussian deformation amplitude; (i0, j0) and (i1, j1) represent the Gaussian deformation center positions; c x0 ,c y0 ,c x1 ,c y1 This represents the standard deviation of Gaussian deformation.

[0093] The loss function used for model training takes into account both the accuracy and smoothness of the displacement field. The loss function L is divided into displacement field loss and displacement field gradient loss.

[0094] L = L u +L grad

[0095] Simultaneously, the displacement difference and displacement gradient between each pixel in the reference image and the corresponding pixel in the deformed image are calculated pixel by pixel. This pixel and its eight neighboring points form a subset for calculating the difference. The center point and its neighboring points have different weights, and the difference between neighboring points should be multiplied by λ when adding the loss.

[0096] The displacement field loss is:

[0097]

[0098] The displacement field gradient loss is:

[0099]

[0100] Where f represents the predicted displacement field, g represents the actual displacement field, and m and n are the displacements in two directions.

[0101] Comparison between the displacement field predicted by the model and the actual displacement field Figure 3 As shown, the prediction results can achieve sub-pixel accuracy. This demonstrates the effectiveness of learnable neural activation operators in predicting displacement fields.

[0102] The above is one embodiment of the neuromorphic image deformation sensing method based on learnable activation operators provided in this embodiment. Based on the same idea, this embodiment also provides a corresponding neuromorphic image deformation sensing system based on learnable activation operators. Specific limitations of the neuromorphic image deformation sensing system based on learnable activation operators can be found in the limitations of the neuromorphic image deformation sensing method based on learnable activation operators described above, and will not be repeated here. Each module in the above-described neuromorphic image deformation sensing system based on learnable activation operators can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0103] This embodiment also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided method is a neuromorphic image deformation perception method based on learnable activation operators.

[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0105] The construction of a small deformation displacement field prediction system includes the following steps:

[0106] A monocular camera is connected to a power supply and a low-power computing platform via cable, simultaneously capturing image data and transmitting it to the platform in real time. Within the computing platform, the incoming image data is sequentially processed using a lightweight, small-deformation displacement field prediction model to obtain continuous predicted displacement fields, which are then stored in a storage medium. The lightweight nature of the model ensures fast real-time computation, enabling instantaneous displacement field prediction from continuous input image data. Users can subsequently retrieve the required displacement field data from the storage medium or sum multiple sets of displacement field data to obtain long-term displacement field prediction results.

[0107] The trained model has a total of 13.9M parameters. Furthermore, by avoiding the use of attention mechanisms and iterative optical flow updates, the model has low memory requirements.

[0108] The computing platforms available are: the NVIDIA Jetson Nano, with 4GB or 2GB of LPDDR4 RAM; the NVIDIA Jetson Xavier NX, with 8GB of 128-bit LPDDR4x RAM, offering more powerful computing capabilities; the Raspberry Pi 4, with a standard configuration of 4GB, 8GB, or 16GB of LPDDR4 SDRAM; and the Qualcomm Snapdragon 820 / 845 / 865 development boards, which, based on Qualcomm processors, offer RAM ranging from 4GB to 16GB and integrate Hexagon DSP and Adreno GPU.

[0109] All of the above development boards can run lightweight deep neural network models of the ONNX type based on trainable neural activation operators for image motion and displacement field estimation.

[0110] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A neuromorphic image deformation perception method based on learnable activation operators, characterized in that, Includes the following steps: The feature maps of the image pair before and after deformation are extracted by the feature encoder, and the feature maps of the image pair are decomposed into multiple sub-regions; wherein, the deformed image sub-region in the image pair is based on the size of the image sub-region before deformation by adding the edges of the adjacent sub-regions. The relevant volumetric feature representations of multiple pairs of image sub-regions before and after deformation are obtained respectively, and the last dimension is standardized by the softmax function to obtain the relevant volumetric feature representations of each sub-region; Two learnable neuron modules are constructed as neural activation operators. The volumetric feature representations of each sub-region are converted into displacement information in the x-direction and y-direction, respectively, and then concatenated to obtain the preliminary predicted displacement field. Among them, the B-spline function is used as the one-dimensional activation function between each input node and the output node in the neural activation operator. All subregion-related volume feature representations are concatenated into a complete related volume feature representation. Then, the complete related volume feature representation and the initially predicted displacement field are concatenated along the channel dimension to obtain a feature map. The concatenated feature map is downsampled and the predicted displacement field is obtained based on the neural activation operator. After concatenating the predicted displacement field with the current feature map, downsampling and displacement field prediction based on neural activation operators are performed sequentially. After multiple executions, upsampling and displacement field prediction based on neural activation operators are performed sequentially on the concatenated feature map. After multiple executions, a displacement field of the same size as the complete related volume feature representation is obtained. The displacement field of the same size as the complete related volume feature representation is expanded by pixels through upsampling to obtain the final displacement field prediction result of the image pair.

2. The neuromorphic image deformation perception method based on learnable activation operators according to claim 1, characterized in that, The acquisition of the relevant volumetric feature representation of the sub-region includes the following steps: A pair of sub-regions at corresponding positions in the feature maps of the images before and after deformation. , Relevant body feature representation The calculation is shown in the following formula: ； in, D Subregions extracted from feature maps M 0, M 1 channel dimension; Relevant volume features are represented using the softmax function. The last dimension is standardized to obtain the matching distribution. : ； Among them, the sub-region related body features are represented The information represented by the last dimension is One of the elements in The relevance information of all elements in the text.

3. The neuromorphic image deformation perception method based on learnable activation operators according to claim 1, characterized in that, The neural activation operator includes multiple input nodes and one output node, forming a linear mapping; each input node has an independent one-dimensional activation function between it and the output node; wherein the one-dimensional activation function is a B-spline function.

4. The neuromorphic image deformation perception method based on learnable activation operators according to claim 1, characterized in that, The process of downsampling the spliced ​​feature map and predicting the displacement field based on a neural activation operator includes the following steps: The concatenated feature map is then subjected to convolutional downsampling to obtain the downsampled feature map. The shape of the downsampled feature map is transformed to match the input dimension requirements of the neural activation operator, which is then input into the neural activation operator, and the displacement field prediction value is output by the neural activation operator module.

5. The neuromorphic image deformation perception method based on learnable activation operators according to claim 1, characterized in that, The process of sequentially upsampling and performing displacement field prediction based on neural activation operators on the stitched feature map includes the following steps: The concatenated feature map is upsampled by transpose convolution to obtain the upsampled feature map; The shape of the upsampled feature map is transformed to match the input dimension requirement of the neural activation operator. The neural activation operator is then input, and the displacement field prediction value is output by the neural activation operator.

6. The neuromorphic image deformation perception method based on learnable activation operators according to claim 1, characterized in that, The step of upsampling the displacement field, which represents the same size as the complete correlated volume feature, includes the following steps: Construct a weight generation network, consisting of sequentially connected convolutional layers, ReLU activation functions, and convolutional layers; The feature map obtained by the final upsampling and the displacement field predicted based on the neural activation operator are input into the weight generation network to obtain the weights; For each original pixel in the displacement field and its 8 surrounding neighboring pixels, each extended pixel in the displacement field is obtained by weighting the original pixel and the surrounding 8 neighboring pixels, thus obtaining the displacement field prediction result.

7. The neuromorphic image deformation perception method based on learnable activation operators according to claim 1, characterized in that, It also includes training using an expanded optical flow dataset, comprising the following steps: An optical flow dataset is constructed, consisting of a reference image and a deformable image. Perform local or global deformation on the deformation diagram to obtain the displacement field and deformation diagram of complex deformation; The new displacement field is obtained by superimposing the complex deformation displacement field with the original displacement field; the new displacement field and deformation map are added to the dataset used to form the expanded optical flow dataset.

8. The neuromorphic image deformation perception method based on learnable activation operators according to claim 1, characterized in that, It also includes constructing a loss function based on displacement field loss and displacement field gradient loss for training, including the following steps: The displacement field loss is: ； The displacement field gradient loss is: ； in, Indicates the predicted displacement field, Represents the true displacement field; m , n This refers to the displacement in two directions; For weights.

9. The neuromorphic image deformation perception method based on learnable activation operators according to claim 7, characterized in that, The acquisition of the displacement field of the complex deformation includes the following steps: By combining random rigid body translation, rotation, stretching, compression, and shearing, and using local deformation represented by a 2D Gaussian function, a 2D displacement field is defined for each deformation diagram: ； In the formula, the subscript x and y Represents horizontal and vertical directions; u , v Indicates the coordinates ( i , j ) x and y Displacement in the direction; t Indicates the amount of translation; k Indicates the degree of tensile and compressive deformation; γ Indicates the degree of shear deformation; θ Indicates the angle of rotational deformation; a Indicates the Gaussian deformation amplitude; , Indicates the location of the Gaussian deformation center. This represents the standard deviation of Gaussian deformation.

10. A neuromorphic image deformation perception system based on learnable activation operators, characterized in that, The system includes: A feature encoder is used to extract feature maps of image pairs before and after deformation. The correlation calculation module is used to decompose the feature maps of image pairs into multiple sub-regions. The deformed image sub-region in the image pair is based on the size of the undeformed image sub-region by adding the edges of the adjacent sub-regions. It is also used to calculate the correlation volume feature representations of multiple pairs of image sub-regions before and after deformation, and to standardize the last dimension by the softmax function to obtain the correlation volume feature representations of each sub-region. The relevant feature activation module is used to use two single-layer learnable neuron modules as neural activation operators to convert the relevant volume feature representations of each sub-region into displacement information in the x-direction and y-direction, respectively, and then concatenate them to obtain the preliminary predicted displacement field; wherein, the B-spline function is used as the one-dimensional activation function between each input node and the output node in the neural activation operator; The displacement field optimization module is used to concatenate the relevant volume feature representations of all sub-regions into a complete relevant volume feature representation, and then concatenate the complete relevant volume feature representation and the initially predicted displacement field along the channel dimension to obtain a feature map. The concatenated feature map is downsampled, and a predicted displacement field is obtained based on a neural activation operator. It is also used to concatenate the predicted displacement field with the current feature map, and then sequentially perform downsampling and displacement field prediction based on a neural activation operator. After multiple executions, the concatenated feature map is sequentially upsampled and subjected to displacement field prediction based on a neural activation operator. After multiple executions, a displacement field of the same size as the complete relevant volume feature representation is obtained. The upsampling module is used to expand the displacement field of the same size as the complete related volume feature representation by pixels through upsampling to obtain the displacement field prediction result of the final image pair.

Images