A dynamic convolution method and device based on Lie group parameterization and electronic equipment

Abstract

The application belongs to the field of computer vision and deep learning, and discloses a dynamic convolution method based on Lie group parameterization, a device and electronic equipment, the method comprising: obtaining an image feature map to be processed; predicting a deformation parameter tensor based on the image feature map, and remodeling the deformation parameter tensor into multiple groups of rotation parameters and translation parameters; constructing a Lie algebra matrix corresponding to each group of rotation parameters and translation parameters, and converting the Lie algebra matrix into an affine transformation matrix through a matrix exponential mapping operation; performing geometric transformation and resampling on a pre-stored base convolution kernel based on the affine transformation matrix, to obtain multiple dynamically deformable convolution kernels subjected to geometric transformation; and performing dynamic convolution operation on the image feature map through the dynamically deformable convolution kernel, to generate an image feature map subjected to geometric adaptive convolution processing. The application ensures that the mapping from the deformation parameter tensor to the affine transformation matrix is continuous and smooth, and the finally generated image feature map can accurately express the features of a deformed object.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and deep learning, and particularly relates to a dynamic convolution method based on Lie group parameterization, a device and an electronic equipment. BACKGROUND

[0002] The convolutional neural network (CNN) is a mainstream framework for current computer vision tasks. The traditional CNN uses a fixed shape and size of convolution kernel. If the object included in the image has a standard shape, the traditional CNN can accurately extract the features of the object.

[0003] However, in actual application scenarios, the object often undergoes complex geometric transformations such as rotation, scaling and non-rigid deformation due to changes in viewing angle and its own movement. The traditional rigid convolution kernel is difficult to effectively adapt to such changes, resulting in inaccurate feature extraction and affecting the performance of the final task. SUMMARY

[0004] The purpose of the present application is to solve the technical problem that the traditional rigid convolution kernel is difficult to effectively adapt to the deformation of the object, resulting in inaccurate feature extraction. The present application maps the deformation parameter tensor to a Lie algebra matrix, and then generates an affine transformation matrix from the Lie algebra matrix through matrix exponential mapping operation. The affine transformation matrix is used to transform and resample the base convolution kernel to obtain a dynamically deformable convolution kernel, ensuring the stability of the topological structure in the convolution sampling process. The final generated image feature map can accurately express the features of the deformed object, which helps to improve the performance of the visual task.

[0005] In a first aspect, an embodiment of the present application provides a dynamic convolution method based on Lie group parameterization, which comprises:

[0006] Obtaining an image feature map to be processed, which is extracted from an image including a deformed object;

[0007] Predicting a deformation parameter tensor based on the image feature map, and reshaping the deformation parameter tensor into multiple groups of rotation parameters and translation parameters, the translation parameters including horizontal translation parameters and vertical translation parameters;

[0008] Constructing a Lie algebra matrix corresponding to each group of rotation parameters and translation parameters, and converting the Lie algebra matrix into an affine transformation matrix in the Lie group space through matrix exponential mapping operation;

[0009] Geometrically transforming and resampling a pre-stored base convolution kernel based on the affine transformation matrix to obtain multiple dynamically deformable convolution kernels that have been geometrically transformed;

[0010] The image feature map is dynamically convolved by the dynamic deformable convolution kernel to generate an image feature map processed by geometric self-adaptive convolution.

[0011] Optionally, the data structure of the image feature map is a four-dimensional tensor, and the four-dimensional tensor includes batch size B, channel number C, height H of the image feature map, and width W of the image feature map.

[0012] The deformation parameter tensor is predicted based on the image feature map, and the deformation parameter tensor is reshaped into multiple groups of rotation parameters and translation parameters, including:

[0013] The parameter prediction network is called to process the image feature map to generate a deformation parameter tensor with a two-dimensional tensor data structure, and the parameter prediction network includes an adaptive pooling layer, a first convolution layer, an activation function, and a second convolution layer, and the two-dimensional tensor includes batch size B and parameter dimension 3G.

[0014] Based on the batch size B and the parameter dimension 3G, the deformation parameter tensor is reshaped into G groups of rotation parameters and translation parameters , , wherein G is the number of transformation groups, is a horizontal translation parameter, is a vertical translation parameter.

[0015] Optionally, a Lie algebra matrix corresponding to each group of rotation parameters and translation parameters is constructed, and the Lie algebra matrix is converted into an affine transformation matrix in Lie group space through matrix exponential mapping operation, including:

[0016] For each sample b in the batch and each transformation group g , the rotation parameters and the translation parameters are denoted as , , , and the obtained Lie algebra matrix is:

[0017] ;

[0018] The Lie algebra matrix is converted into a 3×3 affine transformation matrix in Lie group space through matrix exponential mapping operation:

[0019] .

[0020] Optionally, the pre-stored base convolution kernel is geometrically transformed and resampled based on the affine transformation matrix to obtain multiple dynamically deformable convolution kernels that are geometrically transformed, including:

[0021] obtaining one or more base convolution kernels stored in advance;

[0022] generating deformed sampling network coordinates by using a standard affine network on the affine transformation matrix, and resampling the base convolution kernel by using the sampling network coordinates through a grid sampling operation to obtain a plurality of dynamic deformable convolution kernels of different scales.

[0023] Optionally, the dynamic convolution operation on the image feature map by using the dynamic deformable convolution kernel to generate a geometric self-adaptive convolution processed image feature map comprises:

[0024] applying the dynamic deformable convolution kernel to corresponding groups of the image feature map in a grouped convolution manner to obtain a plurality of convolution results;

[0025] performing an average operation on the plurality of convolution results to calculate an output image feature map fused with geometric deformation information.

[0026] Optionally, the method further comprises:

[0027] applying the geometric self-adaptive convolution processed image feature map to the field of computer vision, wherein the field of computer vision comprises target detection, target segmentation and pose estimation.

[0028] In a second aspect, an embodiment of the present application provides a dynamic convolution device based on Lie group parameterization, which comprises:

[0029] an image feature map acquisition module, configured to acquire an image feature map to be processed, wherein the image feature map is extracted from an image comprising a deformed object;

[0030] a deformation parameter generation module, configured to predict a deformation parameter tensor based on the image feature map, and reshape the deformation parameter tensor into a plurality of groups of rotation parameters and translation parameters, wherein the translation parameters comprise translation parameters in a horizontal direction and translation parameters in a vertical direction;

[0031] a Lie algebra matrix and affine transformation matrix generation module, configured to construct a Lie algebra matrix corresponding to each group of rotation parameters and translation parameters, and convert the Lie algebra matrix into an affine transformation matrix in a Lie group space through a matrix exponential mapping operation;

[0032] a dynamic deformable convolution kernel generation module, configured to perform geometric transformation and resampling on a base convolution kernel stored in advance based on the affine transformation matrix to obtain a plurality of dynamic deformable convolution kernels subjected to geometric transformation;

[0033] An image feature map convolution and output module is configured to perform a dynamic convolution operation on the image feature map by using the dynamic deformable convolution kernel, to generate a geometric self-adaptive convolution processed image feature map.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, comprising:

[0035] at least one processor;

[0036] a memory for storing instructions executable by the at least one processor;

[0037] The at least one processor is configured to execute the instructions to implement the method of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method of the first aspect.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, when the computer program is executed by a processor, the method of the first aspect is implemented.

[0040] The technical scheme provided by the embodiment of the present application acquires an image feature map to be processed, which is extracted from an image including a deformed object; a deformed parameter tensor is predicted based on the image feature map, and the deformed parameter tensor is reshaped into a plurality of groups of rotation parameters and translation parameters, the translation parameters including horizontal direction translation parameters and vertical direction translation parameters; a Lie algebra matrix corresponding to each group of rotation parameters and translation parameters is constructed, and the Lie algebra matrix is converted into an affine transformation matrix in a Lie group space through a matrix exponential mapping operation; this process uses the mathematical properties of the Lie group to generate an affine transformation matrix with a differential homeomorphism characteristic, ensuring the stability of the topological structure in the convolution sampling process, and ensuring that the mapping from the deformed parameter tensor to the affine transformation matrix is continuous and smooth, thereby ensuring the differential homeomorphism characteristic of the transformation.

[0041] Next, the pre-stored base convolution kernel is geometrically transformed and resampled based on the affine transformation matrix to obtain a plurality of geometrically transformed dynamic deformable convolution kernels; and the image feature map is dynamically convolved by the dynamic deformable convolution kernel to generate an image feature map processed by geometrically adaptive convolution. Since the sampling region of the dynamic deformable convolution kernel is adaptive, the sampling position and size can be adjusted according to the content of the input image feature map, so that the shape of the deformed object can be better captured. The output of the deformable convolution is obtained by combining the outputs of different sampling regions, and finally the image feature map processed by geometrically adaptive convolution is generated. Since the finally output image feature map accurately fuses the geometric deformation information, the finally output image feature map can accurately express the features of the deformed object, which helps to improve the performance of the visual task. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 It is a schematic diagram of the overall technical solution of the embodiment of the application;

[0043] Figure 2 It is a flowchart of a dynamic convolution method based on Lie group parameterization provided by the embodiment of the application;

[0044] Figure 3 It is a comparison diagram of a traditional DCN and a deformation field based on Lie group parameterization provided by the embodiment of the application;

[0045] Fig. 4(a) is a visualization effect diagram of the effect of different rotation parameters in the SE(2) Lie group on the original base convolution kernel, taking the original base convolution kernel as a reference;

[0046] Fig. 4(b) is a visualization effect diagram of the effect of different translation parameters in the SE(2) Lie group on the base convolution kernel, taking the base convolution kernel rotated by 90° as a reference;

[0047] Figure 5 It is a structural schematic diagram of a dynamic convolution device based on Lie group parameterization provided by the embodiment of the application;

[0048] Figure 6 It is a structural schematic diagram of an electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0049] The application will be described in detail below through embodiments.

[0050] The convolutional neural network (CNN) is the mainstream framework of current computer vision tasks. The traditional CNN uses a fixed shape and size of convolution kernel. If the object included in the image has a standard shape, the traditional CNN can accurately extract the features of the object.

[0051] However, in real-world applications, objects often undergo complex geometric transformations such as rotation, scaling, and non-rigid deformation due to changes in viewpoint and their own motion. Traditional rigid convolutional kernels struggle to adapt effectively to these changes, leading to inaccurate feature extraction and impacting the final task performance.

[0052] To address this issue, existing technologies have proposed Deformable Convolutional Networks (DCNs). DCNs enable sampling points to dynamically conform to the actual contours of objects by superimposing a learnable two-dimensional offset on the sampling points of standard convolutions.

[0053] However, existing DCNs have the following key shortcomings:

[0054] 1. Lack of strict geometric constraints. The offset field learned by DCN is free and unconstrained. When the object undergoes drastic deformation, this unconstrained offset may cause sampling points to jump randomly or overlap, thereby destroying the topological continuity of the feature map in local regions and producing feature folding.

[0055] 2. Feature discontinuity and performance instability. Due to the disruption of the aforementioned topology, the extracted features may be discontinuous, which can mislead the learning of subsequent network layers, especially in tasks requiring high-precision pixel alignment (such as fine segmentation and pose estimation), leading to performance degradation or training instability.

[0056] Therefore, designing a novel convolution method that can dynamically adapt to object deformation while fundamentally ensuring the continuity of geometric transformations and topological stability is a technical challenge that urgently needs to be solved in this field.

[0057] In view of this, the purpose of this invention is to address the technical problems of feature discontinuity and topological destruction caused by the lack of strict geometric constraints in the prior art, and to provide a dynamic convolution method, apparatus and electronic device based on Lie group parameterization, which aims to generate feature representations that are more geometrically consistent, more topologically stable and therefore more robust to downstream vision tasks, for application in computer vision fields such as object detection and image segmentation.

[0058] To ensure clarity, the overall technical solution of this invention will first be described in detail below, such as... Figure 1 As shown, the overall technical solution of this invention embodiment may include the following steps:

[0059] Step 1: Obtain the image feature map to be processed. The image feature map is extracted from the image including deformed objects, and the data structure of the image feature map is a four-dimensional tensor (B,C,H,W), where B represents the batch size, C is the number of channels, H is the height of the image feature map, and W is the width of the image feature map.

[0060] Step 2: Compute the image feature map in Step 1 by a parameter prediction network to extract high-level semantic information therefrom to generate a deformation parameter tensor θ , which has a data structure of a two-dimensional tensor including a batch size B and a parameter dimension 3G, and further compute the deformation parameter tensor θ , which is further decomposed to output a parameter tensor with a shape of (B, G, 3), where G is the number of transformation groups, i.e., G groups of rotation parameters and translation parameters (including horizontal translation parameters and vertical translation parameters) are obtained for each sample in the batch b and each transformation group g , where the rotation parameters and translation parameters are denoted as , , , which are used as parameters for Lie group transformation in the subsequent steps. The parameter prediction network can include an adaptive average pooling layer, a first convolutional layer, an activation function, and a second convolutional layer.

[0061] Step 3: Generate a Lie algebra matrix. Specifically, for each group of rotation parameters and translation parameters, a Lie algebra matrix ξ is constructed in the memory.

[0062] Step 4: Map the Lie algebra matrix ξ to an affine transformation matrix ξ in a two-dimensional special Euclidean group (SE(2) Lie Group) by performing a matrix exponential mapping operation (exp( M ). The data structure of the affine transformation matrix M is a three-dimensional matrix, and the data processing manner of this step ensures that the mapping from the deformation parameter tensor to the affine transformation matrix is continuous and smooth, thereby ensuring the differential homeomorphism property of the transformation.

[0063] Step 5: Generate a dynamic convolution kernel. Specifically, the processing object of this step is the affine transformation matrix M and a preset base kernel data B 0 (e.g., a unit impulse function) in the memory. The specific processing manner is that the processor performs an affine grid (affine_grid) generation and grid sampling (grid_sample) operation. The affine grid generates a deformed sampling grid coordinate according to the affine transformation matrix M , and the grid sampling resamples the base kernel B 0 according to the sampling grid coordinate. A new dynamic convolution kernel K that has undergone geometric transformation is generated and stored in the memory.

[0064] Step 6: Grouped convolution and output the convolved image feature map. Specifically, the processing object of this step is the dynamic convolution kernel K generated in step 5 and the original image feature map in step 1. The dynamic convolution kernel K is applied to the corresponding group of the image feature map to perform a grouped convolution operation, and the dynamic convolution kernel K is applied to the corresponding group of the image feature map to obtain multiple groups of convolution results. The multiple groups of convolution results are averaged to obtain the final image feature map after geometric adaptive convolution processing. The generated image feature map has stronger robustness to geometric deformation in the image. The data structure of the finally generated image feature map can also be a four-dimensional tensor (B, C, H, W), where B represents the batch size, C is the number of channels, H is the height of the image feature map, and W is the width of the image feature map.

[0065] The technical scheme provided by the present application can accurately extract the features of the deformed object and ultimately improve the performance of the visual task. Moreover, the technical scheme provided by the present application can effectively solve the problem of lack of geometric constraints in the deformable convolution of the prior art, and can at least bring the following technical effects:

[0066] 1. The geometric consistency and topological stability of the features are ensured. The core technical feature of the present application is to ensure that the generated deformation field is continuous, reversible and smooth through Lie group parameterization and exponential mapping. This mathematical guarantee is the direct reason for its beneficial effects, which fundamentally avoids the tearing of sampling points or the folding of features caused by the unconstrained offset of traditional DCN, thereby ensuring that the local topological structure of the feature map is not destroyed.

[0067] 2. The performance accuracy of the visual task is significantly improved. Since the extracted features are more accurate and robust in geometry, the performance of the downstream task is significantly improved. This performance improvement is a natural result of geometric consistency, because more accurate features can enable the neural network to make more accurate judgments. Experiments have shown that when processing the DOTA v1.0 dataset with a large number of rotation and scale transformations, the technical scheme of the present application can improve the mAP50 index by 12.9 percentage points, fully demonstrating its great advantage in accurate target positioning. Among them, the mAP50 index is one of the commonly used evaluation indexes in the target detection task, mainly measuring the performance of the model when the intersection over union (IoU) threshold is 0.5.

[0068] 3. The stability and generalization ability of the model are enhanced: the deformation provided by the application is well-defined in mathematics, avoiding the gradient anomaly problem in the training process caused by unstable sampling. This stability is derived from its inherent mathematical principles, making the model not only perform well in specific scenarios (such as aerial photography), but also generalize well in general scenarios.

[0069] After the overall technical solution of the embodiments of the application is described in detail, the dynamic convolution method based on Lie group parameterization provided by the embodiments of the application will be described in detail.

[0070] As shown in Figure 2 The dynamic convolution method based on Lie group parameterization provided by the embodiments of the application can include the following steps:

[0071] S210, obtaining an image feature map to be processed.

[0072] The image feature map is extracted from an image including a deformed object.

[0073] Specifically, in actual application scenarios, objects often undergo complex geometric transformations such as rotation, scaling, and non-rigid deformation due to changes in viewing angle, motion, etc. The purpose of the embodiments of the application is to provide a new convolution method that can dynamically adapt to object deformation and fundamentally ensure the continuity and topological stability of geometric transformation. Therefore, the image feature map to be processed is extracted from an image including a deformed object. The data structure of the image feature map to be processed is a four-dimensional tensor with dimensions BxCxHxW. Wherein, B represents batch size, C is the number of channels, H is the height of the image feature map, and W is the width of the image feature map.

[0074] S220, predicting a deformation parameter tensor based on the image feature map, and reshaping the deformation parameter tensor into multiple groups of rotation parameters and translation parameters.

[0075] The translation parameters include horizontal translation parameters and vertical translation parameters.

[0076] As an implementation manner of the embodiments of the application, the data structure of the image feature map is a four-dimensional tensor, and the four-dimensional tensor includes batch size B, channel number C, height H of the image feature map, and width W of the image feature map.

[0077] At this time, predicting a deformation parameter tensor based on the image feature map, and reshaping the deformation parameter tensor into multiple groups of rotation parameters and translation parameters can include the following steps a1 and a2:

[0078] Step a1, calling a parameter prediction network to process the image feature map to generate a deformation parameter tensor with a two-dimensional tensor data structure.

[0079] Wherein, the parameter prediction network comprises an adaptive pooling layer, a first convolutional layer, an activation function and a second convolutional layer, and the two-dimensional tensor comprises a batch size B and a parameter dimension 3G.

[0080] Step a2, based on the batch size B and the parameter dimension 3G, reshaping the deformation parameter tensor into G groups of rotation parameters and translation parameters . ).

[0081] Wherein, G is the number of transformation groups, is a translation parameter in the horizontal direction, is a translation parameter in the vertical direction.

[0082] Specifically, a lightweight parameter prediction network net(·) is used to process the image feature map to be processed, and a group of deformation parameter tensors θ with a dimension of Bx(3G) is predicted. The parameter prediction network can be composed of an adaptive pooling layer, an activation layer and a plurality of convolutional layers. Then the deformation parameter tensor θ is reshaped into G groups of rotation parameters and translation parameters and .

[0083] S230, constructing a Lie algebra matrix corresponding to each group of rotation parameters and translation parameters, and converting the Lie algebra matrix into an affine transformation matrix in the Lie group space through matrix exponential mapping operation.

[0084] As an implementation manner of the embodiment of the present application, constructing a Lie algebra matrix corresponding to each group of rotation parameters and translation parameters, and converting the Lie algebra matrix into an affine transformation matrix in the Lie group space through matrix exponential mapping operation can include the following steps b1 and b2:

[0085] Step b1, for each sample b in the batch g , the rotation parameters and the translation parameters are denoted as , , , and the obtained Lie algebra matrix is constructed as:

[0086] ;

[0087] Step b2, converting the Lie algebra matrix into a 3x3 affine transformation matrix in the Lie group space through matrix exponential mapping operation:

[0088] .

[0089] This step ensures that the mapping from the continuous parameter space to the affine transformation matrix space is smooth and continuous, thus guaranteeing the differential homeomorphism property of the final deformation field.

[0090] As shown in Figure 3 is a comparison diagram of the traditional DCN and the deformation field based on Lie group parameterization of the present application.

[0091] Figure 3 The horizontal coordinate of the figure represents the position value in the deformation field, ranging from 0 to 10, to show the change of the deformation at different horizontal positions; Figure 3 The vertical coordinate of the figure represents the position value in the deformation field, to show the change of the deformation at different vertical positions, also ranging from -2 to 10. Figure 3 Through the two coordinate axes, the differences and characteristics between the deformation field of the traditional deformable convolution and the deformation field under the SE(2) Lie group constraint are clearly shown, from Figure 3 It can be seen that the deformation field generated based on Lie group parameterization in the present application is smoother and more continuous.

[0092] S240, geometric transformation and resampling of the pre-stored base convolution kernel based on the affine transformation matrix, to obtain a plurality of geometric transformation dynamic deformable convolution kernels.

[0093] As an implementation manner of the embodiment of the present application, S240, geometric transformation and resampling of the pre-stored base convolution kernel based on the affine transformation matrix, to obtain a plurality of geometric transformation dynamic deformable convolution kernels, can include the following steps, respectively, step c1 and step c2:

[0094] Step c1, obtaining one or more pre-stored base convolution kernels.

[0095] Step c2, using a standard affine network to generate a deformed sampling network coordinate based on the affine transformation matrix, and using the sampling network coordinate to resample the base convolution kernel through a grid sampling operation, to obtain a plurality of dynamic deformable convolution kernels of different scales.

[0096] Specifically, the system pre-stores one or more base convolution kernels, which can be a Dirac delta function (unit impulse response). The affine transformation matrix generated by step S230 is used to generate a deformed sampling network coordinate, and the base convolution kernel is resampled through an affine grid generation (affine_grid) and grid sampling (grid_sample) operation, to obtain the final dynamic deformable convolution kernel. This process can generate a plurality of dynamic deformable convolution kernels of different scales in parallel.

[0097] ​As shown in FIG. 4(a) and FIG. 4(b), the visualization effect diagram of the original basis convolution kernel affected by different deformation parameters in the SE(2) Lie group. The abscissa and ordinate of FIG. 4(a) and FIG. 4(b) represent the discrete space mark of the function range of the basis convolution kernel, without specific units. The whole figure adopts a uniform color scale (value range [-0.6, 0.6]), and the system explains the essential characteristics of the rotation changing the direction of the basis convolution kernel and the translation regulating the position of the basis convolution kernel in the SE(2) Lie group through the change of the morphology and position of the "double spot" structure, which provides a visualization basis for understanding the effect of rigid body transformation on the basis convolution kernel function.

[0098] Specifically, the subgraph in the upper left corner of FIG. 4(a) takes the original basis convolution kernel as the reference, the subgraph in the upper right corner and the subgraph in the lower left corner successively present the results of 30° and 60° rotation transformation, and it can be seen that the "double spot" structure of the basis convolution kernel systematically deflects with the increase of the angle, which directly verifies the continuous modulation effect of the rotation transformation on the direction characteristics of the basis convolution kernel. The subgraph in the upper left corner of FIG. 4(b) first shows the vertical reconstruction of the basis convolution kernel structure under 90° extreme rotation, and then presents the spatial offset of the basis convolution kernel along a single axis through the comparison of horizontal translation =0.5, =0.0 and vertical translation =0.0, =0.5, which clearly reflects the directional regulation of the translation operation on the position of the basis convolution kernel. The subgraph in the lower right corner finally presents the superimposed offset of the basis convolution kernel along the diagonal line through the composite translation =0.5, =0.5, which verifies the independent controllability and combined action law of the translation parameters.

[0099] S250, performing dynamic convolution operation on the image feature map through the dynamic deformable convolution kernel to generate the image feature map processed by the geometric adaptive convolution.

[0100] The sampling region of the dynamic deformable convolution kernel is adaptive, which can adjust the sampling position and size according to the content of the input image feature map, better capture the morphology of the deformed object, and finally generate the image feature map processed by the geometric adaptive convolution by combining the outputs of different sampling regions. The data structure of the finally generated image feature map can also be a four-dimensional tensor (B, C, H, W), B represents the batch size, C is the number of channels, H is the height of the image feature map, and W is the width of the image feature map.

[0101] As an implementation manner of the embodiment of the present application, S250, performing dynamic convolution operation on the image feature map through the dynamic deformable convolution kernel to generate the image feature map processed by the geometric adaptive convolution, can include the following steps, steps d1 and d2 respectively:

[0102] Step d1, applying the dynamically deformable convolution kernel to the corresponding groups of the image feature map in the manner of grouped convolution to obtain a plurality of groups of convolution results.

[0103] Step d2, performing an average operation on the plurality of groups of convolution results to obtain an output image feature map fused with geometric deformation information.

[0104] Specifically, the dynamically deformable convolution kernel performs convolution operation on the image feature map in the manner of grouped convolution, finally obtaining a plurality of groups of convolution results, and the output image feature map fused with geometric deformation information is calculated by averaging the plurality of groups of convolution results. Since the output image feature map accurately fuses the geometric deformation information, the output image feature map can accurately express the features of the deformed object, which helps to improve the performance of the visual task.

[0105] As an implementation manner of the embodiment of the present application, the method can further include the following steps:

[0106] The image feature map processed by the geometric adaptive convolution is applied to the field of computer vision, including target detection, target segmentation and pose estimation.

[0107] Since the image feature map processed by the geometric adaptive convolution can accurately express the features of the deformed object, it helps to improve the performance of the visual task, and the technical scheme of the embodiment of the present application can be widely applied to the field of computer vision such as target detection, target segmentation and pose estimation.

[0108] The technical scheme provided by the embodiment of the present application obtains an image feature map to be processed, which is extracted from an image including a deformed object; a deformation parameter tensor is predicted based on the image feature map, and the deformation parameter tensor is reshaped into a plurality of groups of rotation parameters and translation parameters, the translation parameters including horizontal translation parameters and vertical translation parameters; a Lie algebra matrix corresponding to each group of rotation parameters and translation parameters is constructed, and the Lie algebra matrix is converted into an affine transformation matrix in Lie group space through matrix exponential mapping operation; this process utilizes the mathematical properties of Lie group to generate an affine transformation matrix with differential homeomorphism characteristics, ensuring the stability of the topological structure in the convolution sampling process, and ensuring that the mapping from the deformation parameter tensor to the affine transformation matrix is continuous and smooth, thereby ensuring the differential homeomorphism characteristics of the transformation.

[0109] Next, the pre-stored base convolution kernel is geometrically transformed and resampled based on the affine transformation matrix to obtain a plurality of geometrically transformed dynamic deformable convolution kernels; and the image feature map is dynamically convolved by the dynamic deformable convolution kernel to generate an image feature map processed by geometric adaptive convolution. Since the sampling region of the dynamic deformable convolution kernel is adaptive, the sampling position and size can be adjusted according to the content of the input image feature map, so that the shape of the deformed object can be better captured. The output of the deformable convolution is obtained by combining the outputs of different sampling regions, and finally the image feature map processed by geometric adaptive convolution is generated. Since the finally output image feature map accurately fuses the geometric deformation information, the finally output image feature map can accurately express the features of the deformed object, which helps to improve the performance of the visual task.

[0110] To verify the beneficial effects of the technical solutions of the present application, experiments are also conducted by the embodiments of the present application. YOLOv8n is used as a baseline model, and tests are conducted on the DOTA v1.0 remote sensing image dataset. The DOTA v1.0 dataset contains a large number of targets with arbitrary rotation and scale changes, which is an ideal scene for verifying the deformation modeling capability. Among them, YOLOv8n is an open source model, and those skilled in the art should understand YOLOv8n, and therefore YOLOv8n will not be described here.

[0111] The ablation experiment results of the embodiments of the present application are as follows:

[0112] 1. Baseline model (without using the technical solutions of the present application): mAP50 index is 27.9%, and mAP50-95 index is 15.8%. Among them, mAP50 is a commonly used performance evaluation index in the target detection task, which represents the average precision mean when the intersection over union (IoU) threshold is 0.5; similarly, mAP95 represents the average precision mean when the intersection over union (IoU) threshold is 0.95.

[0113] 2. Baseline model + technical solutions of the present application: mAP50 index is improved to 40.8% (absolute improvement of 12.9 percentage points), and mAP50-95 index is improved to 24.8% (absolute improvement of 9.0 percentage points).

[0114] The above experimental results can show that the introduction of the dynamic convolution module based on Lie group parameterization proposed by the present application can greatly and substantially improve the performance of the model in the high-difficulty target detection task, which fully proves that the present application solves the defects in the background technology and achieves significant technical progress. The visualization results (such as Figure 3 ) also directly show that the deformation field generated by the present application (right image) is more smooth and orderly than the deformation field of the traditional DCN (left image), which effectively avoids the destruction of the topological structure.

[0115] In a second aspect, an apparatus 50 for dynamic convolution based on Lie group parameterization is provided, as shown in Figure 5 The apparatus comprises:

[0116] An image feature map obtaining module 510 is configured to obtain an image feature map to be processed, the image feature map being extracted from an image including a morphing object.

[0117] A morphing parameter generating module 520 is configured to predict a morphing parameter tensor based on the image feature map and reshape the morphing parameter tensor into a plurality of groups of rotation parameters and translation parameters, the translation parameters including a horizontal translation parameter and a vertical translation parameter.

[0118] A Lie algebra matrix and affine transformation matrix generating module 530 is configured to construct a Lie algebra matrix corresponding to each group of rotation parameters and translation parameters and convert the Lie algebra matrix into an affine transformation matrix in a Lie group space through a matrix exponential mapping operation.

[0119] A dynamically morphable convolution kernel generating module 540 is configured to perform geometric transformation and resampling on a pre-stored base convolution kernel based on the affine transformation matrix to obtain a plurality of dynamically morphable convolution kernels that have been geometrically transformed.

[0120] An image feature map convolution and output module 550 is configured to perform dynamic convolution operation on the image feature map through the dynamically morphable convolution kernels to generate an image feature map that has been processed through geometric self-adaptive convolution.

[0121] Optionally, the data structure of the image feature map is a four-dimensional tensor, and the four-dimensional tensor includes batch size B, channel number C, image feature map height H and image feature map width W.

[0122] The morphing parameter generating module is specifically configured to:

[0123] invoke a parameter prediction network to process the image feature map to generate a morphing parameter tensor with a two-dimensional tensor data structure, the parameter prediction network including an adaptive pooling layer, a first convolution layer, an activation function and a second convolution layer, and the two-dimensional tensor including batch size B and parameter dimension 3G.

[0124] reshape the morphing parameter tensor into G groups of rotation parameters and translation parameters (r x,y, t x,y) based on the batch size B and the parameter dimension 3G, where G is the number of transformation groups, r x is the horizontal translation parameter, t x is the vertical translation parameter.

[0125] ​Optionally, the Lie algebra matrix and affine transformation matrix generating module is specifically configured to:

[0126] For each sample in the batch b and each transformation group g , let the rotation parameters and translation parameters be (R, t) , , , and the resulting Lie algebra matrix is:

[0127] ;

[0128] The Lie algebra matrix is converted into a 3x3 affine transformation matrix in the Lie group space by using a matrix exponential mapping operation :

[0129] .

[0130] Optionally, the dynamically deformable convolution kernel generating module is specifically configured to:

[0131] obtain one or more base convolution kernels stored in advance;

[0132] The affine transformation matrix is used to generate a deformed sampling network coordinate by using a standard affine network, and the base convolution kernel is resampled by using the sampling network coordinate through a grid sampling operation, so that a plurality of dynamically deformable convolution kernels of different scales are obtained.

[0133] Optionally, the image feature map convolution and output module is specifically configured to:

[0134] The dynamically deformable convolution kernel is applied to the corresponding group of the image feature map in a grouped convolution manner, so that a plurality of convolution results are obtained;

[0135] The plurality of convolution results are averaged to obtain an output image feature map fused with geometric deformation information.

[0136] Optionally, the device further comprises:

[0137] An image feature map application module is configured to apply the image feature map processed by the geometric adaptive convolution to the field of computer vision, and the field of computer vision includes target detection, target segmentation, and pose estimation.

[0138] In a third aspect, an embodiment of the present application provides an electronic device 600, as shown in the figure, comprising: Figure 6

[0139] at least one processor 601;

[0140] ​​a memory 602 for storing instructions executable by the at least one processor;

[0141] The at least one processor is configured to execute the instructions to implement the method for dynamic convolution based on Lie group parameterization provided by the embodiments of the present application.

[0142] In a fourth aspect, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method for dynamic convolution based on Lie group parameterization provided by the embodiments of the present application.

[0143] In a fifth aspect, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the method for dynamic convolution based on Lie group parameterization provided by the embodiments of the present application is implemented.

[0144] The technical scheme provided by the embodiments of the present application obtains an image feature map to be processed, which is extracted from an image including a deformed object; a deformed parameter tensor is predicted based on the image feature map, and the deformed parameter tensor is reshaped into a plurality of groups of rotation parameters and translation parameters, the translation parameters including a horizontal translation parameter and a vertical translation parameter; a Lie algebra matrix corresponding to each group of rotation parameters and translation parameters is constructed, and the Lie algebra matrix is converted into an affine transformation matrix in a Lie group space through a matrix exponential mapping operation; this process utilizes the mathematical properties of the Lie group to generate an affine transformation matrix with a differential homeomorphism property, ensuring the stability of the topological structure in the convolution sampling process and ensuring that the mapping from the deformed parameter tensor to the affine transformation matrix is continuous and smooth, thereby guaranteeing the differential homeomorphism property of the transformation.

[0145] Next, the pre-stored basis convolution kernel is geometrically transformed and resampled based on the affine transformation matrix to obtain a plurality of geometrically transformed dynamic deformable convolution kernels; the image feature map is dynamically convolved by the dynamic deformable convolution kernel to generate an image feature map processed by geometric adaptive convolution. Since the sampling region of the dynamic deformable convolution kernel is adaptive, the sampling position and size can be adjusted according to the content of the input image feature map, better capturing the shape of the deformed object, and the output of the deformable convolution is obtained by combining the outputs of different sampling regions, and finally an image feature map processed by geometric adaptive convolution is generated. Since the finally output image feature map accurately integrates the geometric deformation information, the finally output image feature map can accurately express the features of the deformed object, which helps to improve the performance of the visual task.

[0146] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and that changes, modifications, substitutions and variations can be made by those skilled in the art without departing from the spirit and scope of the present application.

Claims

1. A dynamic convolution method based on Lie group parameterization, characterized in that, The method includes: Obtain an image feature map to be processed, the image feature map being extracted from an image including a deformed object, wherein the deformed object has non-rigid deformation; Based on the image feature map, a deformation parameter tensor is predicted, and the deformation parameter tensor is reshaped into multiple sets of rotation parameters and translation parameters, wherein the translation parameters include translation parameters in the horizontal direction and translation parameters in the vertical direction. Construct the Lie algebra matrix corresponding to each set of rotation and translation parameters, and transform the Lie algebra matrix into an affine transformation matrix with differential homeomorphism in the Lie group space through matrix exponential mapping operation to ensure the stability of the topological structure during the convolution sampling process. Based on the affine transformation matrix, the pre-stored basis convolution kernels are geometrically transformed and resampled to obtain multiple geometrically transformed dynamically deformable convolution kernels; The image feature map is dynamically convolved by the dynamically deformable convolution kernel to generate an image feature map that has undergone geometrically adaptive convolution. The sampling region of the dynamically deformable convolution kernel is adaptive and can adjust the sampling position and size according to the content of the input image feature map in order to better capture the shape of deformable objects. The data structure of the image feature map is a four-dimensional tensor, which includes batch size B, number of channels C, height H of the image feature map, and width W of the image feature map. The deformation parameter tensor is predicted based on the image feature map, and the deformation parameter tensor is reshaped into multiple sets of rotation and translation parameters, including: The parameter prediction network is invoked to process the image feature map and generate a deformation parameter tensor with a two-dimensional tensor data structure. The parameter prediction network consists of an adaptive pooling layer, a first convolutional layer, an activation function, and a second convolutional layer. The two-dimensional tensor includes a batch size B and a parameter dimension of 3G. Based on the batch size B and the parameter dimension 3G, the deformation parameter tensor is reshaped into G sets of rotation parameters. Translation parameters ( , ), where G is the number of transform groups, The translation parameter is in the horizontal direction. This refers to the translation parameter in the vertical direction; The process of constructing the Lie algebra matrix corresponding to each set of rotation and translation parameters, and transforming the Lie algebra matrix into an affine transformation matrix in the Lie group space through matrix exponential mapping operations, includes: For each sample in the batch b and each transformation group g Let the rotation and translation parameters be denoted as ( , , The resulting Lie algebra matrix is: ； The Lie algebra matrix is ​​transformed using matrix exponential mapping operations. Transform into a 3×3 affine transformation matrix in Lie group space : 。 2. The method according to claim 1, characterized in that, The process involves performing geometric transformations and resampling on pre-stored basis convolutional kernels based on the affine transformation matrix to obtain multiple dynamically deformable convolutional kernels after geometric transformation, including: Obtain one or more pre-stored basis convolutional kernels; The affine transformation matrix is ​​used to generate deformed sampling network coordinates using a standard affine network. Then, the sampling network coordinates are used to resample the basis convolution kernel through a grid sampling operation to obtain multiple dynamically deformable convolution kernels of different scales.

3. The method according to claim 1, characterized in that, The step of performing a dynamic convolution operation on the image feature map using the dynamically deformable convolution kernel to generate an image feature map processed by geometric adaptive convolution includes: The dynamically deformable convolution kernel is applied to the corresponding groups of the image feature maps in a grouped convolution manner to obtain multiple groups of convolution results; The multiple sets of convolution results are averaged to calculate the output image feature map that incorporates geometric deformation information.

4. The method according to claim 1, characterized in that, The method further includes: The image feature map processed by geometric adaptive convolution is applied to the field of computer vision, which includes object detection, object segmentation, and pose estimation.

5. A dynamic convolution device based on Lie group parameterization, characterized in that, The device includes: The image feature map acquisition module is used to acquire the image feature map to be processed, which is extracted from an image including a deformed object, wherein the deformed object has non-rigid deformation; The deformation parameter generation module is used to predict the deformation parameter tensor based on the image feature map, and reshape the deformation parameter tensor into multiple sets of rotation parameters and translation parameters, wherein the translation parameters include translation parameters in the horizontal direction and translation parameters in the vertical direction; The Lie algebra matrix and affine transformation matrix generation module is used to construct the Lie algebra matrix corresponding to each set of rotation parameters and translation parameters, and to transform the Lie algebra matrix into an affine transformation matrix with differential homeomorphism in the Lie group space through matrix exponential mapping operation, so as to ensure the stability of the topological structure during the convolution sampling process. The dynamic deformable convolution kernel generation module is used to perform geometric transformation and resampling on the pre-stored basis convolution kernels based on the affine transformation matrix to obtain multiple geometrically transformed dynamic deformable convolution kernels. The image feature map convolution and output module is used to perform dynamic convolution operation on the image feature map through the dynamic deformable convolution kernel to generate an image feature map processed by geometric adaptive convolution. The sampling area of ​​the dynamic deformable convolution kernel is adaptive and can adjust the sampling position and size according to the content of the input image feature map in order to better capture the shape of deformable objects. The data structure of the image feature map is a four-dimensional tensor, which includes batch size B, number of channels C, height H of the image feature map, and width W of the image feature map. The deformation parameter generation module is specifically used for: The parameter prediction network is invoked to process the image feature map and generate a deformation parameter tensor with a two-dimensional tensor data structure. The parameter prediction network consists of an adaptive pooling layer, a first convolutional layer, an activation function, and a second convolutional layer. The two-dimensional tensor includes a batch size B and a parameter dimension of 3G. Based on the batch size B and the parameter dimension 3G, the deformation parameter tensor is reshaped into G sets of rotation parameters. Translation parameters ( , ), where G is the number of transform groups, The translation parameter is in the horizontal direction. This refers to the translation parameter in the vertical direction; The Lie algebra matrix and affine transformation matrix generation module is specifically used for: For each sample in the batch b and each transformation group g Let the rotation and translation parameters be denoted as ( , , The resulting Lie algebra matrix is: ； The Lie algebra matrix is ​​transformed using matrix exponential mapping operations. Transform into a 3×3 affine transformation matrix in Lie group space : 。 6. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1-4.

8. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-4.

Images