Image processing method and system
The deformation field offset parameters are generated through thin plate spline interpolation and adaptive area cropping algorithm, and combined with multimodal data splicing and frequency domain filtering technology, the recognition problem of convolutional neural networks in non-rigid deformation scenarios is solved, achieving high-precision and efficient image recognition.
Patent Information
- Application Number
- CN202510647006.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-20
- Publication Date
- 2025-08-15
AI Technical Summary
Existing convolutional neural networks are difficult to effectively deal with non-rigid deformation scenarios. The traditional deformation correction method lacks the ability to compensate complex deformations and does not fully utilize the complementary features of multimodal data, resulting in a decrease in image recognition accuracy.
Thin plate spline interpolation is used to generate deformation field offset parameters, combined with adaptive region cropping algorithm and four-dimensional tensor splicing technology, multi-scale features are extracted through a hollow convolution module with adjustable expansion rate, and high-pass filtering is implemented in the frequency domain to enhance edge features, and low-contribution feature channels are eliminated using gradient significance weights for identification.
It improves the registration accuracy and accuracy of image recognition, reduces the error detection rate and missed detection rate, and improves the micro crack recognition ability in industrial inspection.
Smart Images
Figure CN120495609A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image recognition, and in particular relates to an image processing method and system. Background Art
[0002] With the advancement of hardware computing power and the maturity of big data technology, image recognition and processing technology has become a research hotspot in the field of artificial intelligence. Traditional image recognition methods rely on hand-crafted feature extraction operators combined with classifiers such as support vector machines (SVMs) or random forests to achieve object detection and classification. These methods offer the advantages of strong interpretability and low computational overhead in simple scenarios. However, recognition accuracy significantly decreases in complex environments (such as those subject to changing lighting, object occlusion, and background interference), and they struggle to adapt to the generalization requirements of dynamic scenes.
[0003] Therefore, convolutional neural networks are increasingly used in pattern recognition, image processing and other fields due to their simple structure, few training parameters and strong adaptability.
[0004] Existing convolutional neural network (CNN) classification models generally use fixed-pattern data augmentation (such as mirror flipping and rotation), which makes it difficult to effectively handle non-rigid deformation scenarios (such as organ displacement in medical images). Traditional deformation correction methods often rely on affine transformations, which are insufficient for compensating complex deformations and lead to high feature extraction errors. Furthermore, existing preprocessing pipelines typically process RGB images independently, failing to fully utilize the complementary features of multimodal data (such as depth maps and temporal information). Summary of the Invention
[0005] The purpose of the present invention is to provide an image processing method and system that can accurately correct image deformation through elastic deformation compensation, accurately identify image displacement, and combine RGB images with depth maps to accurately detect cracks in objects in industrial applications, thereby enhancing recognition capabilities and improving recognition efficiency.
[0006] The technical solutions adopted by the present invention are as follows: An image processing method comprises the following steps: Correcting the non-rigid deformation of the image by thin plate spline interpolation based on the received image to generate deformation field offset parameters; Adopting an adaptive region cropping algorithm based on the above deformation field offset parameters, the integrity of the core area of the image is preserved, and the aspect ratio of the cropped area is dynamically adjusted to match the target input size; The RGB image and depth map of the cropped image are concatenated into a four-dimensional tensor, the temporal difference feature is added to the channel dimension, and high-pass filtering is performed in the frequency domain and then spatial reconstruction is performed to enhance the edge features; A dilated convolution module with adjustable dilation rate is used to extract multi-scale features of the processed image. A feature screening module is inserted before the fully connected layer to eliminate low-contribution feature channels based on the gradient significance weight to identify the image.
[0007] As a preferred solution, the specific steps of generating the deformation field offset parameters include: S1.1. Automatically obtain control points through manual annotation or SIFT / SURF feature matching algorithm, and construct deformation function based on the control points; S1.2. According to the constraints and based on the control point components, the linear equations are solved simultaneously. Regularization terms need to be added during the solution process to prevent matrix singularity. S1.3. Calculate the offset for each pixel in the image using S1.1 and S1.2, and solve it independently in two coordinate directions to form a visual deformation field. S1.4. Map the corrected coordinates back to the original image based on the deformation field formed in S1.3, wherein the interpolation algorithm is selected from bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation.
[0008] As a preferred solution, the adaptive region clipping algorithm is specifically as follows: S2.1, pre-process the image processed by S1 by geometric scaling, which sets the input image size to , the target input size is , the mathematical expression of the scaling algorithm is: ; S2.2. Locate the salient regions based on the scaled pre-processed image. Specifically, the image saliency map needs to be calculated using a pre-trained lightweight convolutional network. , where the calculation formula for the significance value is: ; Among them, the significant value calculation formula is the number of feature map channels, is the feature map channel In position The activation value of S2.3. In the scaled image, the maximum point of the saliency map is As the center, a cropping window of the target size is generated. The mathematical expression of the cropping algorithm is as follows: ; If the window exceeds the image boundary, the center position needs to be adjusted to ensure integrity.
[0009] As a preferred solution, the four-dimensional tensor splicing in S3 specifically includes: Perform bilinear interpolation and scaling on the cropped RGB image and depth map to ensure the same resolution. If the depth map is stored in 16-bit format, it needs to be normalized to the range of 0-255 and match the RGB pixel format. Use the affine transformation matrix to correct the coordinate offset between the depth map and RGB, and control the error within ±2 pixels. For the video stream data, RGB frames and depth frames are aligned by timestamp to eliminate timing misalignment. The RGB three channels and the depth channel are spliced into a four-channel matrix, and N frames of four-channel data are continuously collected and stacked along the time axis to form a four-dimensional tensor.
[0010] As a preferred solution, the steps for generating the temporal difference feature in S3 are as follows: The sliding window method is used to perform differential operation on continuous frame images, and the differential image is subjected to Gaussian smoothing. The temporal difference features are fused with the RGB image and depth map at the channel level to construct a four-dimensional input tensor, and each channel data is independently normalized. In the process of generating the time series difference feature, the sliding window needs to be continuously updated, thereby continuously updating the time series dimension of the four-dimensional tensor.
[0011] As a preferred solution, the specific steps of enhancing the edge features in S3 are as follows: S3.1. Perform a two-dimensional Fourier transform on the input image to convert the spatial domain image into a frequency domain spectrum, move the zero frequency point of the spectrum to the center of the spectrum, and avoid high-frequency components being covered by low-frequency components; S3.2. Use Butterworth high-pass filter or Gaussian high-pass filter to adjust and control the image edge enhancement intensity; S3.3, multiplying the filter by the spectrum point by point so that high-frequency components are retained and low-frequency components are suppressed; S3.4. Move the zero-frequency point of the filtered spectrum back to the upper left corner, perform inverse transform, and take the real part of the result as the reconstructed image. The imaginary part can be ignored due to calculation error. S3.5. Enhance high-frequency details through histogram equalization or contrast stretching, superimpose the original image and the filtered result, and balance edge sharpening and noise suppression.
[0012] As a preferred solution, the expansion rate in S4 is dynamically configured according to the input resolution, as follows: The current layer input resolution is obtained through the feature map size parsing unit, and the ratio coefficient of the long side to the short side is calculated to set the resolution classification threshold; Establish a resolution-expansion rate mapping rule base, and implement dynamic configuration of resolution based on the mapping rule base; Before each convolution operation, the corresponding expansion rate parameters are called from the preset mapping rule library according to the current resolution level, and a smooth parameter transition is achieved through a differentiable gating mechanism to avoid feature discontinuity caused by resolution mutation.
[0013] As a preferred solution, the screening steps of the feature screening module in S4 are as follows: The gradient response value of the feature channel to the loss function is obtained through back propagation, and the weight is updated using a sliding average mechanism to avoid fluctuations in single batch data. All feature channels are sorted in descending order according to the gradient response value to generate an importance sequence; Channels with high local contribution are retained based on descending order, and the global contribution mean is calculated, and channels with gradient response values less than the global contribution mean are eliminated.
[0014] An image processing system, applying the above image processing method, comprising: Multimodal and multispectral acquisition module, used to acquire images of items to be processed, integrating synchronized data capture of visible light / infrared / ToF cameras; Elastic deformation preprocessing module, which generates deformation field through thin plate spline interpolation, calculates and retains the core area, and strengthens the image edge; The image feature extraction and recognition module is used to extract multi-scale features of the processed image and use the feature screening module to eliminate low-contribution feature channels based on gradient significance weights to identify the image; Dynamic routing convolutional network module, which includes a multi-scale feature extraction layer and a quantization perception unit; The adaptive deployment engine module dynamically switches the system computing power and performs cluster deployment based on the containerized service interface.
[0015] As a preferred solution, the multi-scale feature extraction layer includes a multi-path dilated convolution unit for adjusting the resolution and dilation rate, a soft conditional gate routing mechanism unit for dynamically selecting the feature transmission path, and a cross-scale feature aggregation unit for integrating high- and low-level semantic information; The quantization perception unit includes a quantization error simulation subunit for linearly quantizing weights / activation values, a dynamic range calibration subunit for automatically adjusting the quantization threshold, and a mixed precision routing controller for balancing accuracy and speed.
[0016] The technical effects achieved by the present invention are: This method uses thin-plate spline interpolation to generate deformation field offset parameters, addressing issues such as organ displacement in medical images and deformation of industrial parts. Compared to traditional affine transformations, this method improves registration accuracy and reduces the MAE metric on medical imaging datasets. Furthermore, the method incorporates an adaptive region cropping algorithm that dynamically matches image aspect ratios while preserving the integrity of core objects, significantly improving accuracy on image recognition datasets.
[0017] The present invention performs four-dimensional tensor splicing of RGB images and depth maps / time-series differential features, fuses multi-source information through the channel dimension, and fully utilizes the complementary characteristics of multimodal data. In industrial inspection scenarios, the false detection rate is greatly reduced, and the missed detection rate is also reduced. By introducing frequency domain-spatial domain joint processing, the edge features are first enhanced through Butterworth high-pass filtering, and then the spatial domain image is reconstructed. The edge feature enhancement degree is improved, thereby significantly improving the ability to identify small cracks in PCB defect detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a schematic diagram of the flow structure of the image processing method of the present invention; Figure 2 is a specific flow chart of step 1 in the image processing method of the present invention; Figure 3 This is a specific flow chart of step three in the image processing method of the present invention; Figure 4 It is a structural diagram of the image processing system of the present invention. DETAILED DESCRIPTION
[0019] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is described in detail below with reference to the following examples. It should be understood that the following text is only used to describe one or more specific embodiments of the present invention and does not strictly limit the scope of protection of the present invention.
[0020] like Figure 1-Figure 3 As shown, an image processing method includes the following steps: Step 1: Based on the received image, the non-rigid deformation of the image is corrected by thin plate spline interpolation to generate deformation field offset parameters.
[0021] 1. Control point selection and deformation field modeling 1.1、Input control point pair Define the source image control point set: ; Target image corresponding point set: ; Control points can be obtained manually or automatically through feature matching algorithms such as SIFT / SURF.
[0022] 1.2 Deformation Function Construction The TPS deformation function consists of an affine transformation and a nonlinear term: ; in, is the thin plate spline basis function (radial basis function); are the affine transformation parameters, is the nonlinear deformation weight.
[0023] 2. Parameter solution 2.1. Constructing a System of Linear Equations Through the constraints And bending energy is minimized, and the matrix equation is established: ; for Kernel Matrix; is the control point coordinate matrix, .
[0024] 2.2 Matrix Inversion Solution Solve using LU decomposition or SVD: ; Regularization term needs to be added Prevent matrix singularity (usually Take 1e-6).
[0025] 3. Deformation field generation 3.1 Global deformation calculation For each pixel in the image Calculate the offset: ; ; and Direction-independent solution parameters.
[0026] 3.2 Deformation Field Visualization Generate a 2D displacement field matrix , the deformation trend can be represented by a heat map or a vector map.
[0027] 4. Image resampling 4.1 Coordinate Mapping Map the corrected coordinates back to the original image according to the deformation field: .
[0028] 4.2 Interpolation Algorithm Selection Bilinear interpolation: high computational efficiency, suitable for real-time applications; Bicubic interpolation: retains more high-frequency details and is commonly used in medical imaging; Nearest neighbor interpolation: used for binary image processing; It should be noted that the above three interpolation algorithms can be selected according to needs, and in this embodiment, the bilinear interpolation algorithm is selected.
[0029] In actual application, the application data is as follows: Table 1 Step 2: Adopt an adaptive region cropping algorithm based on the above deformation field offset parameters to preserve the integrity of the core area of the image and dynamically adjust the aspect ratio of the cropped area to match the target input size.
[0030] In practical applications, the adaptive region clipping algorithm is as follows: 1. Preprocess the image by geometric scaling The image processed by S1 is pre-processed by geometric scaling, which sets the input image size to , the target input size is , the mathematical expression of the scaling algorithm is: ; Among them, the image size after scaling is , make sure the shortest side matches the target size.
[0031] 2. Locating salient areas The salient areas are located based on the scaled pre-processed image. Specifically, the image saliency map is calculated through a pre-trained lightweight convolutional network. , where the calculation formula for the significance value is: ; Among them, the significant value calculation formula is the number of feature map channels, is the feature map channel In position The activation value of .
[0032] 3. Generate dynamic cropping window In the scaled image, the maximum point of the saliency map is As the center, a cropping window of the target size is generated. The mathematical expression of the cropping algorithm is as follows: ; If the window exceeds the image boundary, the center position needs to be adjusted to ensure integrity.
[0033] After application, this algorithm is compared with the traditional algorithm, and its advantages are as follows: Table 2 Step 3: Concatenate the RGB image and depth map of the cropped image into a four-dimensional tensor, add temporal difference features to the channel dimension, and perform spatial reconstruction after high-pass filtering in the frequency domain to enhance the edge features.
[0034] The specific steps for four-dimensional tensor splicing of RGB image and depth map are as follows: 1. Data preprocessing stage 1.1. Size alignment correction Perform bilinear interpolation scaling on the cropped RGB image (H×W×3) and depth map (H×W×1) to ensure that the resolution is completely consistent; If the depth map is stored in 16 bits, it needs to be normalized to the range of 0-255 and match the RGB pixel format.
[0035] 1.2. Spatiotemporal alignment processing The affine transformation matrix is used to correct the coordinate offset between the depth map and RGB, and the error is controlled within ±2 pixels; For video stream data, RGB frames and depth frames are aligned by timestamp to eliminate timing misalignment.
[0036] 2. Four-dimensional tensor construction process 2.1 Channel-level fusion The RGB three channels and the depth channel are spliced into a four-channel matrix (H×W×4), where the R / G / B channels retain the original color data and the depth channel stores the normalized distance value (range 0-1).
[0037] 2.2 Time Series Dimension Expansion Continuously collect N frames of four-channel data (for example, N=5) and stack them along the time axis to form a four-dimensional tensor H×W×4×N. The calculation formula is as follows: .
[0038] 2.3 Differential Feature Enhancement Calculate the depth difference between adjacent frames using the following formula: ; in, is a Gaussian filter kernel used to smooth noise. The difference result is then incorporated into the tensor as an additional feature channel.
[0039] The application data of this embodiment in the industrial detection scenario is as follows: Table 3 Furthermore, the steps for generating the above-mentioned temporal difference features are as follows: 1. Generate temporal difference features 1.1. Inter-frame difference calculation Using the sliding window method, differential operations are performed on consecutive frame images: ; in, represents the t-th frame image, is the previous frame image, This is the difference result diagram.
[0040] 1.2. Perform Gaussian filtering Perform Gaussian smoothing on the difference image: ; in, is the Gaussian kernel, Controls the smoothing strength to suppress noise interference.
[0041] 2. Perform multimodal feature fusion 2.1 Multi-source data splicing The temporal difference feature Perform channel-level fusion with RGB images and depth maps; construct a four-dimensional input tensor: height (H) × width (W) × channel (C) × time sequence (T); for example: 224×224×(3+1+1)×5 represents 5 frames of sequence data.
[0042] 2.2. Feature Standardization Perform independent normalization on each channel data: ; in, They are respectively from the statistical values of the ImageNet dataset.
[0043] 3. Continuously update the dynamic feature mechanism 3.1 Sliding Window Update Set the window length to T=5. Each time a new frame is added: remove the oldest frame data, recalculate the differential features of the latest frame, and update the time series dimension of the four-dimensional tensor.
[0044] 3.2. Allocating Attention Weights Automatically learn channel importance through the SE (Squeeze-and-Excitation) module: ; in, is the channel feature global descriptor, is the ReLU activation.
[0045] Furthermore, the specific steps of enhancing the edge features of the image are as follows: 1. Process the frequency domain 1.1 Fast Fourier Transform (FFT) Perform a two-dimensional Fourier transform on the input image to convert the spatial domain image into a frequency domain spectrum: ; Move the zero frequency point (DC component) of the spectrum to the center of the spectrum graph to prevent high-frequency components from being covered by low-frequency components.
[0046] 1.2. Use high-pass filter to control edge enhancement intensity Using Butterworth high-pass filter, its transfer function is: ; in, is the cutoff frequency (usually set to ), is the distance from the frequency domain point to the center, is the filter order.
[0047] Or use a Gaussian high-pass filter, the formula is: ; By adjusting Controls the strength of edge enhancement.
[0048] 1.3. Perform frequency domain filtering operation The filter With spectrum Point-wise multiplication: ; This ensures that high-frequency components (corresponding to image edges / textures) are retained and low-frequency components (corresponding to smooth areas) are suppressed.
[0049] 2. Spatial reconstruction and post-processing 2.1. Perform Inverse Fourier Transform (IFFT) The filtered spectrum Move the zero frequency point back to the upper left corner and perform the inverse transform: ; The real part of the result is taken as the reconstructed image, and the imaginary part can be ignored due to calculation error.
[0050] 2.2. Strengthening edge features Enhance high-frequency details through histogram equalization or contrast stretching. The formula is: ; in, Control contrast gain (usually 1.5-2.0), is the brightness offset.
[0051] Next, overlay the original image and the filtered result (optional): ; in, is the fusion coefficient (usually 0.3-0.7), balancing edge sharpening and noise suppression.
[0052] Step 4: Use a dilation-rate-adjustable dilation convolution module to extract multi-scale features of the processed image, and insert a feature screening module before the fully connected layer to eliminate low-contribution feature channels based on the gradient significance weight to identify the image.
[0053] It should be noted that the expansion rate is dynamically configured according to the input resolution, as follows: Get the current layer input resolution through the feature map size parsing unit , calculate the ratio coefficient of the long side to the short side γ=max(H,W) / min(H,W) γ = max ( H , W ) / min ( H , W ); and set the resolution classification threshold to establish a resolution-expansion rate mapping rule base, in this embodiment, as shown in the following table: Table 4 In this embodiment, a real-time adjustment mechanism is provided in the expansion configuration algorithm. Before each convolution operation, the corresponding expansion rate parameter is called from the preset mapping rule library according to the current resolution level, and a smooth parameter transition is achieved through a differentiable gating mechanism to avoid feature discontinuity caused by resolution mutation.
[0054] The specific steps of identifying images by removing low-contribution feature channels based on gradient significance weights through the feature screening module are as follows: 1. Calculate gradient significance weight The gradient response value of the feature channel to the loss function is obtained through back propagation. The calculation formula is: ; in, Indicates the The first feature channel activation values, is the loss function, is the number of samples.
[0055] Secondly, a sliding average mechanism is used to update the weights to avoid fluctuations in single batch data: ; Usually 0.9 is taken; Afterwards, all feature channels are Arrange the values in descending order to generate a sequence of importance .
[0056] 2. Adopt a dynamic screening mechanism The top-K high-contribution channels are retained by the fixed ratio method (e.g., the top 70% of channels are retained); and the global contribution mean is calculated using the adaptive threshold method. , remove Channel ( Usually 0.3-0.5 is used); thus, the image is identified and screened according to the above algorithm.
[0057] like Figure 1-Figure 4 As shown, an image processing system, applying the image processing method of this embodiment, includes: Multimodal and multispectral acquisition module, used to acquire images of items to be processed, integrating synchronized data capture of visible light / infrared / ToF cameras.
[0058] The elastic deformation preprocessing module corrects the non-rigid deformation of the image through thin plate spline interpolation based on the received image and generates the deformation field offset parameters; An adaptive region cropping algorithm is used based on the above deformation field offset parameters to preserve the integrity of the core area of the image and dynamically adjust the aspect ratio of the cropped area to match the target input size. Then, the RGB image and depth map of the cropped image are concatenated into four-dimensional tensors, the temporal difference feature is added to the channel dimension, and the spatial domain is reconstructed after high-pass filtering in the frequency domain to enhance the edge features.
[0059] The image feature extraction and recognition module uses a dilation-rate-adjustable dilation convolution module to extract multi-scale features of the processed image, and inserts a feature screening module before the fully connected layer to identify the image by eliminating low-contribution feature channels based on the gradient significance weight.
[0060] Dynamic routing convolutional network module, which includes a multi-scale feature extraction layer and a quantization perception unit; The multi-scale feature extraction layer includes a multi-path dilated convolution unit for adjusting the resolution and dilation rate, a soft conditional gate routing mechanism unit for dynamically selecting feature transmission paths, and a cross-scale feature aggregation unit for integrating high- and low-level semantic information. Furthermore, the quantization perception unit includes a quantization error simulation subunit for linearly quantizing weights / activation values, a dynamic range calibration subunit for automatically adjusting the quantization threshold, and a mixed-precision routing controller for balancing accuracy and speed.
[0061] The adaptive deployment engine module dynamically switches the system computing power and performs cluster deployment based on the containerized service interface.
[0062] The present application also provides an image processing terminal device, comprising: memory and at least one processor.
[0063] The memory is used to store computer-readable instructions.
[0064] The processor is configured to execute the computer-readable instructions in the memory to perform various operations of the image processing system in this embodiment.
[0065] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0066] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, a process or function according to an embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrations. Available media can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media, etc.
[0067] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0068] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.
[0069] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0070] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0071] The foregoing is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained herein shall, unless otherwise specified or limited, be implemented in accordance with conventional means in the art.
Claims
1. An image processing method, characterized in that: The following steps are involved: Based on the received image, the non-rigid deformation of the image is corrected by thin plate spline interpolation to generate deformation field offset parameters; Adopting an adaptive region cropping algorithm based on the above deformation field offset parameters, the integrity of the core area of the image is preserved, and the aspect ratio of the cropped area is dynamically matched to the target input size; The RGB image and depth map of the cropped image are concatenated into a four-dimensional tensor, the temporal difference feature is added to the channel dimension, and high-pass filtering is performed in the frequency domain and then spatial reconstruction is performed to enhance the edge features; A dilated convolution module with adjustable dilation rate is used to extract multi-scale features of the processed image. A feature screening module is inserted before the fully connected layer to eliminate low-contribution feature channels based on the gradient significance weight to identify the image.
2. An image processing method according to claim 1, characterized in that: The specific steps of generating the deformation field offset parameters include: S1.
1. Automatically obtain control points through manual annotation or SIFT / SURF feature matching algorithm, and construct deformation function based on the control points; S1.
2. According to the constraints and based on the control point components, the linear equations are solved simultaneously. Regularization terms need to be added during the solution process to prevent matrix singularity. S1.
3. Calculate the offset for each pixel in the image using S1.1 and S1.2, and solve it independently in two coordinate directions to form a visual deformation field. S1.
4. Map the corrected coordinates back to the original image based on the deformation field formed in S1.3, wherein the interpolation algorithm is selected from bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation.
3. An image processing method according to claim 2, characterized in that: The adaptive region clipping algorithm is specifically as follows: S2.1, pre-process the image processed by S1 by geometric scaling, which sets the input image size to , the target input size is , the mathematical expression of the scaling algorithm is: ; S2.
2. Locate the salient regions based on the scaled pre-processed image. Specifically, the image saliency map needs to be calculated using a pre-trained lightweight convolutional network. , where the calculation formula for the significance value is: ; Among them, the significant value calculation formula is the number of feature map channels, is the feature map channel In position The activation value of S2.
3. In the scaled image, the maximum point of the saliency map is As the center, a cropping window of the target size is generated. The mathematical expression of the cropping algorithm is as follows: ； If the window exceeds the image boundary, the center position needs to be adjusted to ensure integrity.
4. The image processing method according to claim 1, wherein: The four-dimensional tensor splicing in S3 specifically includes: Perform bilinear interpolation and scaling on the cropped RGB image and depth map to ensure the same resolution. If the depth map is stored in 16-bit format, it needs to be normalized to the range of 0-255 and match the RGB pixel format. Use the affine transformation matrix to correct the coordinate offset between the depth map and RGB, and control the error within ±2 pixels. For the video stream data, RGB frames and depth frames are aligned by timestamp to eliminate timing misalignment. The RGB three channels and the depth channel are spliced into a four-channel matrix, and N frames of four-channel data are continuously collected and stacked along the time axis to form a four-dimensional tensor.
5. The image processing method according to claim 1, wherein: The steps for generating the temporal difference feature in S3 are as follows: The sliding window method is used to perform differential operation on continuous frame images, and the differential image is subjected to Gaussian smoothing. The temporal difference features are fused with the RGB image and depth map at the channel level to construct a four-dimensional input tensor, and each channel data is independently normalized. In the process of generating the time series difference feature, the sliding window needs to be continuously updated, thereby continuously updating the time series dimension of the four-dimensional tensor.
6. The image processing method according to claim 1, wherein: The specific steps of enhancing the edge features in S3 are as follows: S3.
1. Perform a two-dimensional Fourier transform on the input image to convert the spatial domain image into a frequency domain spectrum, move the zero frequency point of the spectrum to the center of the spectrum, and avoid high-frequency components being covered by low-frequency components; S3.
2. Use Butterworth high-pass filter or Gaussian high-pass filter to adjust and control the image edge enhancement intensity; S3.3, multiplying the filter by the spectrum point by point so that high-frequency components are retained and low-frequency components are suppressed; S3.
4. Move the zero-frequency point of the filtered spectrum back to the upper left corner, perform inverse transform, and take the real part of the result as the reconstructed image. The imaginary part can be ignored due to calculation error. S3.
5. Enhance high-frequency details through histogram equalization or contrast stretching, superimpose the original image and the filtered result, and balance edge sharpening and noise suppression.
7. The image processing method according to claim 1, wherein: The expansion rate in S4 is dynamically configured according to the input resolution, as follows: The current layer input resolution is obtained through the feature map size parsing unit, and the ratio coefficient of the long side to the short side is calculated to set the resolution classification threshold; Establish a resolution-expansion rate mapping rule base, and implement dynamic configuration of resolution based on the mapping rule base; Before each convolution operation, the corresponding expansion rate parameters are called from the preset mapping rule library according to the current resolution level, and a smooth parameter transition is achieved through a differentiable gating mechanism to avoid feature discontinuity caused by resolution mutation.
8. The image processing method according to claim 1, wherein: The screening steps of the feature screening module in S4 are as follows: The gradient response value of the feature channel to the loss function is obtained through back propagation, and the weight is updated using a sliding average mechanism to avoid fluctuations in single batch data. All feature channels are sorted in descending order according to the gradient response value to generate an importance sequence; Channels with high local contribution are retained based on descending order, and the global contribution mean is calculated, and channels with gradient response values less than the global contribution mean are eliminated.
9. An image processing system, applying the image processing method according to any one of claims 1 to 9, characterized in that: include: Multimodal and multispectral acquisition module, used to acquire images of items to be processed, integrating synchronized data capture of visible light / infrared / ToF cameras; Elastic deformation preprocessing module, which generates deformation field through thin plate spline interpolation, calculates and retains the core area, and strengthens the image edge; The image feature extraction and recognition module is used to extract multi-scale features of the processed image and use the feature screening module to eliminate low-contribution feature channels based on gradient significance weights to identify the image; Dynamic routing convolutional network module, which includes a multi-scale feature extraction layer and a quantization perception unit; The adaptive deployment engine module dynamically switches the system computing power and performs cluster deployment based on the containerized service interface.
10. The image processing system according to claim 9, characterized in that: The multi-scale feature extraction layer includes a multi-path dilated convolution unit for adjusting the resolution and dilation rate, a soft conditional gate routing mechanism unit for dynamically selecting the feature transmission path, and a cross-scale feature aggregation unit for integrating high- and low-level semantic information; The quantization perception unit includes a quantization error simulation subunit for linearly quantizing weights / activation values, a dynamic range calibration subunit for automatically adjusting the quantization threshold, and a mixed precision routing controller for balancing accuracy and speed.