Multi-lens image real-time splicing optimization method based on deep learning

Abstract

The invention relates to the technical field of image real-time splicing, in particular to a multi-lens image real-time splicing optimization method based on deep learning, and the method comprises the steps: obtaining image data collected by a plurality of lenses; acquiring initial shooting parameters of the plurality of lenses; based on the image data, using a deep learning model to extract image features; executing image registration operation according to the image features; calculating an image splicing weight based on a registration result; generating a spliced image according to the splicing weight; outputting a spliced image; based on the quality evaluation result of the spliced image, shooting parameters are updated, and by designing a special deep learning model, the accuracy and robustness of feature extraction and matching are greatly improved. The model can adapt to various complex scenes, such as illumination variation and texture repetition, and effectively overcomes the limitation of a traditional method under the conditions. Particularly in a dynamic scene, the method disclosed by the invention shows excellent adaptive capacity, and provides reliable technical support for real-time video stitching.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time image stitching, and more specifically, to a multi-lens image real-time stitching optimization method based on deep learning. Background Art

[0002] With the rapid development of computer vision technology, multi-lens image stitching technology has been widely used in fields such as panoramic photography, video surveillance, and unmanned driving. Traditional image stitching methods mainly rely on feature point matching and image transformation. Although they can achieve good results in simple scenes, they often face many challenges in complex environments.

[0003] At present, the method commonly used in the industry is to extract and match features based on algorithms such as SIFT or SURF, then estimate the transformation matrix through the RANSAC algorithm, and finally perform image fusion. This method has obvious deficiencies in computational efficiency and robustness. Especially in complex situations such as large lighting changes, repeated textures, and dynamic scenes, matching errors are prone to occur, resulting in splicing distortion or splicing failure.

[0004] On the other hand, with the rise of deep learning technology, some researchers have tried to apply convolutional neural networks to image stitching tasks. These methods directly learn the mapping relationship between images in an end-to-end manner, which improves the accuracy of stitching to a certain extent. However, such methods often require a large amount of labeled data for training, and have limited generalization capabilities, making them difficult to adapt to various complex practical scenarios.

[0005] In addition, most existing multi-lens image stitching methods focus on static scenes, and there is little research on real-time stitching in dynamic environments. In practical applications, such as unmanned driving or video surveillance systems, real-time stitching of rapidly changing scenes is often required, which places higher requirements on the real-time and robustness of the algorithm.

[0006] There is also a common problem with existing technologies, which is that it is difficult to effectively deal with the color differences and distortion problems between multiple lenses. Due to the differences in imaging characteristics of different lenses, direct stitching often leads to color inconsistency or geometric distortion, affecting the visual quality of the final stitched image.

[0007] Another urgent problem to be solved is how to achieve high-quality real-time stitching with limited computing resources. Current methods either rely on high-performance GPUs for acceleration or sacrifice image quality in exchange for real-time performance, making it difficult to achieve an ideal balance on ordinary hardware platforms. Summary of the invention

[0008] The present invention aims to solve the above technical problems and proposes a multi-lens image real-time stitching optimization method based on deep learning. The method realizes efficient and high-quality multi-lens image real-time stitching through innovative deep learning model design and hardware acceleration strategy.

[0009] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0010] The multi-lens image real-time stitching optimization method based on deep learning includes:

[0011] The acquisition steps include:

[0012] Acquire image data collected by multiple lenses;

[0013] Acquiring initial shooting parameters of the multiple lenses;

[0014] Processing steps include:

[0015] Based on the image data, extract image features using a deep learning model;

[0016] Performing an image registration operation according to the image features;

[0017] Based on the registration result, calculating the image stitching weight;

[0018] generating a spliced ​​image according to the splicing weight;

[0019] Output steps include:

[0020] Outputting the stitched image;

[0021] The shooting parameters are updated based on the quality assessment result of the stitched image.

[0022] Preferably, the obtaining step specifically includes:

[0023] Arrange and fix multiple lenses around the target area to be photographed;

[0024] Initializing parameters of each of the lenses;

[0025] Each of the lenses is used to shoot the same target from different angles, thereby forming multi-lens images with overlapping areas.

[0026] Preferably, the extracting image features by using a deep learning model specifically includes:

[0027] Calculating a feature map of the image data using a convolutional neural network;

[0028] Detecting key points on the feature map;

[0029] Extract feature descriptors of the key points.

[0030] Preferably, the performing of the image registration operation specifically includes:

[0031] Using the pre-learned similarity comparison function, perform feature value matching on the key points of each pair of images; select multiple groups of the most similar feature value pairs;

[0032] Based on the eigenvalue pairs, transformation parameters for image registration are calculated.

[0033] Preferably, the calculating of image stitching weights specifically includes:

[0034] Use the attention mechanism to analyze the importance of each region in the feature map;

[0035] Based on the importance analysis result, calculating the weight parameter of each region;

[0036] The weight parameters are screened using a non-maximum suppression algorithm.

[0037] Preferably, generating the spliced ​​image specifically includes:

[0038] Based on the stitching weight, weighted averaging is performed on the overlapping areas;

[0039] Use image fusion algorithm to eliminate splicing traces;

[0040] Performs a color consistency optimization operation.

[0041] Preferably, the step of depth estimation is also included:

[0042] Based on the multi-lens images, performing a stereo matching operation;

[0043] Generate a depth map;

[0044] Based on the depth map, a three-dimensional point cloud model is constructed.

[0045] Preferably, the method further includes a multimodal fusion step:

[0046] Acquire image data of different imaging modalities;

[0047] Based on wavelet transform, perform multimodal image fusion operation;

[0048] Dynamically adjust the weights of different modes according to the characteristics of the scene.

[0049] Preferably, the updating of the shooting parameters specifically includes:

[0050] Real-time monitoring of image quality indicators;

[0051] Based on a reinforcement learning algorithm, dynamically adjusting shooting parameters according to the image quality index;

[0052] A parameter adjustment model is established to guide subsequent shooting.

[0053] Preferably, the hardware acceleration step is also included:

[0054] Mapping the convolutional layer of the deep learning model to the FPGA for implementation;

[0055] Use GPU to accelerate matrix operations and deep learning reasoning;

[0056] Implement pipeline parallel processing architecture to maximize hardware utilization:

[0057] The method of the present invention has the following significant technical effects:

[0058] First, the present invention greatly improves the accuracy and robustness of feature extraction and matching by designing a special deep learning model. The model can adapt to various complex scenes, such as lighting changes, texture repetition, etc., and effectively overcomes the limitations of traditional methods in these situations. Especially in dynamic scenes, the method of the present invention shows excellent adaptability and provides reliable technical support for real-time video splicing.

[0059] Secondly, the present invention adopts innovative multimodal fusion technology to effectively solve the color difference and distortion problems between different lenses. Through an intelligent weight distribution strategy, the present method can adaptively adjust the contribution of different modal images to generate a more visually natural and coherent stitching result. This feature greatly improves the quality of the stitched image, making the stitching traces almost imperceptible.

[0060] Furthermore, the method of the present invention has made a breakthrough in real-time performance. Through a carefully designed hardware acceleration strategy, including the coordinated optimization of FPGA and GPU, and an efficient pipeline parallel processing architecture, the method can achieve high frame rate real-time stitching on a common hardware platform. This not only greatly expands the application scope of multi-lens image stitching technology, but also significantly reduces the deployment cost of the system.

[0061] In addition, the present invention has strong scalability and adaptability. By introducing an adaptive parameter adjustment mechanism, the system can dynamically optimize the splicing parameters according to the actual scene and image quality, and continuously improve the splicing effect without manual intervention. This feature enables the present method to adapt to various complex and changeable application environments, greatly improving the practicality of the system.

[0062] Finally, the method of the present invention is not limited to generating two-dimensional stitched images, but can also provide three-dimensional structural information of the scene. Through depth estimation and point cloud construction, this method lays the foundation for subsequent scene understanding and interactive applications, greatly expanding the application prospects of the technology.

[0063] In summary, the deep learning-based multi-lens image real-time stitching optimization method proposed in the present invention has made significant progress in stitching quality, real-time performance, adaptability, and functional diversity, effectively solving many challenges faced by the existing technology and opening up a new direction for the development of multi-lens image stitching technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 The figure is an overall flow chart of the method of the present invention.

[0065] Figure 2 This is a logical block diagram of the deep learning feature extraction of the present invention.

[0066] Figure 3 The logical block diagram of the image registration of the present invention.

[0067] Figure 4 It is a logic block diagram of splicing weight calculation of the present invention.

[0068] Figure 5 The figure is a logic block diagram for generating a spliced ​​image according to the present invention.

[0069] Figure 6 It is a logic block diagram of updating shooting parameters of the present invention. DETAILED DESCRIPTION

[0070] like Figure 1-6 As shown, the present invention provides a multi-lens image real-time stitching optimization method based on deep learning. The method realizes efficient and high-quality multi-lens image real-time stitching through deep learning technology and hardware acceleration, and solves the problems existing in traditional methods when processing complex scenes. The present invention will be described in detail below in conjunction with specific embodiments.

[0071] First, the multi-lens image real-time stitching optimization method based on deep learning of the present invention comprises the following steps:

[0072] The acquisition steps include:

[0073] Acquire image data collected by multiple lenses;

[0074] Acquiring initial shooting parameters of the multiple lenses;

[0075] Processing steps include:

[0076] Based on the image data, extract image features using a deep learning model;

[0077] Performing an image registration operation according to the image features;

[0078] Based on the registration result, calculating the image stitching weight;

[0079] generating a spliced ​​image according to the splicing weight;

[0080] Output steps include:

[0081] Outputting the stitched image;

[0082] The shooting parameters are updated based on the quality assessment result of the stitched image.

[0083] In a preferred embodiment of the present invention, the obtaining step specifically includes:

[0084] Arrange and fix multiple lenses around the target area to be photographed;

[0085] Initializing parameters of each of the lenses;

[0086] Each of the lenses is used to shoot the same target from different angles, thereby forming multi-lens images with overlapping areas.

[0087] Preferably, the method of extracting image features using a deep learning model in the present invention specifically includes:

[0088] Calculating a feature map of the image data using a convolutional neural network;

[0089] Detecting key points on the feature map;

[0090] Extract feature descriptors of the key points.

[0091] The above steps are explained in detail below:

[0092] In the acquisition step, the method first acquires image data collected by multiple lenses. These lenses can be different types of cameras, such as visible light cameras, infrared cameras, etc. By using multiple lenses, a wider field of view and richer scene information can be obtained.

[0093] At the same time, this method also obtains the initial shooting parameters of these lenses. These parameters may include but are not limited to:

[0094] Focal length (usually between 5mm and 50mm);

[0095] Aperture size (common values ​​are f / 1.8 to f / 8);

[0096] Shutter speed (probably between 1 / 1000 second and 1 second);

[0097] ISO sensitivity (common range is 100 to 3200);

[0098] White balance settings (such as automatic, daylight, cloudy, etc.);

[0099] The setting of these initial parameters is crucial to the subsequent image quality and stitching effect.

[0100] In one embodiment of the present invention, a plurality of lenses are arranged and fixed around the target area to be photographed. This arrangement can ensure all-round coverage of the target area and reduce the generation of blind spots. For example, in a typical setting, 4 to 8 lenses may be used, arranged on a circumference at intervals of 60 to 90 degrees.

[0101] Initializing the parameters of each lens is a key step to ensure initial image quality. This includes setting appropriate parameters such as focal length, aperture, shutter speed, etc. For example, in a bright outdoor environment, the aperture may be set to f / 8, the shutter speed to 1 / 500 second, and the ISO to 100; while in a low-light indoor environment, an aperture of f / 2.8, a shutter speed of 1 / 60 second, and an ISO value of 800 may be selected.

[0102] With this setup, each camera can capture the same target from different angles, forming multi-camera images with overlapping areas. These overlapping areas usually occupy 30% to 50% of the area of ​​each image, providing sufficient information for subsequent image registration and stitching.

[0103] In the processing step, this method first uses a deep learning model to extract image features. This step is crucial for subsequent image registration and stitching.

[0104] Specifically, this method uses a convolutional neural network (CNN) to calculate the feature map of image data. The CNN model used here can be a classic ResNet, VGG and other network structures, or a lightweight network specially designed for multi-lens image stitching tasks.

[0105] Taking ResNet-50 as an example, its network structure contains 50 convolutional layers, which can extract rich multi-scale features. After the input image undergoes multiple convolution and pooling operations, the resulting feature map usually has 256 to 2048 channels, each of which represents a specific image feature.

[0106] After obtaining the feature map, this method detects key points on the feature map. Here, the improved FAST (Features from Accelerated Segment Test) corner detection algorithm can be used. The core idea of ​​the FAST algorithm is to compare the grayscale values ​​of 16 pixels around the pixel. If there are n consecutive points (n is usually 9 to 12) that are brighter or darker than the center point, the point is considered to be a corner point.

[0107] Finally, extract feature descriptors for the detected key points. Here, you can use the BRIEF (Binary Robust Independent Elementary Features) descriptor. The BRIEF descriptor randomly selects several pairs of pixels around the key point and compares their grayscale values ​​to obtain a binary string. Usually, the length of the BRIEF descriptor is 128 bits or 256 bits.

[0108] Through the above steps, this method can effectively extract the key features of the image, laying the foundation for subsequent image registration and stitching. This feature extraction method based on deep learning has stronger robustness and discriminability than traditional methods, and can better handle image stitching problems in complex scenes.

[0109] The method of the present invention realizes efficient and accurate image feature extraction through the above steps, providing a reliable basis for subsequent image registration and stitching. Compared with traditional methods, this method based on deep learning has stronger adaptability and robustness, and can better handle the problem of multi-lens image stitching in complex scenes.

[0110] In the following processing, the method will perform image registration, stitching weight calculation and final image stitching based on these extracted features. These steps will be described in detail in the subsequent description. Through this systematic processing flow, the present invention can achieve high-quality, real-time multi-lens image stitching, providing powerful technical support for various application scenarios. In one embodiment of the present invention, performing the image registration operation specifically includes:

[0111] Use the pre-learned similarity comparison function to match the feature values ​​of the key points of each pair of images;

[0112] Select multiple groups of most similar eigenvalue pairs;

[0113] Based on the eigenvalue pairs, transformation parameters for image registration are calculated.

[0114] This method uses a pre-learned similarity comparison function for feature value matching, which has higher accuracy and robustness than traditional Euclidean distance or Hamming distance. Specifically, the similarity comparison function can be implemented by a deep neural network, and its structure may include multiple fully connected layers and normalization layers.

[0115] For example, a typical similarity comparison function network structure is as follows:

[0116] 1. Input layer: Receives two feature descriptors to be compared, each descriptor is usually a 128-dimensional or 256-dimensional vector.

[0117] 2. Fully connected layer 1: input 2048 nodes, output 1024 nodes.

[0118] 3. L2 Normalization Layer

[0119] 4. Fully connected layer 2: input 1024 nodes, output 512 nodes.

[0120] 5. L2 Normalization Layer

[0121] 6. Fully connected layer 3: input 512 nodes and output 1 node, indicating the similarity score.

[0122] During training, positive and negative pairs can be used to optimize network parameters. Positive pairs are feature pairs from the same object or scene, while negative pairs are from different objects or scenes. By minimizing the distance of positive pairs and maximizing the distance of negative pairs, the network can learn an effective similarity measure.

[0123] In the feature matching stage, the method performs feature value matching on the key points of each pair of images. Specifically, for each key point in the first image, the two candidate points with the highest similarity are found in the second image. If the similarity of the best matching point is significantly higher than that of the second best matching point (for example, the ratio threshold is set to 0.7), it is considered a reliable match.

[0124] Next, the method selects multiple sets of the most similar feature value pairs. In practice, the top N pairs of matching points with the highest similarity scores are usually selected, where N can be dynamically adjusted according to the image size and complexity, and a typical value may be between 50 and 200.

[0125] Finally, based on these eigenvalue pairs, the method calculates the transformation parameters for image registration. In most cases, it can be assumed that there is a homography transformation between the images. The homography matrix H can be obtained by solving the following equation:

[0126]

[0127] Among them, (x, y) and (x', y') are the coordinates of the corresponding matching points in the two images.

[0128] In order to robustly estimate the homography matrix, this method uses the RANSAC (Random Sample Consensus) algorithm. The basic steps of RANSAC are as follows:

[0129] 1. Randomly select 4 pairs of matching points (the minimum number of points required to calculate the homography matrix).

[0130] 2. Use these 4 pairs of points to calculate the homography matrix H.

[0131] 3. Apply H to all matching points and calculate the number of inliers. Inliers are points whose transformation error is less than a certain threshold (e.g. 3 pixels).

[0132] 4. Repeat steps 1-3 multiple times (e.g. 1000 times) and select H with the most inliers as the best estimate.

[0133] By this method, the present invention can effectively perform image registration and obtain reliable results even in the presence of noise and mismatching.

[0134] In another embodiment of the present invention, calculating the image splicing weight specifically includes:

[0135] Use the attention mechanism to analyze the importance of each region in the feature map;

[0136] Based on the importance analysis result, calculating the weight parameter of each region;

[0137] The weight parameters are screened using a non-maximum suppression algorithm.

[0138] This method introduces an attention mechanism to analyze the importance of each region in the feature map. This method can adaptively focus on the key areas in the image and improve the accuracy and naturalness of the splicing. Specifically, the SpatialAttentionModule can be used to achieve this function.

[0139] The mathematical expression of the spatial attention module can be written as:

[0140] M(F)=σ(f 7×7 ([AvgPool(F); MaxPool(F)]))

[0141] Among them, F is the input feature map, f 7×7 represents a 7×7 convolution operation, σ is the sigmoid activation function, AvgPoo and MaxPool represent average pooling and maximum pooling operations respectively.

[0142] Based on the importance analysis results, this method calculates the weight parameters for each region. This can be achieved by multiplying the attention map with the original feature map:

[0143]

[0144] in, Represents element-wise multiplication.

[0145] In order to further optimize the weight parameters, this method uses the non-maximum suppression (NMS) algorithm for screening. The basic idea of ​​the NMS algorithm is to suppress adjacent repeated detections and only retain the local optimal results. In this method, NMS is used to screen the weight parameters to ensure that there will be no overly dense high-weight areas during splicing.

[0146] The specific steps of NMS are as follows:

[0147] 1. Sort all weight parameters from large to small.

[0148] 2. Select the largest weight parameter and add it to the result set.

[0149] 3. Calculate the IoU (Intersection over Union) of this weight parameter and all other parameters.

[0150] 4. Remove all weight parameters whose IoU is greater than a certain threshold (e.g. 0.5).

[0151] 5. Repeat steps 2-4 until all weight parameters have been processed.

[0152] Through this method, the present invention can obtain more reasonable and balanced splicing weights, thereby improving the quality of the final spliced ​​image.

[0153] In yet another embodiment of the present invention, generating a stitched image specifically includes:

[0154] Based on the stitching weight, weighted averaging is performed on the overlapping areas;

[0155] Use image fusion algorithm to eliminate splicing traces;

[0156] Performs a color consistency optimization operation.

[0157] This method first performs weighted averaging on the overlapping areas of the images based on the calculated stitching weights. Specifically, for each pixel (x, y) in the overlapping area, its value can be calculated using the following formula:

[0158]

[0159] Among them, I(x,y) is the pixel value after splicing, I i (x,y) is the pixel value of the i-th input image at that position, w i (x,y) is the corresponding weight.

[0160] In order to further eliminate the splicing traces, this method uses an image fusion algorithm. An effective choice is the Multi-resolution Spline Blending algorithm. The basic idea of ​​this algorithm is to perform fusion on different frequency bands and then merge the results. This method can effectively maintain the detailed information of the image while smoothing the transition area.

[0161] Finally, this method performs a color consistency optimization operation to solve the color difference problem that may exist between different cameras. A common method is color transfer based on histogram matching. The specific steps are as follows:

[0162] 1. Select an image as a reference image.

[0163] 2. For each color channel (R, G, B), calculate the cumulative histogram of the reference image and the image to be adjusted.

[0164] 3. For each pixel of the adjusted image, find its position in the original cumulative histogram and then map it to the corresponding position in the reference image cumulative histogram.

[0165] Through this method, the present invention can generate high-quality spliced ​​images, which not only maintains the detail information of the original image, but also achieves natural and smooth transition and consistent color expression.

[0166] In a preferred embodiment of the present invention, it also includes a depth estimation step:

[0167] Based on the multi-lens images, performing a stereo matching operation;

[0168] Generate a depth map;

[0169] Based on the depth map, a three-dimensional point cloud model is constructed.

[0170] This method takes advantage of multi-lens images and estimates the depth information of the scene through stereo matching operations. The core of stereo matching is to find the corresponding points in two images and calculate the depth through disparity. An efficient stereo matching algorithm is Semi-Global Matching (SGM).

[0171] The objective function of the SGM algorithm can be expressed as:

[0172]

[0173] Where D is the disparity map, p and q are image pixels, and C(p,D p ) is the disparity D of pixel p p The matching cost under , P1 and P2 are the penalty parameters of the smoothing term, and T[] is the indicator function.

[0174] By minimizing this energy function, we can get the optimal disparity map and then generate the depth map. The relationship between depth Z and disparity d can be expressed as:

[0175]

[0176] Where f is the focal length of the camera and B is the baseline distance between the two cameras.

[0177] Based on the generated depth map, this method constructs a 3D point cloud model. For each pixel (u,v) in the image, its corresponding 3D point (X,Y,Z) can be calculated by the following formula:

[0178]

[0179] Among them, (c x ,c y ) is the coordinate of the principal point of the image, f x and f y is the focal length in x and y directions (in pixels).

[0180] Through this method, the present invention can not only generate high-quality two-dimensional stitched images, but also provide three-dimensional structural information of the scene, laying the foundation for subsequent scene understanding and interactive applications. This multi-dimensional scene representation greatly expands the application scope of the present invention, making it have broad application prospects in the fields of virtual reality, augmented reality, 3D reconstruction, etc. In another preferred embodiment of the present invention, it also includes a multimodal fusion step:

[0181] Acquire image data of different imaging modalities;

[0182] Based on wavelet transform, perform multimodal image fusion operation;

[0183] Dynamically adjust the weights of different modes according to the characteristics of the scene.

[0184] This method first obtains image data of different imaging modalities, which may include visible light images, infrared images, depth images, etc. Each modality can provide unique scene information, and by fusing this information, a more comprehensive and robust scene representation can be obtained.

[0185] In the multimodal image fusion operation, this method adopts a method based on wavelet transform. Wavelet transform can decompose the image into sub-bands of different scales and directions, which allows us to perform fusion operations in different frequency domains. Specifically, discrete wavelet transform (DWT) can be used to implement this process.

[0186] For the input image I, the two-dimensional DWT can be expressed as:

[0187] LL j+1 =(h*(h*LL j ) ↓2,1 ) ↓1,2 (1)

[0188] LH j+1 =(h*(g*LL j ) ↓2,1 ) ↓1,2 (2)

[0189] HL j+1 =(g*(h*LL j ) ↓2,1 ) ↓1,2 (3)

[0190] HH j+1 =(g*(g*LL j ) ↓2,1 ) ↓1,2 (4)

[0191] Here, h and g are low-pass and high-pass filters respectively, ↓1,2 means 2x downsampling in the first dimension, and j represents the decomposition level.

[0192] After obtaining the wavelet coefficients of each modality image, this method uses different fusion rules according to the characteristics of different sub-bands. For example, for the low-frequency sub-band (LL), a weighted average method can be used:

[0193] LL fused =w1·LL1+w2·LL2+...+w n ·LL n

[0194] For high frequency subbands (LH, HL, HH), the maximum coefficient can be selected:

[0195] HH fused (x,y)=max{HH1(x,y),HH2(x,y),...,HH n (x,y)}

[0196] Among them, w i is the weight of the ith mode, and (x,y) is the spatial position of the coefficient.

[0197] Finally, this method dynamically adjusts the weights of different modalities according to the characteristics of the scene. This can be achieved by analyzing the local information entropy or gradient magnitude of each modality image. For example, the weight w can be defined as i as follows:

[0198]

[0199] Among them, entropy(I i ) is the local information entropy of the i-th modality image.

[0200] Through this dynamic weight adjustment mechanism, this method can adaptively fuse multimodal information according to the characteristics of different scenes, thereby obtaining a more robust and information-rich fused image.

[0201] In yet another embodiment of the present invention, updating the shooting parameters specifically includes:

[0202] Real-time monitoring of image quality indicators;

[0203] Based on a reinforcement learning algorithm, dynamically adjusting shooting parameters according to the image quality index;

[0204] A parameter adjustment model is established to guide subsequent shooting.

[0205] The method first monitors image quality indicators in real time, which may include but are not limited to: clarity, contrast, color saturation, noise level, etc. For example, the Laplacian operator can be used to evaluate the clarity of the image:

[0206]

[0207] Where W and H are the width and height of the image respectively, is the Laplacian operator, and I(x,y) is the pixel value of the image at position (x,y).

[0208] Based on these quality indicators, this method uses a reinforcement learning algorithm to dynamically adjust the shooting parameters. Specifically, the Q-learning algorithm can be used to implement this process. Define the state space S as the current image quality indicator, the action space A as the adjustable shooting parameters (such as exposure time, ISO sensitivity, etc.), and the reward function R as the degree of improvement in image quality.

[0209] The core update formula of Q-learning is as follows:

[0210]

[0211] Among them, Q(s t ,a t ) is the state-action value function, α is the learning rate, γ is the discount factor, r t+1 It’s an instant reward.

[0212] Through continuous trial and learning, the system can gradually build an effective parameter adjustment model. This model can be expressed as a policy function π(s), which outputs the optimal action (i.e., shooting parameter adjustment) based on the current state s.

[0213] Preferably, the method can also use experience replay technology to improve learning efficiency. Specifically, an experience pool D can be maintained to store the past state-action-reward-next state quadruple (s t ,a t ,r t ,s t+1 ). At each update, a mini-batch is randomly sampled from the experience pool for learning, which can break the correlation between samples and improve the stability of learning.

[0214] Through this adaptive parameter adjustment mechanism, the method can continuously optimize the shooting parameters and continuously improve the quality of the original image, thereby providing better input for subsequent image stitching.

[0215] In one embodiment of the present invention, a hardware acceleration step is also included:

[0216] Mapping the convolutional layer of the deep learning model to the FPGA for implementation;

[0217] Use GPU to accelerate matrix operations and deep learning reasoning;

[0218] Implement pipeline parallel processing architecture to maximize hardware utilization.

[0219] This method makes full use of the advantages of heterogeneous computing platforms and improves the real-time performance of the system through hardware acceleration. First, the convolution layer of the deep learning model is mapped to the FPGA for implementation. The convolution operation can be expressed as:

[0220]

[0221] Among them, x is the input, h is the convolution kernel, and y is the output.

[0222] On FPGA, efficient convolution operations can be achieved by designing dedicated convolution accelerators. The typical convolution accelerator architecture includes: input buffer, weight buffer, systolic array, and output buffer. The systolic array consists of multiple processing units (PEs), each of which is responsible for a part of the convolution operation, and efficient parallel processing is achieved through data pipelining.

[0223] For matrix operations and deep learning reasoning, this method uses the parallel processing capabilities of the GPU to accelerate. The GPU's SIMT (Single Instruction, Multiple Threads) architecture is very suitable for processing large-scale parallel floating-point operations. For example, for matrix multiplication C = A\times B, the calculation can be distributed to multiple GPU threads for parallel execution:

[0224]

[0225]

[0226] In order to maximize hardware utilization, this method implements a pipeline parallel processing architecture. The entire processing flow is divided into multiple stages, such as image preprocessing, feature extraction, image registration, splicing weight calculation, image fusion, etc. Each stage is executed on a dedicated hardware unit and data is exchanged through cache or on-chip memory.

[0227] For example, the following pipeline structure can be designed:

[0228] 1.FPGA: image preprocessing → feature extraction;

[0229] 2. GPU: feature matching → image registration;

[0230] 3.CPU: splicing weight calculation;

[0231] 4. GPU: image fusion → post-processing;

[0232] Through this pipeline design, the system can process multiple frames of images at the same time, greatly improving the processing throughput. At the same time, through reasonable task division and scheduling, the utilization rate of each hardware unit can be maximized to avoid performance bottlenecks.

[0233] Preferably, the method also adopts a dynamic power management strategy. According to the current workload and performance requirements, the clock frequency and voltage of each hardware unit are dynamically adjusted to minimize power consumption while ensuring performance. This is particularly important for portable or battery-powered multi-lens image stitching devices.

[0234] Through the above hardware acceleration strategy, the method of the present invention can significantly improve the processing speed while ensuring high-quality image stitching effects, and realize real-time multi-lens image stitching. This makes it possible to apply the present invention in application scenarios with high real-time requirements such as high-speed photography, real-time monitoring, and unmanned driving.

[0235] The above description is only a preferred specific implementation manner of the present invention; however, the protection scope of the present invention is not limited thereto; any person familiar with the art who, within the scope disclosed by the present invention, makes equivalent replacements or changes based on the scheme and improved concepts of the present invention shall be covered within the protection scope of the present invention.

Claims

1. A multi-lens image real-time stitching optimization method based on deep learning, characterized in that: include: The acquisition steps include: Acquire image data collected by multiple lenses; Acquiring initial shooting parameters of the multiple lenses; Processing steps include: Based on the image data, extract image features using a deep learning model; Performing an image registration operation according to the image features; Based on the registration result, calculating the image stitching weight; generating a spliced ​​image according to the splicing weight; Output steps include: Outputting the stitched image; The shooting parameters are updated based on the quality assessment result of the stitched image.

2. The method according to claim 1, characterized in that The acquisition step specifically includes: Arrange and fix multiple lenses around the target area to be photographed; Initializing parameters of each of the lenses; Each of the lenses is used to shoot the same target from different angles, thereby forming multi-lens images with overlapping areas.

3. The method according to claim 1, characterized in that The method of extracting image features by using a deep learning model specifically includes: Calculating a feature map of the image data using a convolutional neural network; Detecting key points on the feature map; A feature descriptor of the key point is extracted.

4. The method according to claim 1, characterized in that: The performing of the image registration operation specifically includes: Use the pre-learned similarity comparison function to match the feature values ​​of the key points of each pair of images; Select multiple groups of most similar eigenvalue pairs; Based on the eigenvalue pairs, transformation parameters for image registration are calculated.

5. The method according to claim 1, characterized in that The calculating of the image stitching weight specifically includes: Use the attention mechanism to analyze the importance of each region in the feature map; Based on the importance analysis result, calculating the weight parameter of each region; The weight parameters are screened using a non-maximum suppression algorithm.

6. The method according to claim 1, characterized in that Generating the spliced ​​image specifically includes: Based on the stitching weight, weighted averaging is performed on the overlapping areas; Use image fusion algorithm to eliminate splicing traces; Performs a color consistency optimization operation.

7. The method according to claim 1, characterized in that It also includes the depth estimation step: Based on the multi-lens images, performing a stereo matching operation; Generate a depth map; Based on the depth map, a three-dimensional point cloud model is constructed.

8. The method according to claim 1, characterized in that It also includes multimodal fusion steps: Acquire image data of different imaging modalities; Based on wavelet transform, perform multimodal image fusion operation; Dynamically adjust the weights of different modes according to the characteristics of the scene.

9. The method according to claim 1, characterized in that: The updating of the shooting parameters specifically includes: Real-time monitoring of image quality indicators; Based on a reinforcement learning algorithm, dynamically adjusting shooting parameters according to the image quality index; A parameter adjustment model is established to guide subsequent shooting.

10. The method according to claim 1, characterized in that Also includes hardware acceleration steps: Mapping the convolutional layer of the deep learning model to the FPGA for implementation; Use GPU to accelerate matrix operations and deep learning reasoning; Implement pipeline parallel processing architecture to maximize hardware utilization.

Images