A camera frame rate self-adaptive adjustment method and system based on image overlap degree

Abstract

The present application relates to the field of image acquisition control, and provide a kind of camera frame rate self-adapting adjustment method and system based on image overlap degree.The method comprises: preset parameter, obtain initialization configuration;According to the initialization configuration, start image acquisition, and the complexity analysis is carried out to current frame image and reference frame image, obtain image complexity index;According to the image complexity index and current computing load, the current frame image and reference frame image are pretreated, and obtain preprocessed image pair;The preprocessed image pair is input based on the initialization configuration feature matching algorithm, obtain interframe overlap degree estimate value and multi-frame average overlap degree;With the interframe overlap degree estimate value and multi-frame average overlap degree and preset overlap degree threshold range are compared, current acquisition frame rate is adjusted, and obtain optimization frame rate.The present application reduces the hardware integration complexity of image system, and improves the adaptability of frame rate adjustment.

Description

Technical Field

[0001] This invention belongs to the field of image acquisition and control technology, and particularly relates to a camera frame rate adaptive adjustment method and system based on image overlap. Background Technology

[0002] In scenarios requiring high imaging continuity, such as UAV remote sensing and industrial inspection, high-resolution image sensors have become crucial data acquisition tools. During data acquisition, the frame rate directly impacts the redundancy and completeness of the acquired data. When the camera is stationary or moving slowly, a high frame rate generates numerous duplicate images, wasting storage and computing resources; conversely, a low frame rate can lead to the omission of key features, affecting the quality of subsequent data processing. Therefore, dynamically adjusting the camera frame rate according to the actual acquisition scenario is a core issue that needs to be addressed in the design of image acquisition systems.

[0003] Existing frame rate adjustment methods typically rely on external sensors to perceive motion or on detecting changes in the image based on pixel differences to trigger frame rate switching. However, these methods cannot directly reflect the actual overlap of image content between frames, are prone to misjudgment in scenes with sparse texture or drastic lighting changes, and their dependence on external sensors increases the integration complexity of the system. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of the invention is to provide a camera frame rate adaptive adjustment method and system based on image overlap.

[0005] This invention provides a camera frame rate adaptive adjustment method based on image overlap, comprising: S1. Set preset parameters to obtain the initial configuration; S2. Start image acquisition according to the initialization configuration, and perform complexity analysis on the current frame image and the reference frame image to obtain the image complexity index. S3. Based on the image complexity index and the current computing load, preprocess the current frame image and the reference frame image to obtain a preprocessed image pair. S4. Input the preprocessed image pair into the feature matching algorithm based on the initial configuration to obtain the estimated value of inter-frame overlap and the average overlap of multiple frames. S5. The estimated inter-frame overlap and the average overlap of multiple frames are compared with the preset overlap threshold range to adjust the current acquisition frame rate and obtain an optimized frame rate.

[0006] According to the present invention, a camera frame rate adaptive adjustment method based on image overlap is provided, wherein step S1 specifically includes: The image acquisition parameters are initialized, including an initial acquisition frame rate for determining the starting point of dynamic frame rate adjustment, and an upper and lower frame rate limit for defining the upper and lower boundaries of frame rate adjustment, thereby obtaining the image acquisition initialization configuration. The image processing parameters are initialized, including the upper and lower limits of the overlap threshold range for frame rate adjustment determination, the low and high entropy thresholds of image complexity for preprocessing strategy selection, the initial frame interval and calculation load threshold for inter-frame sampling, and the image processing initialization configuration is obtained. The algorithm configuration parameters are initialized, including the upper limit of the number of feature points for feature point detection, the image pyramid scaling ratio, the number of pyramid layers and the corner response threshold, as well as the inlier pixel error threshold and the iteration success probability for robust matching, to obtain the algorithm initialization configuration. Pre-allocate the logarithmic lookup table array for Shannon entropy calculation and the descriptor distance lookup table array for descriptor distance calculation to obtain the array configuration.

[0007] According to the camera frame rate adaptive adjustment method based on image overlap provided by the present invention, step S2 further includes: S21. Start image acquisition according to the image acquisition initialization configuration in the initialization configuration to obtain the current frame image; S22. The current frame image is downsampled using bilinear interpolation to obtain a downsampled image; S23. The downsampled image is converted to grayscale using an integer approximate weighted summation method to obtain a grayscale downsampled image; S24. Statistically analyze the pixel grayscale probability distribution of the grayscale downsampled image, and calculate the Shannon entropy by using a logarithmic lookup table array to obtain the image complexity index represented by the image complexity entropy value.

[0008] According to the camera frame rate adaptive adjustment method based on image overlap provided by the present invention, step S23 further includes: S231: Multiply the three channel pixel values ​​of the downsampled image by the corresponding integer approximate weight coefficients respectively and sum them to obtain the summation result; S232: Perform a right shift operation on the summation result to obtain the calculation result; S233: Normalize the calculation result to the grayscale value range to obtain a grayscale downsampled image.

[0009] According to the camera frame rate adaptive adjustment method based on image overlap provided by the present invention, step S3 specifically includes: When the current computing load is less than or equal to the preset computing load threshold, the cached grayscale downsampled image and the corresponding reference frame grayscale downsampled image are reused as a preprocessed image pair. When the current computational load is greater than the preset computational load threshold and the average overlap of multiple frames is less than the lower limit of the overlap threshold range, the downsampled image is downsampled a second time at a high compression ratio to obtain a high compression ratio downsampled image. The high compression ratio downsampled image is used as the current frame in the preprocessed image pair to obtain the preprocessed image pair.

[0010] According to the camera frame rate adaptive adjustment method based on image overlap provided by the present invention, step S3 further includes: S31. When the current computing load is greater than the preset computing load threshold, and the average overlap of multiple frames is less than or equal to the lower limit of the preset overlap threshold range, the frame sequence is sampled every other frame according to the current frame interval, the complete overlap is calculated for the sampled frames, and the non-sampled frames are extrapolated based on the overlap change trend to obtain the current frame overlap estimate. Step S31 further includes: S311. The overlap of the most recent complete calculation frame is called the most recent overlap. The overlap change rate is obtained by dividing the difference in overlap of the most recent multiple complete calculation frames by the corresponding time interval. S312. For non-sampled frames, the ratio of the number of frames from the nearest complete calculation frame to the current sampling frame rate is used as the time offset. The product of the nearest overlap, the overlap change rate, and the time offset is added to obtain the estimated overlap value of the current non-sampled frame. S313. Perform full overlap calculation every fixed number of frames, verify the actual overlap with the most recent estimated value using the relative error formula, and continuously optimize and adjust the current frame interval based on the verification result to obtain the updated frame interval.

[0011] According to the camera frame rate adaptive adjustment method based on image overlap provided by the present invention, step S4 further includes: S41. Perform directional feature point detection on the current frame image and the reference frame image in the preprocessed image pair respectively to obtain the feature point set and the corresponding binary descriptor subset of each frame. S42. Using the descriptor distance lookup table array in the initialization configuration, the descriptor set is matched using the Hamming distance lookup table method to obtain an initial set of matching point pairs; S43. Input the initial set of matching point pairs into a robust estimation algorithm to filter and obtain the set of interior point matching point pairs and the corresponding geometric transformation matrix; S44. Calculate the ratio of the geometric overlap area of ​​the two frames to the area of ​​a single frame image based on the geometric transformation matrix to obtain the inter-frame overlap estimate, and take the average of the inter-frame overlap estimates of the most recent frames to obtain the multi-frame average overlap.

[0012] According to the camera frame rate adaptive adjustment method based on image overlap provided by the present invention, in step S43, when the number of inliers in the inlier matching point pair set is lower than the preset inlier number threshold, it is determined that the reliability of the current overlap calculation is insufficient, the preprocessed image pair is replaced with the original grayscale image pair, and the inter-frame overlap estimate and the multi-frame average overlap are recalculated.

[0013] According to the camera frame rate adaptive adjustment method based on image overlap provided by the present invention, step S5 specifically includes: When the estimated inter-frame overlap and the average overlap of multiple frames are both greater than the upper limit of the overlap threshold range, the current acquisition frame rate is multiplied by the frame reduction coefficient and the frame rate lower limit is taken as the maximum value to obtain the optimized frame rate after reducing the frame rate. When the estimated inter-frame overlap and the average overlap of multiple frames are both less than or equal to the lower limit of the overlap threshold range, the minimum value of the current acquisition frame rate multiplied by the frame increase coefficient and the frame rate upper limit is taken to obtain the optimized frame rate after the frame rate is increased. When the estimated inter-frame overlap is inconsistent with the trend of the average overlap of multiple frames, the current acquisition frame rate is kept unchanged and the current acquisition frame rate is used as the optimized frame rate.

[0014] The present invention also provides a camera frame rate adaptive adjustment system based on image overlap, for performing a camera frame rate adaptive adjustment method based on image overlap as described in any of the above claims, comprising: Initialization module: used to preset parameters and obtain initialization configuration; Analysis module: used to receive image acquisition data based on the initialization configuration, and perform complexity analysis on the current frame image and the reference frame image to obtain image complexity index; Preprocessing module: used to preprocess the current frame image and the reference frame image according to the image complexity index and the current computing load to obtain a preprocessed image pair; Matching module: used to input the preprocessed image pairs into a feature matching algorithm based on the initial configuration to obtain the estimated inter-frame overlap and the average overlap of multiple frames; Adjustment module: used to compare the estimated inter-frame overlap and the average overlap of multiple frames with the preset overlap threshold range, adjust the current acquisition frame rate, and obtain an optimized frame rate.

[0015] This invention uses the estimated inter-frame overlap and the average overlap of multiple frames as the core basis for frame rate adjustment, thus eliminating the dependence on external motion sensors. This allows the entire image acquisition system to operate independently, relying only on the content of the images themselves, thereby significantly reducing the hardware integration complexity of the system and broadening its application scope on resource-constrained platforms such as UAV remote sensing and industrial inspection. In the image preprocessing stage, this invention compresses high-resolution images to a scale suitable for real-time processing on embedded platforms through cascaded downsampling and grayscale conversion. Grayscale conversion and Shannon entropy calculation are completed by integer approximate weighted operations and lookup table methods, respectively, so that the operation process is completely free from floating-point dependence. While preserving the main texture information, it significantly reduces the time overhead of single-frame processing, enabling the system to maintain stable real-time processing capabilities under high-resolution input conditions. In the overlap estimation stage, the introduction of pre-allocated logarithmic lookup table arrays and Hamming distance lookup table arrays allows the most time-consuming descriptor distance calculation in feature matching to be completed by memory access instead of bit-by-bit operations. Combined with the frame-by-frame sampling and overlap extrapolation estimation mechanism, it can still effectively track the trend of inter-frame overlap changes even under high computational load. The frame interval is continuously optimized by periodically forcing complete calculation and error checking, achieving a dynamic balance between estimation efficiency and estimation accuracy. In the frame rate adjustment stage, the upper and lower limits of the overlap threshold range are used as the judgment boundary. Combined with the frame rate adjustment coefficient and the upper and lower limits of the frame rate, the frame rate adjustment can not only respond quickly to scene changes, but also avoid frequent and drastic fluctuations in the frame rate. This reduces redundant data collection while ensuring that key image content is not missed. Attached Figure Description

[0016] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0017] Figure 1 A schematic flowchart of a camera frame rate adaptive adjustment method based on image overlap is provided for an embodiment of the present invention; Figure 2 This is a schematic diagram of a camera frame rate adaptive adjustment system based on image overlap, provided as an embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0019] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0020] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0022] The embodiments of the present invention are described below with reference to the figures.

[0023] like Figure 1 As shown, the present invention provides a camera frame rate adaptive adjustment method based on image overlap, comprising: S1. Set parameters to obtain the initial configuration.

[0024] Specifically, step S1 includes: The image acquisition parameters are initialized, including an initial acquisition frame rate for determining the starting point of dynamic frame rate adjustment, and an upper and lower frame rate limit for defining the upper and lower boundaries of frame rate adjustment, thereby obtaining the image acquisition initialization configuration. The image processing parameters are initialized, including the upper and lower limits of the overlap threshold range for frame rate adjustment determination, the low and high entropy thresholds of image complexity for preprocessing strategy selection, the initial frame interval and calculation load threshold for inter-frame sampling, and the image processing initialization configuration is obtained. The algorithm configuration parameters are initialized, including the upper limit of the number of feature points for feature point detection, the image pyramid scaling ratio, the number of pyramid layers and the corner response threshold, as well as the inlier pixel error threshold and the iteration success probability for robust matching, to obtain the algorithm initialization configuration. Pre-allocate the logarithmic lookup table array for Shannon entropy calculation and the descriptor distance lookup table array for descriptor distance calculation to obtain the array configuration.

[0025] Furthermore, in step S1, when the system starts, the present invention first completes the initial configuration of three types of parameters. Regarding image acquisition parameters, the initial acquisition frame rate is set to 20 FPS as the starting point for dynamic frame rate adjustment. At the same time, a lower limit and an upper limit of the frame rate are set to constrain the subsequent frame rate adjustment range within a reasonable range.

[0026] Regarding image processing parameters, a high overlap threshold of 75% and a low overlap threshold of 55% are set, which together constitute the decision boundary for frame rate adjustment; a low entropy threshold of 5.0 and a high entropy threshold of 7.0 are set, which are used to select different preprocessing strategies according to the complexity of the image scene in subsequent steps; an initial frame interval of 2 and a computational load threshold of 70% are set, which are used to trigger the frame-by-frame sampling strategy.

[0027] Regarding the algorithm configuration parameters, the maximum number of feature points for the ORB feature detector is set to 500, the image pyramid scaling factor is 1.2, the number of pyramid layers is 8, and the FAST corner response threshold is 20; for the RANSAC robust matching algorithm, the in-point pixel error threshold is set to 5 pixels and the iteration success probability is 0.99.

[0028] After completing the above parameter configuration, this invention also pre-allocates the log2 lookup table array and the Hamming distance lookup table array. Both of them have their complete calculation results written in memory in advance. Subsequently, index access is used directly instead of real-time numerical calculation. At the same time, a CPU / GPU load detection thread is started to continuously monitor the computing resource usage. This completes all initialization configuration.

[0029] S2. Start image acquisition according to the initialization configuration, and perform complexity analysis on the current frame image and the reference frame image to obtain the image complexity index.

[0030] Step S2 further includes: S21. Start image acquisition according to the image acquisition initialization configuration in the initialization configuration to obtain the current frame image.

[0031] In step S21, the present invention starts the image sensor according to the initial acquisition frame rate set in step S1, and continuously acquires high-resolution RGB images at the frame rate. The single-frame color image acquired each time is the current frame image. At the same time, the frame that has been processed in the previous moment is taken as the reference frame image. The two together are used as input data for subsequent overlap analysis.

[0032] S22. The current frame image is downsampled using bilinear interpolation to obtain a downsampled image.

[0033] Since the original high-resolution color image has a large data volume, directly feeding it into the subsequent feature detection process would result in high computational overhead. Therefore, in step S22, this invention performs downsampling processing on the current frame image. Specifically, using a downsampling factor of 1 / 4, bilinear interpolation is employed to compress the spatial resolution of the current frame image. During execution, bilinear interpolation performs two linear interpolations at each pixel position in the output image, based on the gray values ​​of its four adjacent pixels in the original image, weighted by horizontal and vertical distances, to finally obtain the output pixel value at that position. After the processing in step S22, the spatial resolution of the current frame image is reduced to 1 / 4 of the original, and the data volume is compressed to 1 / 16 of the original, resulting in a downsampled image.

[0034] S23. The downsampled image is converted to grayscale using an integer approximate weighted summation method to obtain a grayscale downsampled image.

[0035] Step S23 further includes: S231: Multiply the three channel pixel values ​​of the downsampled image by the corresponding integer approximate weight coefficients and sum them to obtain the summation result.

[0036] Furthermore, this invention performs grayscale processing on the downsampled image, converting the red, green, and blue channel values ​​of each pixel into a single grayscale value. The standard method for grayscale processing is based on the ITU-R BT.601 standard, using 0.299, 0.587, and 0.114 as floating-point weighting coefficients for the red, green, and blue channels, respectively, and then weighting and summing the values ​​of the three channels to obtain the grayscale value.

[0037] However, floating-point multiplication operations have high computational overhead on embedded processors. Therefore, this invention converts the above floating-point weight coefficients into integers for operation. The conversion method is to multiply the three floating-point coefficients by 256 and round them to the nearest integer to obtain the integers 77 for the red channel, 150 for the green channel, and 29 for the blue channel. These three integers are used to replace the original floating-point coefficients in subsequent multiplication operations.

[0038] Subsequently, for each pixel in the downsampled image, its red channel value is multiplied by 77, its green channel value by 150, and its blue channel value by 29. These three multipliers are then summed to obtain the result. For example, if a pixel has a red channel value of 200, a green channel value of 150, and a blue channel value of 100, the sum would be 200 multiplied by 77 plus 150 multiplied by 150 plus 100 multiplied by 29, which equals 15400 plus 22500 plus 2900, resulting in a sum of 40800. At this point, the sum is in an extended value range much greater than 255, requiring steps S232 and S233 to compress it back to the grayscale value range of 0 to 255.

[0039] S232: Perform a right shift operation on the summation result to obtain the calculation result.

[0040] After obtaining the summation result, in step S232, the present invention performs a right shift operation of 8 bits on the above summation result, which is equivalent to dividing the summation result by 256. After the operation, the numerical range is compressed from the extended range of the three-channel weighted accumulation back to the integer range to obtain the operation result.

[0041] S233: Normalize the calculation result to the grayscale value range to obtain a grayscale downsampled image.

[0042] In step S233, the present invention constrains the calculation result to a gray value range of 0 to 255. Each pixel position finally corresponds to a gray value. The single-channel image composed of the gray values ​​of all pixels is the gray downsampled image. After obtaining the gray downsampled image, the present invention simultaneously caches the gray downsampled image for direct reuse when selecting the preprocessing strategy in step S3.

[0043] S24. Statistically analyze the pixel grayscale probability distribution of the grayscale downsampled image, and calculate the Shannon entropy by using a logarithmic lookup table array to obtain the image complexity index represented by the image complexity entropy value.

[0044] In step S24, the present invention statistically analyzes the grayscale value distribution of all pixels in the grayscale downsampled image, calculates the frequency of each grayscale level in all pixels, and divides the frequency by the total number of pixels to obtain the probability value of each grayscale level. Then, using each probability value as an index, it queries the logarithmic lookup table array pre-allocated in step S1 to directly obtain the corresponding logarithmic value. The probability value of each grayscale level is multiplied by the corresponding logarithmic value, and the sum and negative are taken to obtain the image complexity entropy value, which is the image complexity index.

[0045] The higher the final image complexity entropy value, the more dispersed the pixel grayscale distribution of the grayscale downsampled image and the more complex the scene texture; the lower the image complexity entropy value, the more concentrated the grayscale distribution and the simpler the scene texture. This entropy value will be used as the judgment input for the selection of the preprocessing strategy in step S3.

[0046] S3. Based on the image complexity index and the current computational load, preprocess the current frame image and the reference frame image to obtain a preprocessed image pair.

[0047] Specifically, step S3 includes: When the current computing load is less than or equal to the preset computing load threshold, the cached grayscale downsampled image and the corresponding reference frame grayscale downsampled image are reused as a preprocessed image pair.

[0048] When the current computational load is greater than the preset computational load threshold and the average overlap of multiple frames is less than the lower limit of the overlap threshold range, the downsampled image is downsampled a second time at a high compression ratio to obtain a high compression ratio downsampled image. The high compression ratio downsampled image is used as the current frame in the preprocessed image pair to obtain the preprocessed image pair.

[0049] After step S2 completes the calculation of the image complexity entropy value, the present invention uses the comparison result between the current computing load and the computing load threshold as the first judgment condition, and the comparison result between the average overlap of multiple frames and the lower limit of the overlap threshold range as the second judgment condition, and selects the corresponding preprocessing strategy based on the combination of the two conditions.

[0050] Furthermore, when the current computing load does not exceed the computing load threshold, the present invention directly reuses the grayscale downsampled image and the corresponding reference frame grayscale downsampled image that have been cached in step S2, and combines the two as a preprocessed image pair for output, without having to re-perform any downsampling or grayscale operation on the original image.

[0051] When the current computational load exceeds the computational load threshold and the average overlap of multiple frames is less than the lower limit of the overlap threshold range, it indicates that the content changes rapidly between frames in the current scene, and computational resources are strained. This invention takes the downsampled image obtained in step S22 and performs a second downsampling process on it at a compression ratio of 1 / 6 to 1 / 8 to obtain a high-compression-ratio downsampled image. This high-compression-ratio downsampled image is used as the current frame in the preprocessed image pair and combined with the corresponding reference frame to output the preprocessed image pair.

[0052] S31. When the current computing load is greater than the preset computing load threshold, and the average overlap of multiple frames is less than or equal to the lower limit of the preset overlap threshold range, the frame sequence is sampled every other frame according to the current frame interval, the complete overlap is calculated for the sampled frames, and the non-sampled frames are extrapolated based on the overlap change trend to obtain the current frame overlap estimate.

[0053] When the current computational load is greater than the computational load threshold, and the average overlap of multiple frames is greater than or equal to the lower limit of the overlap threshold range, the process proceeds to step S31, which involves inter-frame sampling.

[0054] In step S31, the present invention uses the modulo of the current frame number and the current frame interval to determine whether the current frame is a sampled frame. For sampled frames with a modulo result of zero, the present invention performs a complete overlap calculation to obtain the actual overlap of the frame and updates the most recent overlap record. For non-sampled frames with a modulo result of non-zero, the present invention skips the complete calculation and instead extrapolates the overlap of the current frame based on the overlap change trend to obtain the estimated overlap value of the current frame.

[0055] Step S31 further includes: S311. Let the overlap of the most recent complete calculation frame be the most recent overlap. Divide the overlap difference of the most recent multiple complete calculation frames by the corresponding time interval to obtain the overlap change rate.

[0056] In step S311, the present invention records the actual overlap obtained from the most recent complete calculation frame as the most recent overlap. Then, it takes the overlap values ​​from the five most recent complete calculation frames, divides the difference between two adjacent overlap values ​​by the corresponding time interval to obtain the single-step change rate for each step, and then takes the average of the five single-step change rates to obtain the overlap change rate. It should be noted that the present invention uses multiple averages rather than a single difference in step S311 to smooth out the interference of short-term noise on the change rate estimation.

[0057] S312. For non-sampled frames, the ratio of the number of frames from the nearest complete calculated frame to the current sampling frame rate is used as the time offset. The product of the nearest overlap, the overlap change rate, and the time offset is added to obtain the estimated overlap value of the current non-sampled frame.

[0058] For non-sampled frames, this invention uses the difference between the current frame number and the most recent complete calculated frame number as the frame number offset. The frame number offset is divided by the current sampling frame rate to obtain the time offset. Then, the overlap change rate is multiplied by the time offset to obtain the overlap change. The most recent overlap is then added to the overlap change to obtain the estimated overlap value of the current non-sampled frame. This estimated value is based on the actual overlap of the most recent complete calculated frame and is linearly extrapolated to the current frame's time according to the overlap change rate, reflecting the predicted overlap of the current frame.

[0059] S313. Perform full overlap calculation every fixed number of frames, verify the actual overlap with the most recent estimated value using the relative error formula, and continuously optimize and adjust the current frame interval based on the verification result to obtain the updated frame interval.

[0060] Furthermore, in step S313, the present invention forces a full overlap calculation to be performed on the current frame every ten frames to obtain the actual overlap. Then, the difference between the actual overlap and the estimated overlap value of the most recent non-sampled frame is divided by the actual overlap to obtain the relative error. When the relative error is greater than 5%, the current frame interval is decreased by 1; when the relative error is less than 3%, the current frame interval is increased by 1. The adjustment range of the frame interval is limited to between 1 and 5 to obtain the updated frame interval, and the updated frame interval is used to drive the inter-frame sampling determination of subsequent frame sequences.

[0061] S4. Input the preprocessed image pair into the feature matching algorithm based on the initial configuration to obtain the estimated value of inter-frame overlap and the average overlap of multiple frames.

[0062] Step S4 further includes: S41. Perform directional feature point detection on the current frame image and the reference frame image in the preprocessed image pair respectively to obtain the feature point set and the corresponding binary descriptor set of each frame.

[0063] In step S41, the present invention independently performs directional feature point detection on the current frame image and the reference frame image in the preprocessed image pair. Specifically, the directional feature point detector uses the corner response threshold as the screening criterion to detect corner points with significant local gray-level changes on each layer of the multi-scale pyramid of the image. Each layer of the pyramid is obtained by scaling down the previous layer image according to the scaling factor, and a total of eight layers of pyramid are constructed.

[0064] After detecting corner points, this invention calculates the direction of the gray-scale centroid of the neighborhood of each corner point, and rotates and aligns the binary descriptor of the corner point in this direction. Finally, each corner point corresponds to a binary string descriptor of fixed length.

[0065] The coordinates of all corner points in each of the two frames constitute a set of feature points, and the corresponding set of all binary descriptors constitutes a set of binary descriptors. Finally, each frame outputs a set of feature points and a set of binary descriptors.

[0066] S42. Using the descriptor distance lookup table array in the initialization configuration, the descriptor set is matched using the Hamming distance lookup table method to obtain an initial set of matching point pairs.

[0067] In step S42, the present invention uses each descriptor in the current frame binary descriptor set as the query object, and calculates the Hamming distance with each descriptor in the reference frame binary descriptor set. The Hamming distance measures the total number of bits with different values ​​at corresponding positions in two binary strings of equal length. The smaller the value, the more similar the two descriptors are.

[0068] Specifically, this invention replaces the bitwise XOR operation with the pre-allocated descriptor distance lookup table array in step S1. For each pair of descriptors, the table is looked up and accumulated byte by byte to directly obtain the Hamming distance value. After obtaining the value, the reference frame descriptor with the smallest Hamming distance is taken as the matching object of the current frame descriptor. The feature point coordinate pairs that are the best match between the two frames are combined to finally obtain the initial set of matching point pairs.

[0069] S43. Input the initial set of matching point pairs into a robust estimation algorithm to filter and obtain the set of interior point matching point pairs and the corresponding geometric transformation matrix.

[0070] In step S43, the present invention inputs the initial set of matched point pairs into a random consistency sampling algorithm to remove outliers. Each time, the random consistency sampling algorithm randomly selects the minimum number of point pairs from the initial set of matched point pairs, calculates a candidate geometric transformation matrix based on these point pairs, then substitutes all point pairs from the initial set of matched point pairs into the candidate matrix, calculates the reprojection error of each point pair after transformation, marks point pairs with reprojection errors less than the inlier pixel error threshold as inliers, and counts the number of inliers.

[0071] Finally, after the above random sampling and interior point statistics process is repeated for the number of iterations corresponding to the success probability, the candidate matrix with the largest number of interior points is taken as the final geometric transformation matrix, and the corresponding set of interior point pairs is the set of interior point matching pairs.

[0072] In step S43, when the number of inner points in the inner point matching pair set is lower than the preset inner point number threshold, it is determined that the reliability of the current overlap calculation is insufficient, the preprocessed image pair is replaced with the original grayscale image pair, and the inter-frame overlap estimate and multi-frame average overlap are recalculated.

[0073] Furthermore, when the number of inliers in the inlier matching pair set is less than the preset inlier number threshold of 50, the present invention determines that the matching result of the current preprocessed image pair is not reliable enough, replaces the preprocessed image pair with the original grayscale image pair that has not been downsampled, and re-executes steps S41 to S44 to recalculate the inter-frame overlap estimate and the multi-frame average overlap with the complete resolution data of the original grayscale image pair.

[0074] S44. Calculate the ratio of the geometric overlap area of ​​the two frames to the area of ​​a single frame image based on the geometric transformation matrix to obtain the inter-frame overlap estimate, and take the average of the inter-frame overlap estimates of the most recent frames to obtain the multi-frame average overlap.

[0075] Furthermore, based on the geometric transformation matrix obtained in step S43, this invention substitutes the coordinates of the four corner points of the reference frame image into the matrix to perform coordinate transformation, obtaining the vertex coordinates of the projected quadrilateral of the reference frame image in the current frame coordinate system. Then, the area of ​​the intersection region between the projected quadrilateral and the boundary rectangle of the current frame image is calculated, and the area of ​​the intersection region is divided by the area of ​​a single frame image to obtain an estimated value of inter-frame overlap. Subsequently, this invention takes the estimated inter-frame overlap values ​​of the most recent five frames, sums them, and divides the sum by 5 to obtain the average overlap of multiple frames.

[0076] S5. The estimated inter-frame overlap and the average overlap of multiple frames are compared with the preset overlap threshold range to adjust the current acquisition frame rate and obtain an optimized frame rate.

[0077] Specifically, step S5 includes: When the estimated inter-frame overlap and the average overlap of multiple frames are both greater than the upper limit of the overlap threshold range, the current acquisition frame rate is multiplied by the frame reduction coefficient and the frame rate lower limit is taken as the maximum value to obtain the optimized frame rate after reducing the frame rate. When the estimated inter-frame overlap and the average overlap of multiple frames are both less than or equal to the lower limit of the overlap threshold range, the minimum value of the current acquisition frame rate multiplied by the frame increase coefficient and the frame rate upper limit is taken to obtain the optimized frame rate after the frame rate is increased. When the estimated inter-frame overlap is inconsistent with the trend of the average overlap of multiple frames, the current acquisition frame rate is kept unchanged and the current acquisition frame rate is used as the optimized frame rate.

[0078] After outputting the estimated inter-frame overlap and the average overlap of multiple frames in step S4, the present invention compares the two with the upper and lower limits of the overlap threshold range set in step S1, and selects the corresponding frame rate adjustment strategy based on the comparison results.

[0079] When both the estimated inter-frame overlap and the average overlap across multiple frames exceed the upper limit of the overlap threshold range of 75%, this invention multiplies the current acquisition frame rate by a frame reduction coefficient of 0.7 to obtain a candidate frame rate for frame reduction. Then, the maximum value between the candidate frame rate and the lower limit of the frame rate is taken to obtain the optimized frame rate after reduction. The lower limit of the frame rate serves as a constraint to prevent the frame rate from falling below the minimum acceptable acquisition frequency of the system.

[0080] When the estimated inter-frame overlap and the average overlap across multiple frames are both less than or equal to the lower limit of the overlap threshold range (55%), this invention multiplies the current acquisition frame rate by a frame-up factor of 1.3 to obtain a candidate frame rate for frame-up. Then, the minimum value between the candidate frame rate and the upper frame rate limit is taken to obtain the optimized frame rate after frame-up. The upper frame rate limit serves as a constraint to prevent the frame rate from exceeding the processing limits of the hardware and storage system.

[0081] When the estimated inter-frame overlap does not match the trend of the average overlap across multiple frames—that is, one is above the upper limit of the threshold range while the other is below the lower limit, or both are within the threshold range but change in opposite directions—this invention determines that the current frame rate adjustment conditions are not yet clear, maintains the current acquisition frame rate unchanged, and directly uses the current acquisition frame rate as the optimized frame rate output. This invention uses the obtained optimized frame rate to drive the image sensor to adjust the acquisition frequency, and re-enters the continuous optimization process in steps S2 to S5 with the updated frame rate.

[0082] like Figure 2 As shown, the present invention also provides a camera frame rate adaptive adjustment system based on image overlap, comprising: Initialization module 100: Used to preset parameters and obtain initialization configuration; Analysis module 200: used to receive image acquisition data based on the initialization configuration, and perform complexity analysis on the current frame image and the reference frame image to obtain image complexity index; Preprocessing module 300: Used to preprocess the current frame image and the reference frame image according to the image complexity index and the current computing load to obtain a preprocessed image pair; Matching module 400: used to input the preprocessed image pair into a feature matching algorithm based on the initial configuration to obtain an estimated value of inter-frame overlap and an average overlap of multiple frames; Adjustment module 500: is used to compare the estimated inter-frame overlap and the average overlap of multiple frames with the preset overlap threshold range, adjust the current acquisition frame rate, and obtain an optimized frame rate.

[0083] This invention provides a camera frame rate adaptive adjustment method and system based on image overlap. It uses image overlap as the basis for controlling camera frame rate adjustment, operating independently without relying on any external sensors, significantly reducing external dependence and facilitating integration. Secondly, compared to traditional fixed frame rate acquisition, this invention significantly reduces the amount of data collected by dynamically adjusting the camera acquisition frame rate (lowering the frame rate in high-overlap scenes and increasing the frame rate in low-overlap scenes), reducing storage requirements and facilitating subsequent data processing and management. Furthermore, users do not need to pre-calculate acquisition parameters and speed; they only need to focus on the required data overlap to automatically complete data acquisition, making operation simpler. In addition, this invention employs downsampling + frame-interval processing to meet the real-time processing needs of high-resolution images. Combined with the lightweight ORB+RANSAC algorithm, it enables rapid overlap calculation, improving computational efficiency.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A camera frame rate adaptive adjustment method based on image overlap, characterized in that, include: S1. Set preset parameters to obtain the initial configuration; S2. Start image acquisition according to the initialization configuration, and perform complexity analysis on the current frame image and the reference frame image to obtain the image complexity index. S3. Based on the image complexity index and the current computing load, preprocess the current frame image and the reference frame image to obtain a preprocessed image pair. S4. Input the preprocessed image pair into the feature matching algorithm based on the initial configuration to obtain the estimated value of inter-frame overlap and the average overlap of multiple frames. S5. The estimated inter-frame overlap and the average overlap of multiple frames are compared with the preset overlap threshold range to adjust the current acquisition frame rate and obtain an optimized frame rate.

2. The camera frame rate adaptive adjustment method based on image overlap as described in claim 1, characterized in that, Step S1 specifically includes: The image acquisition parameters are initialized, including an initial acquisition frame rate for determining the starting point of dynamic frame rate adjustment, and an upper and lower frame rate limit for defining the upper and lower boundaries of frame rate adjustment, thereby obtaining the image acquisition initialization configuration. The image processing parameters are initialized, including the upper and lower limits of the overlap threshold range for frame rate adjustment determination, the low and high entropy thresholds of image complexity for preprocessing strategy selection, the initial frame interval and calculation load threshold for inter-frame sampling, and the image processing initialization configuration is obtained. The algorithm configuration parameters are initialized, including the upper limit of the number of feature points for feature point detection, the image pyramid scaling ratio, the number of pyramid layers and the corner response threshold, as well as the inlier pixel error threshold and the iteration success probability for robust matching, to obtain the algorithm initialization configuration. Pre-allocate the logarithmic lookup table array for Shannon entropy calculation and the descriptor distance lookup table array for descriptor distance calculation to obtain the array configuration.

3. The camera frame rate adaptive adjustment method based on image overlap as described in claim 1, characterized in that, Step S2 further includes: S21. Start image acquisition according to the image acquisition initialization configuration in the initialization configuration to obtain the current frame image; S22. The current frame image is downsampled using bilinear interpolation to obtain a downsampled image; S23. The downsampled image is converted to grayscale using an integer approximate weighted summation method to obtain a grayscale downsampled image; S24. Statistically analyze the pixel grayscale probability distribution of the grayscale downsampled image, and calculate the Shannon entropy by using a logarithmic lookup table array to obtain the image complexity index represented by the image complexity entropy value.

4. The camera frame rate adaptive adjustment method based on image overlap as described in claim 3, characterized in that, Step S23 further includes: S231: Multiply the three channel pixel values ​​of the downsampled image by the corresponding integer approximate weight coefficients respectively and sum them to obtain the summation result; S232: Perform a right shift operation on the summation result to obtain the calculation result; S233: Normalize the calculation result to the grayscale value range to obtain a grayscale downsampled image.

5. The camera frame rate adaptive adjustment method based on image overlap as described in claim 4, characterized in that, Step S3 specifically includes: When the current computing load is less than or equal to the preset computing load threshold, the cached grayscale downsampled image and the corresponding reference frame grayscale downsampled image are reused as a preprocessed image pair. When the current computational load is greater than the preset computational load threshold and the average overlap of multiple frames is less than the lower limit of the overlap threshold range, the downsampled image is downsampled a second time at a high compression ratio to obtain a high compression ratio downsampled image. The high compression ratio downsampled image is used as the current frame in the preprocessed image pair to obtain the preprocessed image pair.

6. The camera frame rate adaptive adjustment method based on image overlap as described in claim 5, characterized in that, Step S3 also includes: S31. When the current computing load is greater than the preset computing load threshold, and the average overlap of multiple frames is less than or equal to the lower limit of the preset overlap threshold range, the frame sequence is sampled every other frame according to the current frame interval, the complete overlap is calculated for the sampled frames, and the non-sampled frames are extrapolated based on the overlap change trend to obtain the current frame overlap estimate. Step S31 further includes: S311. The overlap of the most recent complete calculation frame is called the most recent overlap. The overlap change rate is obtained by dividing the difference in overlap of the most recent multiple complete calculation frames by the corresponding time interval. S312. For non-sampled frames, the ratio of the number of frames from the nearest complete calculation frame to the current sampling frame rate is used as the time offset. The product of the nearest overlap, the overlap change rate, and the time offset is added to obtain the estimated overlap value of the current non-sampled frame. S313. Perform full overlap calculation every fixed number of frames, verify the actual overlap with the most recent estimated value using the relative error formula, and continuously optimize and adjust the current frame interval based on the verification result to obtain the updated frame interval.

7. The camera frame rate adaptive adjustment method based on image overlap as described in claim 1, characterized in that, Step S4 further includes: S41. Perform directional feature point detection on the current frame image and the reference frame image in the preprocessed image pair respectively to obtain the feature point set and the corresponding binary descriptor subset of each frame. S42. Using the descriptor distance lookup table array in the initialization configuration, the descriptor set is matched using the Hamming distance lookup table method to obtain an initial set of matching point pairs; S43. Input the initial set of matching point pairs into a robust estimation algorithm to filter and obtain the set of interior point matching point pairs and the corresponding geometric transformation matrix; S44. Calculate the ratio of the geometric overlap area of ​​the two frames to the area of ​​a single frame image based on the geometric transformation matrix to obtain the inter-frame overlap estimate, and take the average of the inter-frame overlap estimates of the most recent frames to obtain the multi-frame average overlap.

8. The camera frame rate adaptive adjustment method based on image overlap as described in claim 7, characterized in that, In step S43, when the number of interior points in the set of interior point matching pairs is lower than the preset threshold for the number of interior points, it is determined that the reliability of the current overlap calculation is insufficient. The preprocessed image pair is replaced with the original grayscale image pair, and the inter-frame overlap estimate and the average overlap of multiple frames are recalculated.

9. The camera frame rate adaptive adjustment method based on image overlap as described in claim 1, characterized in that, Step S5 specifically includes: When the estimated inter-frame overlap and the average overlap of multiple frames are both greater than the upper limit of the overlap threshold range, the current acquisition frame rate is multiplied by the frame reduction coefficient and the frame rate lower limit is taken as the maximum value to obtain the optimized frame rate after reducing the frame rate. When the estimated inter-frame overlap and the average overlap of multiple frames are both less than or equal to the lower limit of the overlap threshold range, the minimum value of the current acquisition frame rate multiplied by the frame increase coefficient and the frame rate upper limit is taken to obtain the optimized frame rate after the frame rate is increased. When the estimated inter-frame overlap is inconsistent with the trend of the average overlap of multiple frames, the current acquisition frame rate is kept unchanged and the current acquisition frame rate is used as the optimized frame rate.

10. A camera frame rate adaptive adjustment system based on image overlap, used to execute a camera frame rate adaptive adjustment method based on image overlap as described in any one of claims 1 to 9, characterized in that, include: Initialization module: used to preset parameters and obtain initialization configuration; Analysis module: used to receive image acquisition data based on the initialization configuration, and perform complexity analysis on the current frame image and the reference frame image to obtain image complexity index; Preprocessing module: used to preprocess the current frame image and the reference frame image according to the image complexity index and the current computing load to obtain a preprocessed image pair; Matching module: used to input the preprocessed image pairs into a feature matching algorithm based on the initial configuration to obtain the estimated inter-frame overlap and the average overlap of multiple frames; Adjustment module: used to compare the estimated inter-frame overlap and the average overlap of multiple frames with the preset overlap threshold range, adjust the current acquisition frame rate, and obtain an optimized frame rate.

Images