LK optical flow estimation method and system, electronic equipment and storage medium

Abstract

The invention provides an LK optical flow estimation method and system, electronic equipment and a storage medium, and relates to the field of image processing. The method comprises the following steps: acquiring original image data comprising a first image frame and a second image frame which are sequentially acquired in time; extracting a feature point set from the original image data through a corner detection algorithm; and extracting an illumination feature vector according to the original image data. According to a joint optimization objective function, taking the feature point set, the illumination feature vector and brightness data carried by the original image data as input, executing a cyclic process until a preset convergence condition is met, and outputting an optical flow vector during convergence; the circulation process comprises the following steps: solving an optical flow vector when a weight parameter is fixed; weight parameters are updated when the optical flow vector is fixed. And generating a final optical flow vector according to the optical flow vector during convergence and the illumination compensation model with the updated weight parameters. According to the scheme, the illumination change can be effectively adapted and compensated while the advantage of calculation efficiency is kept.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an LK optical flow estimation method, system, electronic device, and storage medium. Background Technology

[0002] Optical flow estimation is a core technology in computer vision. By analyzing pixel motion between consecutive frames, it provides crucial motion information for applications such as video analysis and autonomous driving. Ideally, the brightness of a pixel remains constant during motion. Based on this "brightness constancy" assumption, basic optical flow constraint equations can be established.

[0003] Currently, mainstream optical flow estimation methods can be divided into three categories: The first category is traditional methods, such as LK and Horn-Schunck algorithms. These are based on the assumptions of illumination invariance and smoothness, and solve by precisely setting the optimization function and using different optimization algorithms. They achieve high accuracy and efficiency in ideal scenarios. However, their core assumption of illumination invariance is easily violated under real-world illumination changes, leading to a significant performance degradation. The second category is deep learning-based methods. While they perform well in complex scenarios, they rely on large amounts of labeled data, are computationally expensive, and still have insufficient generalization ability and accuracy in simple scenarios. The third category is matching methods. Although they are computationally efficient, they are also sensitive to complex conditions such as illumination changes and occlusion, limiting their stability.

[0004] In summary, traditional methods cannot adapt to changes in illumination, deep learning methods are resource-intensive and have weak generalization ability, and matching rules lack robustness. Therefore, there is an urgent need for a novel optical flow estimation solution that can effectively adapt to and compensate for changes in illumination while maintaining the computational efficiency advantages of traditional methods. Summary of the Invention

[0005] The purpose of this application is to provide an LK optical flow estimation method, system, electronic device and storage medium that can effectively adapt to and compensate for changes in illumination while maintaining the computational efficiency advantages of traditional methods.

[0006] This application is implemented as follows: In a first aspect, this application provides an LK optical flow estimation method, comprising the following steps: acquiring original image data, the original image data including a first image frame and a second image frame acquired sequentially in time; extracting features from the original image data using a corner detection algorithm to obtain a set of feature points; extracting illumination features from the original image data to obtain an illumination feature vector; and performing a loop process based on a joint optimization objective function, taking the set of feature points, the illumination feature vector, and the brightness data carried by the original image data as input, until a preset convergence condition is met, and outputting the optical flow vector at convergence; wherein, the joint optimization objective function is an objective function that takes the optical flow vector and the weight parameters of the illumination compensation model as optimization variables; the illumination compensation model is a model that adjusts the pixel brightness of the second image frame based on the illumination feature vector through weighted combination and nonlinear activation; the loop process includes: solving for the optical flow vector while fixing the weight parameters; updating the weight parameters while fixing the optical flow vector; and generating the final optical flow vector based on the optical flow vector at convergence and the illumination compensation model with updated weight parameters.

[0007] Secondly, this application provides an LK optical flow estimation system, comprising: a data acquisition unit configured to acquire raw image data, the raw image data including a first image frame and a second image frame acquired sequentially in time; a feature extraction unit configured to extract features from the raw image data using a corner detection algorithm to obtain a set of feature points; a first determination unit configured to extract illumination features from the raw image data to obtain an illumination feature vector; and a second determination unit configured to, based on a joint optimization objective function, take the set of feature points, the illumination feature vector, and the brightness data carried by the raw image data as input, execute a loop process until a preset convergence condition is met, and output the optical flow vector at convergence; wherein, the joint optimization objective function is an objective function that takes the optical flow vector and the weight parameters of the illumination compensation model as optimization variables; the illumination compensation model is a model that adjusts the pixel brightness of the second image frame based on the illumination feature vector through weighted combination and nonlinear activation; the loop process includes: solving for the optical flow vector while fixing the weight parameters; and updating the weight parameters while fixing the optical flow vector. The third determining unit is configured to generate the final optical flow vector based on the optical flow vector at convergence and the illumination compensation model with updated weight parameters.

[0008] Thirdly, this application provides an electronic device including a memory for storing one or more programs; a processor; and, when the one or more programs are executed by the processor, implementing the method as described in any one of the first aspects above.

[0009] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects above.

[0010] Fifthly, this application provides a computer program product including computer program instructions that, when executed by a processor, implement the method as described in any one of the first aspects above.

[0011] Compared with the prior art, this application has at least the following advantages or beneficial effects: This application introduces an illumination compensation mechanism and a joint optimization strategy based on the traditional LK optical flow method. First, a feature point set is obtained through corner detection, and illumination feature vectors are extracted by combining this with the original image data. Then, based on the joint optimization objective function, the optical flow vector and the weight parameters of the illumination compensation model are used as optimization variables. This illumination compensation model, based on the illumination feature vector, adaptively adjusts the pixel brightness of the second image frame through weighted combination and nonlinear activation. Then, with the goal of minimizing the joint optimization objective function, an alternating iterative optimization process is adopted: solving for the optical flow vector with fixed weight parameters, updating the weight parameters while keeping the optical flow vector fixed, and iterating cyclically until the convergence condition is met, outputting the optical flow vector at convergence. Finally, the converged optical flow vector and the illumination compensation model with updated weight parameters are combined to generate an accurate final optical flow vector after illumination compensation. Thus, this application enhances the adaptability to illumination changes, ensuring accurate estimation of optical flow under different illumination conditions while maintaining the computational efficiency advantage of the traditional LK optical flow estimation method. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of an embodiment of an LK optical flow estimation method according to this application; Figure 2 This is a flowchart illustrating the steps for solving the optical flow vector with fixed weight parameters in one embodiment of this application. Figure 3 This is a flowchart illustrating the steps of updating weight parameters when the optical flow vector is fixed in one embodiment of this application. Figure 4 This is a structural block diagram of an embodiment of an LK optical flow estimation system according to this application; Figure 5This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0014] Icons: 201, Processor; 202, Memory; 203, Communication Interface. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0016] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0017] Application Overview In realizing the technical solution of this application, the inventors recognized that the fundamental reason why traditional LK estimation methods and their derived algorithms fail in scenarios with changing lighting is that they adhere to the ideal assumption of "constant brightness". When there are changes in light sources, shadow movement, or automatic exposure adjustment of the camera in practical applications, this assumption no longer holds, causing the optimization objective function based on constant brightness to fail to accurately describe the real motion relationship.

[0018] Furthermore, deep learning-based optical flow estimation methods, especially those utilizing convolutional neural networks, train models with large amounts of labeled data to learn the complex mapping relationship between input image pairs and their corresponding optical flow fields. This approach achieves an end-to-end estimation process that directly predicts the optical flow field from image pairs, without relying on manually designed features or independent optimization steps. In numerous benchmark tests, such methods have demonstrated superior performance, effectively handling complex scenes and large-scale motion.

[0019] However, deep learning-based methods also have several limitations. In simple environments, their optical flow estimation accuracy may be lower than that of fully optimized traditional methods. Furthermore, deep learning algorithms have high requirements for the quality and scale of training data, resulting in relatively weak scene generalization ability. From a computational perspective, most deep learning-based optical flow algorithms focus on dense optical flow calculations, leading to a proportional increase in computational cost with increasing image resolution. Existing classic optical flow networks typically have large models and numerous parameters, resulting in significant computational demands, and their performance is limited by device computing power, posing a clear constraint on practical deployment.

[0020] Furthermore, while matching techniques, which estimate optical flow by detecting and matching feature points (such as SIFT, SURF, ORB, etc.) or image patches, exhibit certain advantages in specific scenarios, such as high computational efficiency and good robustness to certain conditions, their performance is significantly limited when faced with complex scenarios involving changes in illumination, occlusion, large motion, and noise.

[0021] Based on this analysis, this application provides an LK optical flow estimation method that abandons the traditional single optimization approach and instead proposes an innovative joint optimization framework. Specifically, it includes a joint objective function that uses both the optical flow vector and the illumination compensation parameters (weight parameters of the illumination compensation model) as optimization variables, and adopts an alternating iterative optimization strategy—accurately solving for the optical flow when the illumination parameters are fixed, and updating the illumination model when the optical flow is fixed. Through this collaborative optimization approach, the algorithm can dynamically adapt to illumination changes, fundamentally breaking through the limitations of traditional methods. Thus, it can effectively adapt to and compensate for illumination changes while maintaining the computational efficiency advantages of traditional methods.

[0022] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings. Unless otherwise specified, the various embodiments and features described below can be combined with each other.

[0023] Exemplary methods Please see Figure 1 The LK optical flow estimation method includes the following steps: Step S101: Acquire raw image data, which includes a first image frame and a second image frame acquired sequentially in time. The time difference between the first image frame and the second image frame determines the range of motion that can be captured.

[0024] Step S102: Extract features from the original image data using a corner detection algorithm to obtain a set of feature points.

[0025] To reduce computational load from all image pixels to a small number of feature points, meeting the demands of real-time applications, step S102 employs a corner detection algorithm (such as Shi-Tomasi) to automatically filter out points in the original image data that exhibit significant gradient changes and unique textures (such as object corners, edge intersections, etc.). These feature points are unique within their local neighborhoods, facilitating tracking and matching in subsequent frames. Furthermore, the unique texture patterns of these feature points ensure stable matching even in noisy or slightly deformed conditions, improving the accuracy and reliability of subsequent optical flow calculations.

[0026] Step S103: Extract illumination features from the original image data to obtain an illumination feature vector. This transforms the abstract, global illumination variation into a specific mathematical vector to describe the illumination conditions of the image. For example, this illumination feature vector can be composed of global features (average brightness and contrast of the entire image) and local features (brightness and contrast of each area after dividing the image into blocks).

[0027] Step S104: Based on the joint optimization objective function, the feature point set, illumination feature vector, and brightness data carried by the original image data are used as inputs. A loop process is executed until the preset convergence condition is met, and the optical flow vector at the convergence point is output. The joint optimization objective function is an objective function that uses the optical flow vector and the weight parameters of the illumination compensation model as optimization variables. The illumination compensation model is a model that adjusts the pixel brightness of the second image frame based on the illumination feature vector through weighted combination and nonlinear activation. The loop process includes: solving for the optical flow vector while fixing the weight parameters; and updating the weight parameters while fixing the optical flow vector.

[0028] Step S105: Generate the final optical flow vector based on the optical flow vector at convergence and the illumination compensation model with updated weight parameters.

[0029] In step S104, the objective of optical flow calculation is redefined. It no longer pursues strict equality of pixel brightness between two frames, but allows for a brightness transformation relationship. That is, based on traditional estimation, a compensation term is introduced into the original objective function to construct a new joint optimization objective function. The optical flow vector and the weight parameters of the illumination compensation model are used as optimization variables, simultaneously seeking the optimal optical flow vector and the optimal illumination compensation value (achieved through the optimal weight parameters of the illumination compensation model).

[0030] In the execution loop, the following two steps are iterated alternately to converge to a globally better solution: (1) Solve for the optical flow vector while fixing the weight parameters. At this stage, the illumination compensation model is considered to be optimal. Under the assumption of consistent illumination conditions, the optical flow vector is accurately solved by utilizing the fast convergence of the traditional LK algorithm. It should be noted that the solution is performed on the image of the second image frame using the illumination compensation model to compensate for illumination, which can avoid illumination interference. (2) Update the weight parameters while fixing the optical flow vector. At this stage, the optical flow estimation is considered accurate. By aligning the first and second image frames using the optical flow vector, the difference between the current compensation effect and the expected effect (such as consistent brightness and similar structure) (i.e., the loss function) is calculated. Then, the weight parameters of the illumination compensation model can be updated in reverse by using the gradient descent algorithm to make the illumination compensation model better. It should be noted that the processing is performed on the image after "motion alignment", which can more clearly separate the illumination changes.

[0031] In other words, during the execution of step S104, the illumination compensation model is actively used for illumination compensation based on the joint optimization objective function, rather than passively relying on its invariance, thus breaking through the theoretical bottleneck of the traditional LK algorithm. Simultaneously, the optical flow vector and the weight parameters of the illumination compensation model provide each other with more accurate inputs during iteration, forming a positive feedback loop that converges to a globally better solution. Unlike deep learning-based methods, it does not require a large amount of labeled data or expensive computational resources, effectively ensuring computational efficiency. Furthermore, the weight parameters of the illumination compensation model are learned during iteration, eliminating the need for manual setting and allowing it to adapt to various unknown and dynamic lighting scenarios.

[0032] When the loop process in step S104 meets the convergence condition (such as minimal changes in the optical flow vector and the weight parameters of the illumination compensation model), it indicates that the current optimal optical flow vector has been found. Then, using the illumination compensation model updated with the optimal weight parameters, the optical flow vector obtained at this point is illumination compensated, thus obtaining an optical flow vector that best explains both motion and illumination under the current conditions. This final optical flow vector is estimated after fully considering the influence of illumination changes, and its accuracy and robustness are far superior to traditional methods, providing a reliable basis for optical flow estimation for subsequent visual tasks (such as tracking and navigation).

[0033] In summary, this application, based on a joint optimization objective function, places the optical flow vector and the weight parameters of the illumination compensation model within a unified optimization framework. Through an alternating iterative optimization strategy—accurately solving for the optical flow while fixing the weight parameters of the illumination compensation model, and updating the weight parameters while fixing the optical flow—the two optimization variables mutually promote and converge collaboratively. This design enables the illumination compensation model to be based on accurate motion estimation, which in turn can be performed under corrected illumination conditions, forming a virtuous cycle of optimization. This simultaneously ensures the accuracy of motion estimation and the rationality of illumination compensation.

[0034] Furthermore, unlike the "black box" nature of deep learning methods, this application improves upon the traditional optimization framework. The weight parameters of the illumination compensation model are automatically learned from the data through an optimization algorithm, eliminating the need for manual setting or reliance on large amounts of labeled data. This design retains the advantages of high computational efficiency and strong interpretability of traditional methods while achieving adaptive capabilities similar to deep learning. The algorithm can automatically adjust the compensation strategy according to actual illumination conditions, effectively adapting to different types and degrees of illumination changes.

[0035] Based on the aforementioned scheme, in some implementations of this application, the step of extracting features from the original image data using a corner detection algorithm includes: converting the original image data into single-channel grayscale data and filtering it to obtain a preprocessed image; and extracting features from the preprocessed image using a corner detection algorithm.

[0036] In the above implementation, a preprocessing step is added to the step of extracting features from the preprocessed image using a corner detection algorithm. Specifically, this includes: first, converting the original image data into single-channel grayscale data; then, filtering the grayscale image to obtain a preprocessed image; and finally, performing the corner detection algorithm based on the preprocessed image to complete feature extraction. This layered processing architecture unifies the data format through grayscale conversion and effectively suppresses image noise through filtering, creating ideal conditions for subsequent feature extraction.

[0037] In other words, by eliminating color information interference through grayscale conversion, the corner detection algorithm focuses on the texture and structural features of the image, significantly improving the representativeness and stability of feature points. Simultaneously, its filtering process effectively suppresses noise introduced during image acquisition, reducing the probability of erroneous feature point detection. Furthermore, the single-channel grayscale image reduces the data volume by approximately two-thirds compared to the original RGB three-channel image, significantly reducing the computational complexity of the corner detection algorithm. The improved image quality after preprocessing also accelerates the convergence speed of subsequent feature extraction processes, improving the overall real-time performance of optical flow estimation.

[0038] Based on the aforementioned scheme, in some implementations of this application, the step of extracting illumination features from the original image data to obtain illumination feature vectors includes: extracting global brightness features, global contrast features, and local region features from the original image data, and then concatenating them into illumination feature vector data.

[0039] In the above implementation, three levels of illumination features are first extracted from the original image data: global brightness features reflect the overall brightness distribution of the image, global contrast features describe the overall texture intensity of the image, and local region features capture local illumination changes centered on feature points. Then, these three levels of features are concatenated and fused to form comprehensive illumination feature vector data, providing a comprehensive and accurate feature description for subsequent illumination compensation.

[0040] It's important to note that by combining global and local features, the overall illumination trend of the image is captured while preserving illumination variation information in detailed areas, overcoming the limitations of single-scale feature description. This multi-level feature structure can accurately reflect complex changes in the lighting environment. Furthermore, global features ensure the overall consistency of illumination compensation, preventing local overcompensation or undercompensation; local features guarantee the accuracy of illumination adjustment in key areas around feature points. The combination of these two approaches allows the illumination compensation model to simultaneously consider both overall effect and local detail.

[0041] Based on the aforementioned scheme, in some implementations of this application, the step of extracting illumination features from the original image data to obtain an illumination feature vector includes: extracting at least one set of global illumination features and... Group of local illumination characteristics, The number of local regions; wherein, the global illumination features include at least features characterizing the global average brightness and global contrast of the original image data; the local illumination features are used to divide the original image data into After identifying several non-overlapping local regions, at least one set of features characterizing the local average brightness and local contrast of each region is extracted from the original image data. The local illumination features are spliced ​​together to obtain the illumination feature vector data.

[0042] In the above implementation, the illumination feature vector mainly considers the brightness difference features and contrast deviation features of the original image data. The illumination feature vector includes a set of global illumination features and... Group of local illumination characteristics, This represents the number of local regions. Each feature group includes two dimensions: "average brightness" and "contrast," for a total of [number of dimensions]. .

[0043] (1) Global illumination characteristics Extracted from the original image data, reflecting the overall illumination trend.

[0044] 1) Global average brightness The calculation formula: ; in, Image height, Image width, For the number of channels, For pixels From the first image frame to the second image frame, in channel... The brightness difference is calculated by considering that the RGB image has 3 channels and the grayscale image has 1 channel.

[0045] 2) Global contrast The calculation formula:

[0046] in, For the first The global average brightness of the channel is used to quantize the contrast using the standard deviation.

[0047] (2) Local features ( (dimensions) The original image data is evenly divided into... After dividing into non-overlapping local regions (e.g., 2×2 partitioning), The 3×3 partition yields Two features are extracted from each region: 1) Local region division method Let the size of the region be ,in, , Take the integer part.

[0048] Then, the first Local area The coordinate range is:

[0049] in, For the row index of the region, For column indexes of the region.

[0050] Then, the first Local area The characteristic calculation formulas include: Formula for calculating local average brightness: ; Formula for calculating local contrast: ; in, For the first The local region Average brightness of the channel.

[0051] Finally, the global illumination features and local illumination features are concatenated in sequence to obtain the illumination feature vector: .

[0052] Based on the aforementioned scheme, in some implementations of this application, the joint optimization objective function The expression is: ; in, A window centered on the current pixel. The brightness value of the first image frame. The brightness value of the second image frame. The optical flow vector represents the pixel. The displacement from the first image frame to the second image frame. These are the weight parameters for the illumination compensation model. For the activation function of the neural network, This is the illumination feature vector input to the illumination compensation model. This is the fitting structure for the illumination compensation model.

[0053] In the above implementation, the constructed joint optimization objective function introduces an illumination compensation model on top of the traditional constant brightness term. This function uses the optical flow vector and the weight parameters of the illumination compensation model as optimization variables, breaking the limitations of the traditional assumption of constant brightness. On the one hand, by introducing a learnable illumination compensation term, the algorithm can dynamically adjust the brightness distribution of the second frame image, effectively counteracting the interference caused by illumination changes, fundamentally solving the failure problem of traditional methods in scenarios with changing illumination. On the other hand, by placing the optical flow vector and the weight parameters of the illumination compensation model under a unified framework for synchronous optimization, the two parameters promote each other and converge collaboratively, ensuring both the accuracy of motion estimation and the rationality of illumination compensation, forming a virtuous cycle of optimization.

[0054] It should be noted that, to simplify the calculation, a Taylor expansion can be used to... Approximately .in, and They are exist and gradient in direction, It is about time The derivative of .

[0055] Thus, the joint optimization objective function can be simplified to:

[0056] To avoid gradient explosion during neural network parameter updates, this application selects the Tanh function as the core activation function, with the specific expression as follows: .

[0057] It should be noted that the final optical flow vector is generated based on the optical flow vector at convergence and the illumination compensation model with updated weight parameters. When the time comes, its calculation formula can be expressed as:

[0058] in, for A 3D illumination feature vector from the current image With historical original frames Extract from the differences (which may include global brightness mean, local contrast, gradient magnitude, etc.). for The dimensional weight parameter vector is the core parameter for optimizing the illumination compensation model.

[0059] Based on the aforementioned scheme, in some implementations of this application, the step of solving the optical flow vector while fixing the weight parameters includes: fixing the weight parameters of the illumination compensation model in the joint optimization objective function, calculating the corresponding gradient vector and Hessian matrix; iteratively updating the corresponding optical flow vector according to the gradient vector and Hessian matrix until a preset condition is met, wherein the preset condition is that the change between the current optical flow vector and the updated optical flow vector is less than a preset convergence threshold.

[0060] The above implementation provides a solution for optical flow vectors based on second-order optimization. Specifically, under the premise of fixing the weight parameters of the illumination compensation model, it first calculates the gradient vector and Hessian matrix corresponding to the joint optimization objective function. The gradient vector indicates the direction of the fastest descent of the objective function, and the Hessian matrix provides curvature information of the objective function, enabling more accurate estimation of the optimal step size and direction, thus accelerating the convergence speed. Subsequently, the optical flow vector is iteratively updated using this second-order information. The update step size and direction are determined by the product of the inverse of the Hessian matrix and the gradient vector. The iterative process continues until the change between the current optical flow vector and the updated optical flow vector is less than a preset convergence threshold, ensuring the quality and stability of the solution.

[0061] Specifically, such as Figure 2 As shown, to minimize the joint optimization objective function, the steps for solving the optical flow vector while fixing the weight parameters can be as follows: (1) Construct the Jacobian matrix sum vector : ,

[0062] in, It is a window The pixel coordinates within the area are fixed, and the illumination compensation is adjusted accordingly. Solving for the optical flow vector under certain conditions.

[0063] (2) Calculate the corresponding gradient : ; (3) Calculate the Hessian matrix : ; (4) Iterative update of optical flow vector: ;footmark and Indicates the time step of the iteration.

[0064] (5) Check convergence if If the condition is met, stop the iteration and output the corresponding optical flow vector, where This is the preset convergence threshold.

[0065] Based on the aforementioned scheme, in some implementations of this application, the step of updating the weight parameters when fixing the optical flow vector includes: fixing the optical flow vector in the joint optimization objective function, calculating the gradient of each weight parameter of the illumination compensation model based on the bi-objective loss function, wherein the bi-objective loss function includes illumination difference loss and motion information preservation loss; and updating each weight parameter using a gradient descent algorithm until each weight parameter converges or reaches a preset number of iterations.

[0066] In the above implementation, under the condition of a fixed optical flow vector, a dual-objective loss function is first constructed, which includes illumination difference loss and motion information preservation loss. Illumination difference loss is responsible for quantifying the brightness difference between two frames of images, ensuring the effectiveness of illumination compensation; motion information preservation loss, through structural similarity measurement, prevents the compensation process from destroying the motion features of the image. Based on this composite loss function, the gradient of each weight parameter is calculated, and gradient descent algorithm is used for iterative updates until the parameters converge or the preset number of iterations is reached, ensuring that the illumination compensation model reaches its optimal state.

[0067] It should be noted that in the above implementation, the design of the dual-objective loss function establishes a balance between eliminating illumination differences and preserving motion information. It effectively compensates for illumination changes while avoiding image structure damage caused by overcompensation, thus effectively ensuring the accuracy of optical flow estimation.

[0068] Specifically, such as Figure 3 As shown, to minimize the joint optimization objective function, the steps for updating the weight parameters while fixing the optical flow vector can be as follows: First, the vector representation of the weight parameters of the illumination compensation model can be expressed in the following form:

[0069] in, and These correspond to the weights of the global average brightness and global contrast features, respectively. and Corresponding to the first The weights of local average brightness and local contrast characteristics in each local region.

[0070] (2) Design of the dual-objective loss function The dual-objective loss function must simultaneously constrain "elimination of illumination differences" and "prevention of information loss":

[0071] in, , For the corresponding weights.

[0072] The formula for calculating the loss due to illumination difference is:

[0073] The loss for preserving motion information is calculated as follows:

[0074] in, Indicates in the channel The compensation correction below , Indicates in the channel Below . It is an abbreviation for Structural Similarity Index, a metric used to measure the structural similarity between two images. It calculates a value between 0 and 1 by comparing the similarity of two images in three dimensions: brightness, contrast, and structure.

[0075] When SSIM = 1, it means that the two images are exactly the same; When SSIM is close to 0, it indicates that the two images are extremely different.

[0076] The calculation details of SSMI can be simplified as follows: .

[0077] in, and These are images and The local mean, and They are and Local standard deviation, It is the local covariance of two images (a measure of structural similarity). and It is a constant that avoids a denominator of 0.

[0078] (2) Gradient calculation Calculate the gradient for each weight parameter: 1) Global feature weight gradient:

[0079]

[0080]

[0081] for right The partial derivative of is expressed as:

[0082] in, For the first The global average brightness characteristics of the channel. 2) Local feature weight gradient:

[0083]

[0084]

[0085] in, For the first Local area Local average brightness characteristics of the channel.

[0086] 3) Parameter update Parameter updates are performed using a gradient algorithm: ;in, and They represent the first Each weight parameter at time step and The value, Learn the step size for neural network parameters.

[0087] Exemplary System Please see Figure 4 This application provides an LK optical flow estimation system, comprising: a data acquisition unit configured to acquire raw image data, the raw image data including a first image frame and a second image frame acquired sequentially in time; a feature extraction unit configured to extract features from the raw image data using a corner detection algorithm to obtain a set of feature points; a first determination unit configured to extract illumination features from the raw image data to obtain an illumination feature vector; and a second determination unit configured to, based on a joint optimization objective function, take the set of feature points, the illumination feature vector, and the brightness data carried by the raw image data as input, execute a loop process until a preset convergence condition is met, and output the optical flow vector at convergence; wherein, the joint optimization objective function is an objective function that takes the optical flow vector and the weight parameters of the illumination compensation model as optimization variables; the illumination compensation model is a model that adjusts the pixel brightness of the second image frame based on the illumination feature vector through weighted combination and nonlinear activation; the loop process includes: solving for the optical flow vector while fixing the weight parameters; and updating the weight parameters while fixing the optical flow vector. The third determining unit is configured to generate the final optical flow vector based on the optical flow vector at convergence and the illumination compensation model with updated weight parameters.

[0088] For the specific implementation process of the above system, please refer to the LK optical flow estimation method provided in the "Exemplary Methods" section, which will not be repeated here.

[0089] Based on the aforementioned scheme, in some implementations of this application, the step of extracting features from the original image data using a corner detection algorithm includes: converting the original image data into single-channel grayscale data and filtering it to obtain a preprocessed image; and extracting features from the preprocessed image using a corner detection algorithm.

[0090] Based on the aforementioned scheme, in some implementations of this application, the step of extracting illumination features from the original image data to obtain illumination feature vectors includes: extracting global brightness features, global contrast features, and local region features from the original image data, and then concatenating them into illumination feature vector data.

[0091] Based on the aforementioned scheme, in some implementations of this application, the step of extracting illumination features from the original image data to obtain an illumination feature vector includes: extracting at least one set of global illumination features and... Group of local illumination characteristics, The number of local regions; wherein, the global illumination features include at least features characterizing the global average brightness and global contrast of the original image data; the local illumination features are used to divide the original image data into After identifying several non-overlapping local regions, at least one set of features characterizing the local average brightness and local contrast of each region is extracted from the original image data. The local illumination features are spliced ​​together to obtain the illumination feature vector data.

[0092] Based on the aforementioned scheme, in some implementations of this application, the joint optimization objective function The expression is: ; in, A window centered on the current pixel. The brightness value of the first image frame. The brightness value of the second image frame. The optical flow vector represents the pixel. The displacement from the first image frame to the second image frame. These are the weight parameters for the illumination compensation model. For the activation function of the neural network, This is the illumination feature vector input to the illumination compensation model. This is the fitting structure for the illumination compensation model.

[0093] Based on the aforementioned scheme, in some implementations of this application, the step of solving the optical flow vector while fixing the weight parameters includes: fixing the weight parameters of the illumination compensation model in the joint optimization objective function, calculating the corresponding gradient vector and Hessian matrix; iteratively updating the corresponding optical flow vector according to the gradient vector and Hessian matrix until a preset condition is met, wherein the preset condition is that the change between the current optical flow vector and the updated optical flow vector is less than a preset convergence threshold.

[0094] Based on the aforementioned scheme, in some implementations of this application, the step of updating the weight parameters when fixing the optical flow vector includes: fixing the optical flow vector in the joint optimization objective function, calculating the gradient of each weight parameter of the illumination compensation model based on the bi-objective loss function, wherein the bi-objective loss function includes illumination difference loss and motion information preservation loss; and updating each weight parameter using a gradient descent algorithm until each weight parameter converges or reaches a preset number of iterations.

[0095] Exemplary electronic devices Please see Figure 5This application provides an electronic device including at least one processor 201 and at least one memory 202. The processor 201 and the memory 202 are directly connected to each other, or communicate with each other through a communication interface 203, or are electrically connected through one or more communication buses or signal lines to achieve data transmission or interaction. The memory 202 stores program instructions that can be executed by the processor 201. The processor 201 can call the program instructions to execute the program instructions to implement an LK optical flow estimation method according to various embodiments of this application as described in the "Exemplary Methods" section above.

[0096] The memory 202 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0097] The processor 201 can be an integrated circuit chip with signal processing capabilities. The processor 201 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0098] Understandable. Figure 5 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 5 The more or fewer components shown, or having the same Figure 5 The different configurations shown. Figure 5 The components shown can be implemented using hardware, software, or a combination thereof.

[0099] Exemplary computer-readable storage media and computer program products This application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor 201, implements an LK optical flow estimation method according to various embodiments of this application as described in the "Exemplary Methods" section above.

[0100] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0101] Furthermore, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, implement the steps of an LK optical flow estimation method according to various embodiments of this application as described in the "Exemplary Methods" section above.

[0102] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0103] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. An LK optical flow estimation method, characterized in that, Includes the following steps: Acquire raw image data, which includes a first image frame and a second image frame acquired sequentially in time. Feature points are obtained by extracting features from the original image data using a corner detection algorithm. Illumination features are extracted from the original image data to obtain an illumination feature vector; According to the joint optimization objective function, the feature point set, illumination feature vector, and brightness data carried by the original image data are taken as inputs, and a loop process is executed until a preset convergence condition is met, outputting the optical flow vector at convergence. The joint optimization objective function is an objective function that uses the optical flow vector and the weight parameters of the illumination compensation model as optimization variables. The illumination compensation model is a model that adjusts the pixel brightness of the second image frame based on the illumination feature vector through weighted combination and nonlinear activation. The loop process includes: solving for the optical flow vector while fixing the weight parameters; and updating the weight parameters while fixing the optical flow vector. The final optical flow vector is generated based on the optical flow vector at convergence and the illumination compensation model with updated weight parameters.

2. The method according to claim 1, characterized in that, The step of extracting features from the original image data using the corner detection algorithm includes: The original image data is converted into single-channel grayscale data and then filtered to obtain a preprocessed image; Feature extraction is performed on the preprocessed image using a corner detection algorithm.

3. The method according to claim 1, characterized in that, The steps for extracting illumination features from the original image data to obtain illumination feature vectors include: Global brightness features, global contrast features, and local region features are extracted from the original image data and then concatenated to form illumination feature vector data.

4. The method according to claim 1 or 3, characterized in that, The steps for extracting illumination features from the original image data to obtain illumination feature vectors include: Extract at least one set of global illumination features from the original image data and Group of local illumination characteristics, The number of local regions; wherein, the global illumination features include at least features characterizing the global average brightness and global contrast of the original image data; the local illumination features are used to divide the original image data into After identifying several non-overlapping local regions, each local region is extracted with features that characterize the local average brightness and local contrast of that region. At least one set of global illumination features extracted from the original image data and The local illumination features are spliced ​​together to obtain the illumination feature vector data.

5. The method according to claim 1, characterized in that, The joint optimization objective function The expression is: ； in, A window centered on the current pixel. The brightness value of the first image frame. The brightness value of the second image frame. The optical flow vector represents the pixel. The displacement from the first image frame to the second image frame. These are the weight parameters for the illumination compensation model. For the activation function of the neural network, This is the illumination feature vector input to the illumination compensation model. This is the fitting structure for the illumination compensation model.

6. The method according to claim 1, characterized in that, The steps for solving the optical flow vector while fixing the weight parameters include: By fixing the weight parameters of the illumination compensation model in the joint optimization objective function, the corresponding gradient vector and Hessian matrix are calculated. Based on the gradient vector and Hessian matrix, the corresponding optical flow vector is iteratively updated until a preset condition is met. The preset condition is that the change between the current optical flow vector and the updated optical flow vector is less than a preset convergence threshold.

7. The method according to claim 1, characterized in that, The step of updating the weight parameters while fixing the optical flow vector includes: The optical flow vector in the joint optimization objective function is fixed, and the gradient of each weight parameter of the illumination compensation model is calculated based on the bi-objective loss function, wherein the bi-objective loss function includes illumination difference loss and motion information preservation loss; The gradient descent algorithm is used to update each weight parameter until each weight parameter converges or reaches the preset number of iterations.

8. An LK optical flow estimation system, characterized in that, include: The data acquisition unit is configured to acquire raw image data, which includes a first image frame and a second image frame acquired sequentially in time. The feature extraction unit is configured to extract features from the original image data using a corner detection algorithm to obtain a set of feature points. The first determining unit is configured to: extract illumination features based on the original image data to obtain an illumination feature vector; The second determining unit is configured to: take the feature point set, the illumination feature vector, and the brightness data carried by the original image data as inputs according to the joint optimization objective function, execute a loop process until a preset convergence condition is met, and output the optical flow vector at the convergence point; wherein, the joint optimization objective function is an objective function that takes the optical flow vector and the weight parameters of the illumination compensation model as optimization variables; the illumination compensation model is a model that adjusts the pixel brightness of the second image frame based on the illumination feature vector through weighted combination and nonlinear activation; the loop process includes: solving for the optical flow vector while fixing the weight parameters; and updating the weight parameters while fixing the optical flow vector. The third determining unit is configured to generate the final optical flow vector based on the optical flow vector at convergence and the illumination compensation model with updated weight parameters.

9. An electronic device, characterized in that, include: Memory, used to store one or more programs; processor; When the one or more programs are executed by the processor, the method as described in any one of claims 1-7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images