Display optical flow estimation method based on semantic segmentation

Abstract

The invention relates to the technical field of optical flow estimation, in particular to a display optical flow estimation method based on semantic segmentation. The method comprises the following specific steps: establishing an optical flow estimation model on the basis of an RAFT model; pre-training an image feature extraction part in the optical flow estimation model network by using an SAM image segmentation model, processing and enhancing image features in the optical flow estimation model network through pre-training, and obtaining processed and enhanced image features; and establishing a loss function to carry out granularity fine tuning on the processed and enhanced image features. According to the method, boundary regions of different types of targets in the input image are distinguished; according to the method provided by the invention, the pixels of the same type of targets have the maximum matching degree, the boundary of the network target is explicitly informed, and the problem of target adhesion is effectively avoided; the problem that when an original SAM image segmentation model is directly used for an optical flow estimation model, different numbers of targets exist due to different segmentation granularities is effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of optical flow estimation, and in particular to a display optical flow estimation method based on semantic segmentation. Background Art

[0002] Optical flow refers to the pixel motion change vector generated by the motion of objects in a scene between consecutive image frames. It is expressed as the velocity vector of all pixels in two adjacent image frames. Intuitively, optical flow describes the motion trajectory of each pixel in an image along the time axis. Optical flow not only reflects the direction and rate of object motion but also reveals, to a certain extent, the depth and structure of the scene. It is one of the most fundamental and widely used motion description methods in computer vision.

[0003] Traditional optical flow estimation algorithms use hand-crafted image features and constraint equations to calculate optical flow. The Lucas-Kanade method assumes that the optical flow of pixels within a small window (local area) is consistent, meaning that all pixels have the same direction and speed of motion. By establishing constraint equations for all pixels within the window and solving the optical flow using the least squares method, it is suitable for small-amplitude motion, requires little computation, and is fast. However, due to the assumption of consistent motion within the window, it is less effective for larger displacements or complex motions (such as rotations and deformations). The Horn-Schunck method uses a global optimization approach. While satisfying the optical flow constraint equations, it also adds a smoothing term to minimize the variation in optical flow between adjacent pixels. This means that the optical flow between adjacent pixels on the image does not change too dramatically, thereby obtaining a globally consistent optical flow field. The optical flow is solved through energy minimization (based on a variational method). However, the drawback is that it is computationally intensive, requiring the solution of partial differential equations, and can produce large errors in situations where the smoothness assumption is not met (such as motion boundaries and occluded areas).

[0004] In recent years, with the development of deep learning in computer vision, deep learning-based optical flow estimation methods have made significant progress. The core idea is to use deep learning models to extract image features from the input image and then predict pixel motion information through pixel matching. The backbone networks of early models used a stacked CNN block for feature extraction, and then directly extracted the image's optical flow information through deconvolution at deeper layers. FlowNet is a prime example of this type of network. While the accuracy of this type of network is relatively poor, its main contribution is the initial demonstration that end-to-end optical flow computation can be performed using deep neural networks. Subsequently, the RAFT model proposed by Teed et al. achieved extremely high accuracy on multiple datasets, breaking new state-of-the-art results and becoming a new generation of backbone network architecture. The core module of the RAFT model is the 4D cost volume matrix (4DCorrelation volume), a 4D matrix with the shape H*W*H*W, where H and W represent the shapes of the first and second frames, respectively. The value of each element represents the correlation between a pixel in the first frame and the corresponding pixel in the second frame. A higher correlation indicates a closer match between the two pixels. It can be seen that the higher the accuracy of the correlation of the four-dimensional cost volume matrix, the better the optical flow estimation effect.

[0005] In recent years, much research has been devoted to improving the accuracy of the cost volume matrix, such as using more powerful image feature extraction modules or using richer datasets for training. However, the drawback of these methods is that they only consider the optical flow estimation problem from a global perspective, hoping to use general feature extraction methods to estimate the optical flow estimation problem, without considering the semantic information of the image, which is more meaningful for optical flow estimation. Relying solely on the semantic information learned during model pre-training is completely insufficient. This is because most optical flow datasets are synthesized by image and video production software. Their background and foreground objects are generally simple, which is quite different from real-world data. The objects in the image are also relatively simple and repetitive, making it impossible to directly learn rich semantic information. Summary of the Invention

[0006] The purpose of the embodiments of the present application is to provide a display optical flow estimation method based on semantic segmentation to improve the above technical problems. To achieve the above purpose, the present application provides the following technical solutions:

[0007] The display optical flow estimation method based on semantic segmentation has the following specific steps:

[0008] S1. Establish an optical flow estimation model based on the RAFT model;

[0009] S2. Pre-training the image feature extraction portion of the optical flow estimation model network using a SAM (Segment Anything Model) image segmentation model, processing and enhancing the image features in the optical flow estimation model network through the pre-training, and obtaining processed and enhanced image features; during the pre-training of the image feature extraction portion, the back-layer parameters of the SAM image segmentation model are updated as the network is updated;

[0010] S3. Establish a loss function (Context Encoder) to fine-tune the granularity of the processed and enhanced image features.

[0011] The SAM image segmentation model includes a SAM encoder, a semantic segmentation module, and a semantic matching module. The processing and enhancement method of the image features in the optical flow estimation model using the SAM image segmentation model includes:

[0012] A1. Encode two input images into implicit tokens using the SAM encoder; use the SAM image segmentation model to pre-train the image feature extraction part of the network;

[0013] A2. In the semantic segmentation module, an improved segmentation decoder is used to convert the implicit token into the segmentation results of the two images. The semantic segmentation module uses an improved segmentation decoder that can directly perform self-attention decoding through the SAM encoder and directly input a set of global segmentation prediction masks after upsampling, thereby outputting the segmentation result of a complete image. The improved segmentation decoder is subjected to a recurrent self-attention decoding operation;

[0014] A3. In the semantic matching module, pixel-level matching is performed on the predicted mask corresponding to the segmentation result. In the semantic matching module, a class threshold is first set, and pixels in the first frame whose pixel matching degree exceeds the class threshold are regarded as pixels that can be semantically matched. A four-dimensional cost matching matrix is ​​constructed for these pixels that can be semantically matched. Using the four-dimensional cost matching matrix, the pixels that can be semantically matched are computationally connected with the image features in the input image, and the computational connection results are input into the GRU.

[0015] The loss function includes a semantic classification loss function and a segmentation granularity loss function. In the semantic classification loss function, a four-dimensional similarity matrix is ​​constructed based on the true optical flow values ​​in the dataset. In the semantic classification loss function, after constructing the four-dimensional similarity matrix, a binary cross entropy loss function is used to predict the similarity of pixels between the four-dimensional similarity matrix and the true value matrix to determine the size of the semantic classification loss function; wherein, the higher the similarity, the smaller the loss function.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The present invention performs explicit semantic segmentation on the input image based on the existing optical flow estimation model and the SAM image segmentation model. First, in the semantic segmentation module, an improved segmentation decoder is used to distinguish the boundary areas of targets of different categories in the input image; in the semantic matching module, pixel-level matching is performed on the prediction mask corresponding to the segmentation result, and the segmented targets in the two frames of the image are matched so that the pixels of the same type of target have the maximum matching degree, explicitly informing the network of the target boundary, thereby effectively avoiding the target adhesion problem. At the same time, the present invention introduces two different loss functions, which effectively solves the problem of different target numbers due to different segmentation granularity when the original SAM image segmentation model is directly used in the optical flow estimation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which constitute part of the present invention, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0019] Figure 1 Flowchart of the displayed optical flow estimation method based on semantic segmentation;

[0020] Figure 2 Flowchart for processing and enhancing image features in the optical flow estimation model using the SAM image segmentation model;

[0021] Figure 3 This is a module diagram of the optical flow estimation method based on explicit semantics;

[0022] Figure 4 After segmentation using the SAM image segmentation model, segmentation effect images of different granularities are obtained. DETAILED DESCRIPTION

[0023] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other.

[0024] The following detailed description is an exemplary description and is intended to provide further detailed description of the present invention. Unless otherwise indicated, all technical terms used in the present invention have the same meaning as those generally understood by those skilled in the art to which the present invention belongs. The terms used in the present invention are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention.

[0025] It is understood from common technical knowledge that the present invention may be implemented by other embodiments that do not depart from its spirit or essential features. Therefore, the embodiments disclosed above are, in all respects, merely illustrative and not exclusive. All modifications within the scope of the present invention or equivalent to the scope of the present invention are intended to be encompassed by the present invention.

[0026] The present invention provides a method for estimating display optical flow based on semantic segmentation, such as Figure 1-2 As shown in the figure, it effectively solves the problem of insufficient utilization of semantic information and single dataset in the existing optical flow estimation model. The specific steps are:

[0027] S1. Establish an optical flow estimation model based on the RAFT model;

[0028] S2. Pre-training the image feature extraction part of the optical flow estimation model network using the SAM image segmentation model, processing and enhancing the image features in the optical flow estimation model network through the pre-training, and obtaining the processed and enhanced image features; when pre-training the image feature extraction part, the back-layer parameters of the SAM image segmentation model are updated as the network is updated;

[0029] S3. Establish a loss function to fine-tune the granularity of the processed and enhanced image features.

[0030] Among them, this model uses the SAM image segmentation model to improve the RAFT model, and the added modules are Figure 3 The pink module is used to represent the SAM image segmentation model, which includes a SAM encoder, a semantic segmentation module, and a semantic matching module. The processing and enhancement method of the SAM image segmentation model for processing and enhancing the image features in the optical flow estimation model includes:

[0031] A1. Encode the two input images into implicit tokens through the SAM encoder;

[0032] The SAM image segmentation model is a general, pre-trained model that has demonstrated strong performance in image segmentation tasks across a variety of scenarios. It was trained on the SA-1B dataset, which contains over one billion self-annotated masks and 11 million images. This large-scale training enables strong zero-shot generalization, enabling it to adapt to unseen domains.

[0033] The present invention uses the SAM image segmentation model to pre-train the network for feature extraction. Instead of freezing the network parameters in the SAM image segmentation model, the parameters of subsequent layers are updated as the network is updated during image feature extraction. This is because research on the SAM image segmentation model has shown that even for images with similar adjacent frames, segmentation using the SAM image segmentation model yields segmentation results of varying granularity.

[0034] like Figure 4 As shown in the figure, in the first frame, the SAM image segmentation model segments the person's right hand into an independent individual. However, in the next frame, due to unclear semantics, the SAM image segmentation model is more inclined to regard the right hand as part of the whole body, which is more common in complex scenes.

[0035] In the case of uneven segmentation granularity, one-to-many, many-to-one, or one-to-zero situations are easily generated during matching operations. To address this situation, the present invention sets an appropriate loss function without freezing network parameters, so that the network in the SAM image segmentation model can obtain the most appropriate granularity through optical flow estimation model segmentation, thereby facilitating network convergence.

[0036] A2. In the semantic segmentation module, an improved segmentation decoder is used to convert the implicit token into the segmentation results of the two images;

[0037] In the semantic segmentation module, the decoder is located in the next layer of the model and is used to generate the final output prediction mask, that is, the segmentation result. In order to solve the problem that the SAM image segmentation model can only output a set of masks according to the prompt, but cannot directly output the complete segmentation result of a picture, the present invention establishes an improved segmentation decoder, which is used to directly output a set of global segmentation masks according to the input token. The improved segmentation decoder directly performs self-attention decoding through the SAM encoder, thereby directly inputting a set of global segmentation prediction masks after upsampling, thereby outputting the segmentation result of a complete picture. According to experiments, performing two cycles of self-attention decoding operations on the improved segmentation decoder has been able to achieve sufficiently good results.

[0038] A3. In the semantic matching module, pixel-level matching is performed on the predicted mask corresponding to the segmentation result.

[0039] The shape of the prediction mask output in step A2 is H*W*C, where C is the number of channels, which is generally set to 100, representing 99 foreground objects and 1 background object class.

[0040] Based on the masks of the two frames, the pixels at the specified positions in the first frame and the second frame that belong to the same class are determined. Specifically, in the semantic matching module, a class threshold is first set, and pixels in the first frame whose pixel matching degree exceeds the class threshold are regarded as pixels that can be semantically matched. After performing the above operation on each pixel in the first frame, a four-dimensional cost matching matrix is ​​constructed for the pixels that can be semantically matched, and the pixels that can be semantically matched are calculated and connected with the image features in the input image, and the calculated connection results are input into the GRU. In this way, the boundary of the target is judged in the network through the information of the segmentation module to avoid the problem of target adhesion in the image features and increase the pixel matching degree.

[0041] Specifically, the core idea of ​​constructing a four-dimensional cost matching matrix through semantic matching is that after two input images pass through the semantic segmentation encoder, a low-resolution image is obtained. By calculating the inner product of any pixel in the two input images, their similarity in the feature space is calculated to represent the probability that the pixels belong to the same category.

[0042] In one embodiment, the loss function includes a semantic classification loss function and a segmentation granularity loss function, which are used to speed up the iteration of the network.

[0043] When using the SAM image segmentation model to process and enhance image features in the optical flow estimation model, some problems will arise if the SAM image segmentation model is directly used to generate prediction masks for segmentation. Since the targets in the two frames of the image will be deformed or displaced when the SAM image segmentation model performs the segmentation task, an target in the first frame will be segmented into a different category from an target in the second frame, resulting in mismatching problems.

[0044] For example, in the first frame, a pixel represents the implicit semantics of "hand," but in the second frame, the corresponding pixel is segmented into the implicit semantics of "person." This means that the confidence of these two pixels in different channels reaches the maximum. In this case, if the dot product is performed using the features of these two pixels, a very low match will be obtained. To further address the above problem, the present invention establishes a loss function to fine-tune the granularity of the processed and enhanced image features, thereby better adjusting the adaptability of the SAM image segmentation model for optical flow tasks.

[0045] Among them, the semantic classification loss function first constructs a four-dimensional similarity matrix S according to the true optical flow value of the data set. Specifically, according to the true value optical flow vector, the corresponding position of a pixel in the first frame in the second frame is found, and the matrix value of the position is set to 1, and the rest are set to 0. In this way, a sparse four-dimensional similarity matrix S is generated according to the ground truth. Its shape is the same as that of the semantic matching matrix, both of which are H*W*H*W, and a binary cross entropy loss function is used to measure the similarity between the predicted four-dimensional similarity matrix S and the true value matrix G; wherein, the higher the similarity, the smaller the loss function, wherein the binary cross entropy loss function is defined as:

[0046]

[0047] The loss function above measures the similarity between a pixel in the ground-truth matrix and the corresponding predicted pixel. As the matrix definition indicates, the loss function is minimal for a pixel in the first frame only if it belongs to the same category as the corresponding ground-truth pixel in the second frame, or if there is a high degree of semantic match. This constraint effectively ensures that corresponding pixels in the two frames are segmented into the same feature channel implicitly representing the same category by the SAM image segmentation model network.

[0048] In the described segmentation granularity loss function, at first SAM image segmentation model is introduced into the network, and at this moment, SAM image segmentation model produces different segmentation granularities when segmenting two pictures.A certain target that occurs in the first frame will be divided into a plurality of smaller sub-targets in the second frame.This is not conducive to the convergence of network in feature space.In order to address this problem, the segmentation granularity loss function is introduced among the present invention, is used for balancing the granularity of the SAM image segmentation model on the segmentation task, makes it accomplish the consistency of segmentation granularity as far as possible on the target of two adjacent two frames.

[0049] Specifically, the segmentation masks obtained from the previous and next frames are first clustered separately, so that pixels with similar large values ​​in a certain channel are grouped into the same category, indicating that these pixels belong to the same segmentation target. Next, the Hungarian matching algorithm is used to find the optimal match based on the distance between cluster centers. After finding the optimal match, the intersection over union (IoU) of the corresponding mask range is calculated and used as the loss function.

[0050]

[0051] In summary, the present invention proposes an explicit optical flow estimation model algorithm that incorporates the SAM image segmentation model. This algorithm aims to provide the network with additional semantic information, building on the optical flow prediction information provided by existing networks. This allows the network to not only match objects based on local image features but also identify different foreground objects using global semantic information, thereby avoiding the problem of object adhesion. First, a SAM image segmentation model network is proposed to preprocess the input image and extract features, thereby obtaining a feature map with strong semantic information. The SAM encoder accepts an image as input, requiring the image to be split into small blocks before input. The improved segmentation decoder then generates a prediction mask, which indicates the class of a particular object in the image. Finally, semantic matching is performed on all objects in the two frames, determining the corresponding object in the second frame for each pixel in the first frame. The results are then concatenated via a four-dimensional cost volume matrix and input into a GRU. Based on this, the similarity between the segmentation masks of the two frames is calculated to indicate which classes represent the same object, thereby weighting their matching scores accordingly, thereby addressing the problem of object adhesion. Finally, two different loss functions were designed. The first semantic classification loss function is used to solve the problem that the corresponding pixels of the previous and next frames are segmented into different categories. The second segmentation granularity loss is used to solve the problem that the previous and next frames have different segmentation granularity and thus have different numbers of objects.

[0052] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0053] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0054] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0055] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A display optical flow estimation method based on semantic segmentation, characterized in that: The specific steps are: S1. Establish an optical flow estimation model based on the RAFT model; S2. Pre-training the image feature extraction part of the optical flow estimation model network using the SAM image segmentation model, processing and enhancing the image features in the optical flow estimation model network through the pre-training, and obtaining the processed and enhanced image features; S3. Establish a loss function to fine-tune the granularity of the processed and enhanced image features.

2. The display optical flow estimation method based on semantic segmentation according to claim 1, characterized in that: When the image feature extraction part is pre-trained, the back layer parameters of the SAM image segmentation model are updated as the network is updated.

3. The display optical flow estimation method based on semantic segmentation according to claim 1, characterized in that: The SAM image segmentation model includes a SAM encoder, a semantic segmentation module and a semantic matching module; wherein, in the step of using the SAM image segmentation model to process and enhance image features in the optical flow estimation model, the processing and enhancement method includes: A1. Encode the two input images into implicit tokens through the SAM encoder; A2. In the semantic segmentation module, an improved segmentation decoder is used to convert the implicit token into the segmentation results of the two images; A3. In the semantic matching module, pixel-level matching is performed on the predicted mask corresponding to the segmentation result.

4. The display optical flow estimation method based on semantic segmentation according to claim 3, characterized in that: The semantic segmentation module described in step A2 uses an improved segmentation decoder that can directly perform self-attention decoding through the SAM encoder and directly input a set of global segmentation prediction masks after upsampling, thereby outputting the segmentation result of a complete image.

5. The display optical flow estimation method based on semantic segmentation according to claim 4, characterized in that: A recurrent self-attention decoding operation is performed on the improved segmentation decoder.

6. The display optical flow estimation method based on semantic segmentation according to claim 3, characterized in that: In the semantic matching module, a class threshold is first set, and pixels in the first frame image whose pixel matching degree exceeds the class threshold are regarded as pixels capable of semantic matching, and a four-dimensional cost matching matrix is ​​constructed based on this.

7. The display optical flow estimation method based on semantic segmentation according to claim 6, characterized in that: The pixels capable of semantic matching are calculated and connected with the image features in the input image through the four-dimensional cost matching matrix, and the calculation connection results are input into the GRU.

8. The display optical flow estimation method based on semantic segmentation according to claim 1, characterized in that: The loss function includes a semantic classification loss function and a segmentation granularity loss function.

9. The display optical flow estimation method based on semantic segmentation according to claim 8, characterized in that: In the semantic classification loss function, a four-dimensional similarity matrix is ​​constructed according to the true optical flow values ​​in the data set.

10. The display optical flow estimation method based on semantic segmentation according to claim 9, characterized in that: In the semantic classification loss function, after constructing the four-dimensional similarity matrix, the binary cross entropy loss function is used to predict the similarity of the pixels between the four-dimensional similarity matrix and the true value matrix to determine the size of the semantic classification loss function; wherein, the higher the similarity, the smaller the loss function.

Images