A method, device and medium for inter-frame image segmentation based on recursive neural network

Abstract

The present invention discloses a recurrent neural network-based inter-frame image segmentation method, device, and medium, relating to the field of image processing technology. The method comprises the following steps: extracting local feature information of the current continuous inter-frame image using a convolutional neural network; constructing a geometric transformation model based on each local feature information using Lie group theory, and extracting geometric change information of the target object in the image based on the geometric transformation model; fusing the dynamic information of adjacent inter-frame images using a recurrent neural network to extract temporal features between adjacent inter-frame images based on the geometric change information; and performing segmentation prediction on each inter-frame image in the current continuous inter-frame image using a segmentation network based on the temporal features. The present invention utilizes Lie groups to capture the continuous geometric transformations between image frames, maintaining the accuracy of the target object's shape and edges. This makes the segmentation method more robust to geometric transformations, and is particularly effective when processing complex backgrounds and fast-moving scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an inter-frame image segmentation method, device and medium based on a recursive neural network. Background Art

[0002] Video segmentation theory aims to divide frames in a video sequence into distinct semantic regions for better understanding and analysis of the scene. Its core task is to detect and segment target objects or regions of interest in a video, often incorporating temporal and spatial information to improve segmentation accuracy. Classic methods include pixel-level frame differencing, optical flow, region growing, and edge detection. With the development of deep learning, methods such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have become increasingly mainstream, achieving more accurate segmentation by learning from image features and temporal information. Video segmentation has broad applications in various fields. For example, in healthcare, it is used to analyze medical images, helping doctors accurately locate and identify lesions. In sporting events, video segmentation can be used for motion capture and game analysis, enhancing the audience experience. In film and television production, video segmentation technology enables more accurate and efficient special effects creation. In virtual reality and augmented reality, video segmentation helps achieve a seamless fusion of real environments and virtual objects. Furthermore, video segmentation is used in robotic vision for environmental perception and object recognition, improving automation. With technological advancements, the application scope and accuracy of video segmentation will continue to expand. In autonomous driving, it can identify and segment roads, pedestrians, and vehicles to aid navigation and obstacle avoidance. In intelligent surveillance, video segmentation can detect abnormal behavior or suspicious individuals in real time, improving safety.

[0003] Traditional video segmentation methods such as frame differencing and optical flow are computationally simple and suitable for real-time applications, but they suffer from low accuracy when processing complex scenes. Region growing and edge detection methods can provide more detailed segmentation results, but are sensitive to noise and lighting changes and prone to missegmentation. Machine learning-based segmentation methods, such as random forests, have some generalization capabilities but rely on manual feature extraction and have limited flexibility. Deep learning methods such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) can automatically extract multi-level features and achieve high-precision segmentation, but require large amounts of labeled data for training and consume a lot of computing resources. Summary of the Invention

[0004] In order to improve the accuracy of video segmentation and reduce computing resource consumption, the present invention proposes an inter-frame image segmentation method based on a recursive neural network, comprising the steps of:

[0005] S1: Extract local feature information of the current continuous frame image through convolutional neural network;

[0006] S2: Construct a geometric transformation model based on each local feature information through Lie group theory, and extract the geometric change information of the target object in the image based on the geometric transformation model;

[0007] S3: Based on the geometric change information, the dynamic information of adjacent inter-frame images is fused through a recurrent neural network to extract the temporal features between adjacent inter-frame images;

[0008] S4: Perform segmentation prediction on each inter-frame image in the current continuous inter-frame image through the segmentation network according to the temporal characteristics.

[0009] Furthermore, in step S2, the geometric change information of the target object is expressed as the following formula:

[0010]

[0011] Where, To describe Lie algebra elements of geometric transformations under the time, is an exponential map, is an element in the Lie group, expressed as Image feature map between frames at a certain moment Geometric transformations between frames, for Image feature map between frames at a certain moment The geometric transformation information.

[0012] Furthermore, in step S3, the time series features are extracted using the following formula:

[0013]

[0014] Where, To capture timing dependencies Always hide the status, is the activation function, and is the weight matrix, is the bias, is the weight matrix of the output layer, is the bias of the output layer, for The temporal characteristics of the moment, for Image feature map between frames at a certain moment The geometric transformation information.

[0015] Furthermore, in step S4, the segmentation prediction is expressed as the following formula:

[0016]

[0017] Where, for The segmentation result of the image between frames at a certain moment is used to indicate the probability that each pixel belongs to a different category. is the activation function, is the convolution operation, for The temporal characteristics of moments.

[0018] Furthermore, the geometric consistency loss is used to ensure the consistency of the segmentation results under geometric transformation. The formula is expressed as follows:

[0019]

[0020] Where, is the geometric consistency loss, for The segmentation result of the image between frames at time t, To describe Lie algebra elements of geometric transformations under the time, is an exponential map, The total number of frames in the current continuous inter-frame image.

[0021] Furthermore, the temporal consistency loss is used to ensure the smoothness of the segmentation results in the time dimension. The formula is expressed as follows:

[0022]

[0023] Where, It is the loss of timing consistency.

[0024] Furthermore, the overall model training constraint is implemented through the total loss function, which is expressed as follows:

[0025]

[0026] Where, is the total loss, is the cross entropy loss for segmentation, is the weight coefficient of geometric consistency loss, is the weight coefficient of temporal consistency loss.

[0027] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the inter-frame image segmentation method based on a recursive neural network.

[0028] Also included is a device for processing data, comprising:

[0029] a memory having a computer program stored thereon;

[0030] A processor is used to execute the computer program in the memory to implement the steps of the inter-frame image segmentation method based on recursive neural network.

[0031] Compared with the prior art, the present invention has at least the following beneficial effects:

[0032] (1) The present invention discloses a method, apparatus, and medium for inter-frame image segmentation based on a recursive neural network. By utilizing Lie groups to capture continuous geometric transformations between image frames, the method can maintain the accuracy of object shapes and edges, which makes the segmentation method more robust to geometric transformations, especially when dealing with complex backgrounds and fast-moving scenes.

[0033] (2) Combining recurrent neural networks (RNNs) to model temporal relationships in video sequences, this method can effectively capture the changes of objects in dynamic scenes, better process long sequence data, and ensure the coherence and consistency of segmentation results in the temporal dimension;

[0034] (3) By simulating the human brain’s ability to understand visual motion, the Lie group and RNN are combined to enhance the ability to segment object changes in complex scenes. This not only improves the segmentation accuracy, but also enhances the overall stability of the system, making it closer to the performance of the human visual system.

[0035] (4) CNN is used to automatically extract local features from each frame, capture edge and texture information in the image, and provide a solid foundation for subsequent segmentation. This automated feature extraction method reduces dependence on manually labeled data and improves the generalization ability and adaptability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A step diagram of an inter-frame image segmentation method based on recurrent neural networks. DETAILED DESCRIPTION

[0037] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solutions of the present invention, but the present invention is not limited to these embodiments.

[0038] The core of video segmentation methods lies in dividing each frame into regions with similar semantic features, enabling accurate object recognition and tracking in dynamic scenes. Traditional frame differencing and optical flow methods rely on spatiotemporal variations between pixels, but are limited in their effectiveness against rapid motion and complex backgrounds. Methods such as region growing and edge detection leverage pixel similarity and edge information to achieve segmentation, and are suitable for static scenes or scenes with relatively simple backgrounds. In recent years, the application of deep learning techniques, particularly convolutional neural networks (CNNs), has enabled the automatic learning of high-dimensional features, significantly improving segmentation accuracy. Recurrent neural networks (RNNs) further exploit the temporal nature of video to effectively capture dynamic information and enhance the ability to track moving objects. Advanced models such as generative adversarial networks (GANs) achieve more detailed and realistic segmentation results through adversarial learning mechanisms, making them suitable for fine-grained segmentation tasks in complex environments. Overall, video segmentation methods are continuously evolving towards higher accuracy, greater robustness, and broader adaptability. Recurrent neural networks (RNNs) are a neural network architecture specialized for processing sequential data, capturing temporal dependencies within a sequence through recurrently connected nodes. RNN performs well in tasks such as time series processing, natural language processing, and speech recognition because it can remember previous input information and pass its influence to subsequent time steps. However, classic RNN faces the problems of gradient vanishing and gradient exploding, which limits its application in long sequences. For this reason, Figure 1 As shown, the present invention proposes an inter-frame image segmentation method based on a recursive neural network, comprising the steps of:

[0039] S1: Extract local feature information of the current continuous frame image through convolutional neural network;

[0040] S2: Construct a geometric transformation model based on each local feature information through Lie group theory, and extract the geometric change information of the target object in the image based on the geometric transformation model;

[0041] S3: Based on the geometric change information, the dynamic information of adjacent inter-frame images is fused through a recurrent neural network to extract the temporal features between adjacent inter-frame images;

[0042] S4: Perform segmentation prediction on each inter-frame image in the current continuous inter-frame image through the segmentation network according to the temporal characteristics.

[0043] In the present invention, the shortcomings of RNN in long sequence applications are compensated by combining the time series processing capability of RNN with the advantages of Lie groups and Lie algebras in processing continuous transformations and geometric symmetries.

[0044] A Lie group is a set that has both group structure and smooth manifold structure, where group operations (multiplication and inversion) are smooth mappings. This means that a Lie group is not only an abstract algebraic entity, but its elements can be understood as points in a geometric space, and the transitions between these points are continuous and differentiable. An important property of Lie groups is that their local properties can be described by so-called Lie algebras. Lie algebras are vector spaces consisting of the tangent spaces of Lie groups around the identity element, coupled with a binary operation called a Lie bracket. This Lie bracket captures the relationship between the "infinitesimal" transformations between elements in the Lie group.

[0045] Assume that the input video frame sequence is ,in express The inter-frame image at the moment, is the total duration of the input video frames, so The inter-frame image at the moment is also the We first use a convolutional neural network (CNN) to extract the feature representation of each inter-frame image:

[0046]

[0047] Where, for Feature map of the image between frames at a certain moment, It is a convolutional neural network.

[0048] In Lie Group Under the representation of , the geometric transformation of the video frame (such as rotation, translation, scaling, etc.) is performed on each element Under the representation of The geometric transformation between different frames can be expressed as:

[0049]

[0050] In this formula, Expressed as Image feature map between frames at a certain moment Geometric transformations between frames, for Image feature map between frames at a certain moment The geometric transformation information.

[0051] The elements in the Lie algebra is related to the Lie group elements through exponential mappings:

[0052]

[0053] Where, is an exponential map that maps Lie algebra elements to Lie group elements.

[0054] Then, at each time step Down, Moment feature graph The geometric transformation of can be expressed as a time-dependent Lie algebra element under the element description of the Lie group Function:

[0055]

[0056] In this formula, To describe Lie algebra elements of geometric transformations under the time, Expressed as Image feature map between frames at a certain moment Geometric transformations between frames.

[0057] Based on the feature information after Lie group transformation, we combine it with the recurrent neural network (RNN) through the hidden state To capture the temporal dependencies between consecutive frames. The state update equation of RNN is:

[0058]

[0059] in, is the activation function, and is the weight matrix, For bias.

[0060] Output of RNN ( The temporal characteristics of the moment can be expressed as:

[0061]

[0062] Where, is the weight matrix of the output layer, is the bias of the output layer, for The temporal characteristics of moments.

[0063] The output of the RNN Input to the segmentation head, and generate the classification probability of each pixel through the Softmax activation function:

[0064]

[0065] Where, for The segmentation result of the image between frames at a certain moment is used to indicate the probability that each pixel belongs to a different category. is the activation function, is the convolution operation.

[0066] In order to ensure the accuracy of image segmentation between model frames, during the model training process, the geometric consistency loss is used to ensure the consistency of the segmentation results under geometric transformation, and the temporal consistency loss is used to ensure the smoothness of the segmentation results in the time dimension, which can be expressed as the following formula:

[0067]

[0068] Where, is the geometric consistency loss, is the timing consistency loss, The total number of frames in the current continuous inter-frame image.

[0069] The overall loss of the model can be expressed as:

[0070]

[0071] Where, is the total loss, is the cross entropy loss for segmentation.

[0072] By continuously optimizing the loss function during the training process, the weights of the model are trained using the gradient descent method. and Lie algebra elements , so that the model can generate the segmentation results of each frame through Lie group transformation and RNN according to the input video frame sequence, and ensure the consistency of geometry and timing.

[0073] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the inter-frame image segmentation method based on a recursive neural network.

[0074] Also included is a device for processing data, comprising:

[0075] a memory having a computer program stored thereon;

[0076] A processor is used to execute the computer program in the memory to implement the steps of the inter-frame image segmentation method based on recursive neural network.

[0077] In summary, the present invention proposes a method, device and medium for inter-frame image segmentation based on recursive neural networks. By utilizing Lie groups to capture continuous geometric transformations between image frames, this method can maintain the accuracy of object shapes and edges, which makes the segmentation method more robust to geometric transformations, especially when dealing with complex backgrounds and fast-motion scenes.

[0078] Combining recurrent neural networks (RNNs) to model temporal relationships in video sequences, this method can effectively capture the changes of objects in dynamic scenes, better process long sequence data, and ensure the coherence and consistency of segmentation results in the temporal dimension.

[0079] By simulating the human brain's ability to understand visual motion, the combination of Lie groups and RNNs enhances the ability to segment object changes in complex scenes. This not only improves segmentation accuracy, but also enhances the overall stability of the system, bringing it closer to the performance of the human visual system.

[0080] CNN is used to automatically extract local features from each frame, capturing edge and texture information in the image, providing a solid foundation for subsequent segmentation. This automated feature extraction method reduces dependence on manually labeled data and improves the generalization and adaptability of the model.

[0081] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0082] In addition, in the present invention, descriptions such as "first," "second," and "one" are for descriptive purposes only and should not be understood to indicate or imply their relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0083] In the present invention, unless otherwise specified or limited, the terms "connection" and "fixation" should be understood in a broad sense. For example, "fixation" can mean fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two elements or interaction between two elements, unless otherwise specified. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0084] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A method for inter-frame image segmentation based on recurrent neural network, characterized in that: Including steps: S1: Extract local feature information of the current continuous frame image through convolutional neural network; S2: Construct a geometric transformation model based on each local feature information through Lie group theory, and extract the geometric change information of the target object in the image based on the geometric transformation model; S3: Based on the geometric change information, the dynamic information of adjacent inter-frame images is fused through a recurrent neural network to extract the temporal features between adjacent inter-frame images; S4: performing segmentation prediction on each inter-frame image in the current continuous inter-frame image through a segmentation network according to the temporal features; In step S2, the geometric change information of the target object is expressed as the following formula: Where, is the Lie algebra element describing the geometric transformation at time t, is an exponential map, is an element in the Lie group, represented as the inter-frame image feature map at time t Geometric transformations between frames, is the inter-frame image feature map at time t Geometric transformation information; In step S3, the time series features are extracted using the following formula: Where, is the hidden state at time t used to capture the temporal dependency, is the activation function, and is the weight matrix, is the bias, is the weight matrix of the output layer, is the bias of the output layer, is the time series feature at time t, is the inter-frame image feature map at time t The geometric transformation information.

2. The inter-frame image segmentation method based on a recurrent neural network according to claim 1, wherein: In the step S4, the segmentation prediction is expressed as the following formula: Where, for The segmentation result of the image between frames at a certain moment is used to indicate the probability that each pixel belongs to a different category. is the activation function, is the convolution operation, for The temporal characteristics of moments.

3. The inter-frame image segmentation method based on recurrent neural network according to claim 1, wherein: The geometric consistency loss is used to ensure the consistency of the segmentation results under geometric transformation. The formula is as follows: Where, is the geometric consistency loss, for The segmentation result of the image between frames at time t, To describe Lie algebra elements of geometric transformations under the time, is an exponential map, The total number of frames in the current continuous inter-frame image.

4. The inter-frame image segmentation method based on recursive neural network according to claim 3, wherein: The temporal consistency loss is used to ensure the smoothness of the segmentation results in the time dimension. The formula is as follows: Where, It is the loss of timing consistency.

5. The inter-frame image segmentation method based on recursive neural network according to claim 4, characterized in that: The total loss function is used to constrain the training of the entire model. The formula is as follows: Where, is the total loss, is the cross entropy loss for segmentation, is the weight coefficient of geometric consistency loss, is the weight coefficient of temporal consistency loss.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the segmentation method according to any one of claims 1 to 5 are implemented.

7. A device for processing data, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the segmentation method according to any one of claims 1 to 5.

Images