Interframe image segmentation method and device based on recurrent neural network, and medium

Abstract

The invention discloses an inter-frame image segmentation method and device based on a recurrent neural network, and a medium, and relates to the technical field of image processing, and the method comprises the steps: extracting the local feature information of a current continuous inter-frame image through a convolutional neural network; constructing a geometric transformation model through the Lie group theory according to the local feature information, and extracting geometric change information of the target object in the image based on the geometric transformation model; based on the geometric change information, fusing the dynamic information of the adjacent inter-frame images through a recurrent neural network to extract time sequence features between the adjacent inter-frame images; and performing segmentation prediction on each inter-frame image in the current continuous inter-frame images through a segmentation network according to the time sequence characteristics. According to the method, the Lie group is used for capturing continuous geometric transformation between image frames, and the accuracy of the shape and the edge of a target object is kept, so that the segmentation method has better robustness for geometric transformation, and particularly, the method is excellent in performance when processing complex backgrounds and rapid motion scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an inter-frame image segmentation method, device and medium based on a recursive neural network. Background Art

[0002] Video segmentation theory aims to divide frames in a video sequence into different semantic regions in order to better understand and analyze the scene. Its core task is to detect and segment target objects or regions of interest in the video, usually combining temporal and spatial information to improve the accuracy of segmentation. Classic methods include techniques such as pixel-level frame difference, optical flow, region growing, and edge detection. With the development of deep learning, methods such as convolutional neural networks (CNN) and recurrent neural networks (RNN) have gradually become mainstream, achieving more accurate segmentation by learning image features and temporal information. Video segmentation has a wide range of applications in many fields. For example, in the medical field, it is used to analyze medical images to help doctors accurately locate and identify lesion areas. In sports events, video segmentation can be used for motion capture and game analysis to enhance the audience experience. In film and television production, video segmentation technology makes special effects production more accurate and efficient. In virtual reality and augmented reality, video segmentation helps achieve seamless integration of real environments and virtual objects. In addition, video segmentation is used in robot vision for environmental perception and object recognition to improve the degree of automation. With the advancement of technology, the application scope and accuracy of video segmentation will continue to expand. In autonomous driving, it can identify and segment roads, pedestrians, and vehicles to help with navigation and obstacle avoidance. In intelligent surveillance, video segmentation can detect abnormal behavior or suspicious people in real time to improve safety.

[0003] Traditional video segmentation methods such as frame difference and optical flow technology are simple to calculate and suitable for real-time applications, but have low accuracy when processing complex scenes. Region growing and edge detection methods can provide more detailed segmentation results, but are sensitive to noise and lighting changes and are prone to mis-segmentation. Machine learning-based segmentation methods, such as random forests, have certain generalization capabilities, but rely on manual feature extraction and have limited flexibility. Deep learning methods such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) can automatically extract multi-level features and achieve high-precision segmentation, but require a large amount of labeled data for training and consume a lot of computing resources. Summary of the invention

[0004] In order to improve the accuracy of video segmentation and reduce the consumption of computing resources, the present invention proposes an inter-frame image segmentation method based on a recursive neural network, comprising the steps of: S1: Extract local feature information of the current continuous frame image through convolutional neural network; S2: construct a geometric transformation model based on each local feature information through Lie group theory, and extract the geometric change information of the target object in the image based on the geometric transformation model; S3: Based on the geometric change information, the dynamic information of adjacent inter-frame images is fused through a recurrent neural network to extract the temporal features between adjacent inter-frame images; S4: Perform segmentation prediction on each inter-frame image in the current continuous inter-frame image through a segmentation network according to the temporal characteristics.

[0005] Furthermore, in the step S2, the geometric change information of the target object is expressed as the following formula: In the formula, For description Lie algebra elements of geometric transformations under time, is an exponential map, is an element in the Lie group, expressed as Image feature map between frames at a certain moment The geometric transformation between different frames, for Image feature map between frames at a certain moment The geometric transformation information.

[0006] Furthermore, in step S3, the time series features are extracted by the following formula: In the formula, To capture timing dependencies Always hide the status, is the activation function, and is the weight matrix, is the bias, is the weight matrix of the output layer, is the bias of the output layer, for The temporal characteristics of the moment, for Image feature map between frames at a certain moment The geometric transformation information.

[0007] Furthermore, in step S4, the segmentation prediction is expressed as the following formula: In the formula, for The segmentation result of the image between frames at a certain moment is used to indicate the probability that each pixel belongs to a different category. is the activation function, is the convolution operation, for The timing characteristics of the moment.

[0008] Furthermore, the geometric consistency loss is used to ensure the consistency of the segmentation results under geometric transformation. The formula is expressed as follows: In the formula, is the geometric consistency loss, for The segmentation result of the image between frames at time t, For description Lie algebra elements of geometric transformations under time, is an exponential map, It is the total number of frames of the current continuous inter-frame image.

[0009] Furthermore, the temporal consistency loss is used to ensure the smoothness of the segmentation results in the time dimension. The formula is as follows: In the formula, It is the timing consistency loss.

[0010] Furthermore, the overall model training constraints are implemented through the total loss function, and the formula is expressed as follows: In the formula, is the total loss, is the cross entropy loss for segmentation, is the weight coefficient of geometric consistency loss, is the weight coefficient of the temporal consistency loss.

[0011] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the inter-frame image segmentation method based on a recursive neural network are implemented.

[0012] Also included is a device for processing data, comprising: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of the inter-frame image segmentation method based on recursive neural network.

[0013] Compared with the prior art, the present invention has at least the following beneficial effects: (1) The present invention discloses a method, device and medium for image segmentation between frames based on a recursive neural network. By using Lie groups to capture continuous geometric transformations between image frames, the method can maintain the accuracy of object shapes and edges, which makes the segmentation method more robust to geometric transformations, especially when dealing with complex backgrounds and fast-moving scenes. (2) Combining recurrent neural networks (RNNs) to model temporal relationships in video sequences, this method can effectively capture the changes of objects in dynamic scenes, better process long sequence data, and ensure the coherence and consistency of segmentation results in the temporal dimension; (3) By simulating the human brain’s ability to understand visual motion, the Lie group and RNN are combined to enhance the ability to segment object changes in complex scenes. This not only improves the segmentation accuracy, but also improves the overall stability of the system, making it closer to the performance of the human visual system. (4) CNN is used to automatically extract local features from each frame, capture edge and texture information in the image, and provide a solid foundation for subsequent segmentation. This automated feature extraction method reduces the reliance on manually labeled data and improves the generalization and adaptability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A step diagram of an inter-frame image segmentation method based on recurrent neural network. DETAILED DESCRIPTION

[0015] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.

[0016] The core of video segmentation methods is to divide each frame in the video into regions with similar semantic features, so as to achieve accurate recognition and tracking of objects in dynamic scenes. Traditional frame difference and optical flow methods rely on the spatiotemporal changes between pixels, but have limited effects when dealing with fast motion and complex backgrounds. Methods such as region growing and edge detection achieve segmentation through pixel similarity and edge information, which are suitable for static scenes or scenes with relatively simple backgrounds. In recent years, the application of deep learning technology, especially convolutional neural networks (CNN), has made it possible to automatically learn high-dimensional features, significantly improving segmentation accuracy. Recurrent neural networks (RNNs) further utilize the temporal characteristics of videos to effectively capture dynamic information and enhance the ability to track moving objects. Advanced models such as generative adversarial networks (GANs) achieve more delicate and realistic segmentation effects through the mechanism of adversarial learning, which are suitable for fine segmentation tasks in complex environments. Overall, video segmentation methods are developing towards higher accuracy, stronger robustness, and wider adaptability. Recurrent neural networks (RNNs) are a neural network architecture that specializes in processing sequence data, capturing temporal dependencies in sequences through cyclically connected nodes. RNN performs well in processing time series, natural language processing, speech recognition and other tasks because it can remember the previous input information and pass its influence to the subsequent time steps. However, the classic RNN faces the problem of gradient vanishing and gradient exploding, which limits its application in long sequences. For this reason, Figure 1 As shown, the present invention proposes an inter-frame image segmentation method based on a recursive neural network, comprising the steps of: S1: Extract local feature information of the current continuous frame image through convolutional neural network; S2: construct a geometric transformation model based on each local feature information through Lie group theory, and extract the geometric change information of the target object in the image based on the geometric transformation model; S3: Based on the geometric change information, the dynamic information of adjacent inter-frame images is fused through a recurrent neural network to extract the temporal features between adjacent inter-frame images; S4: Perform segmentation prediction on each inter-frame image in the current continuous inter-frame image through a segmentation network according to the temporal characteristics.

[0017] In the present invention, the shortcomings of RNN in long sequence applications are compensated by combining the time series processing capability of RNN with the advantages of Lie groups and Lie algebras in processing continuous transformations and geometric symmetries.

[0018] A Lie group is a set that has both a group structure and a smooth manifold structure, where group operations (multiplication and inversion) are smooth mappings. This means that the Lie group is not only an abstract algebraic entity, but its elements can be understood as points in a geometric space, and the transitions between these points are continuous and differentiable. An important property of Lie groups is that their local properties can be described by the so-called Lie algebra. Lie algebra is a vector space consisting of the tangent space of the Lie group near the identity element, coupled with a binary operation called the Lie bracket. This Lie bracket captures the relationship between the "infinitesimal" transformations between elements in the Lie group.

[0019] Assume that the input video frame sequence is ,in express The inter-frame image at the moment, is the total duration of the input video frame, so The inter-frame image at the moment is also the We first use a convolutional neural network (CNN) to extract the feature representation of each frame of the inter-frame image: In the formula, for The feature map of the image between frames at time instant, It is a convolutional neural network.

[0020] In Lie Group Under the representation of , the geometric transformation of the video frame (such as rotation, translation, scaling, etc.) is performed on each element Under the representation of The geometric transformation between different frames can be expressed as: In this formula, Expressed as Image feature map between frames at a certain moment The geometric transformation between different frames, for Image feature map between frames at a certain moment The geometric transformation information.

[0021] The elements of the Lie algebra is related to the Lie group elements through exponential mapping: In the formula, is an exponential map that maps Lie algebra elements to Lie group elements.

[0022] Then, at each time step Down, Moment feature graph The geometric transformation of can be expressed as a time-dependent Lie algebra element under the description of the elements in the Lie group. Function: In this formula, For description Lie algebra elements of geometric transformations under time, Expressed as Image feature map between frames at a certain moment Geometric transformations between frames.

[0023] Based on the feature information after Lie group transformation, we combine it with recurrent neural network (RNN) through hidden state To capture the temporal dependencies between consecutive frames. The state update equation of RNN is: in, is the activation function, and is the weight matrix, For bias.

[0024] Output of RNN ( The time series characteristics of the moment can be expressed as: In the formula, is the weight matrix of the output layer, is the bias of the output layer, for The timing characteristics of the moment.

[0025] The output of the RNN Input to the segmentation head, and generate the classification probability of each pixel through the Softmax activation function: In the formula, for The segmentation result of the image between frames at a certain moment is used to indicate the probability that each pixel belongs to a different category. is the activation function, is the convolution operation.

[0026] In order to ensure the accuracy of image segmentation between model frames, during the model training process, the geometric consistency loss is used to ensure the consistency of the segmentation results under geometric transformation, and the temporal consistency loss is used to ensure the smoothness of the segmentation results in the time dimension, which can be expressed as the following formula: In the formula, is the geometric consistency loss, is the timing consistency loss, It is the total number of frames of the current continuous inter-frame image.

[0027] The overall loss of the model can be expressed as: In the formula, is the total loss, is the cross entropy loss for segmentation.

[0028] By continuously optimizing the loss function during the training process, the weights of the model are trained using the gradient descent method. And the Lie algebra elements The parameters of the model enable the model to generate the segmentation results of each frame through Lie group transformation and RNN according to the input video frame sequence, and ensure the consistency of geometry and timing.

[0029] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the inter-frame image segmentation method based on a recursive neural network are implemented.

[0030] Also included is a device for processing data, comprising: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of the inter-frame image segmentation method based on recursive neural network.

[0031] In summary, the present invention proposes an inter-frame image segmentation method, device and medium based on recursive neural network. By utilizing Lie groups to capture the continuous geometric transformation between image frames, the method can maintain the accuracy of object shape and edges, which makes the segmentation method more robust to geometric transformation, especially when dealing with complex backgrounds and fast-motion scenes.

[0032] Combining recurrent neural networks (RNNs) to model temporal relationships in video sequences, this method can effectively capture the changes of objects in dynamic scenes, better process long sequence data, and ensure the coherence and consistency of segmentation results in the temporal dimension.

[0033] By simulating the human brain's ability to understand visual motion, the combination of Lie groups and RNNs enhances the ability to segment object changes in complex scenes. This not only improves the segmentation accuracy, but also improves the overall stability of the system, making it closer to the performance of the human visual system.

[0034] CNN is used to automatically extract local features from each frame, capture edge and texture information in the image, and provide a solid foundation for subsequent segmentation. This automated feature extraction method reduces dependence on manually labeled data and improves the generalization and adaptability of the model.

[0035] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0036] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0037] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0038] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A method for inter-frame image segmentation based on recursive neural network, characterized in that: Includes steps: S1: Extract local feature information of the current continuous frame image through convolutional neural network; S2: construct a geometric transformation model based on each local feature information through Lie group theory, and extract the geometric change information of the target object in the image based on the geometric transformation model; S3: Based on the geometric change information, the dynamic information of adjacent inter-frame images is fused through a recurrent neural network to extract the temporal features between adjacent inter-frame images; S4: performing segmentation prediction on each inter-frame image in the current continuous inter-frame image through a segmentation network according to the temporal features; In the step S2, the geometric change information of the target object is expressed as the following formula: In the formula, For description Lie algebra elements of geometric transformations under time, is an exponential map, is an element in the Lie group, expressed as Image feature map between frames at a certain moment The geometric transformation between different frames, for Image feature map between frames at a certain moment The geometric transformation information.

2. The inter-frame image segmentation method based on a recursive neural network as claimed in claim 1, characterized in that: In the step S3, the time series features are extracted by the following formula: In the formula, To capture timing dependencies Always hide the status, is the activation function, and is the weight matrix, is the bias, is the weight matrix of the output layer, is the bias of the output layer, for The temporal characteristics of the moment, for Image feature map between frames at a certain moment The geometric transformation information.

3. The inter-frame image segmentation method based on recursive neural network according to claim 1, characterized in that: In the step S4, the segmentation prediction is expressed as the following formula: In the formula, for The segmentation result of the image between frames at a certain moment is used to indicate the probability that each pixel belongs to a different category. is the activation function, is the convolution operation, for The timing characteristics of the moment.

4. The inter-frame image segmentation method based on a recursive neural network as claimed in claim 1, characterized in that: The geometric consistency loss is used to ensure the consistency of the segmentation results under geometric transformation. The formula is expressed as follows: In the formula, is the geometric consistency loss, for The segmentation result of the image between frames at time t, For description Lie algebra elements of geometric transformations under time, is an exponential map, It is the total number of frames of the current continuous inter-frame image.

5. The inter-frame image segmentation method based on recursive neural network as claimed in claim 4, characterized in that: The temporal consistency loss is used to ensure the smoothness of the segmentation results in the time dimension. The formula is as follows: In the formula, It is the timing consistency loss.

6. The inter-frame image segmentation method based on recursive neural network as claimed in claim 5, characterized in that: The total loss function is used to constrain the training of the entire model. The formula is as follows: In the formula, is the total loss, is the cross entropy loss for segmentation, is the weight coefficient of geometric consistency loss, is the weight coefficient of the temporal consistency loss.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the segmentation method according to any one of claims 1 to 6 are implemented.

8. A device for processing data, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the segmentation method according to any one of claims 1 to 6.

Images