Brain heuristic inter-frame image segmentation method and device based on 3D-CNN and medium

Abstract

The invention relates to a brain enlightenment inter-frame image segmentation method and device based on 3D-CNN and a medium, and the method comprises the following steps: obtaining a video image, and extracting spatial-temporal features from continuous video image frames by using the 3D-CNN; analyzing the extracted spatial-temporal features by using a differential geometric tool, and extracting geometric features; feature fusion is carried out on the spatial-temporal features and the geometric features to form fused features, and feature fusion comprises convolution operation and convolution smoothing on manifolds; and realizing accurate segmentation of inter-frame images through a segmentation network by using the fusion features. Compared with the prior art, the method has the advantages of being more accurate in segmentation result, more efficient, high in robustness, suitable for applications with high real-time requirements and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and in particular to a 3D-CNN-based brain-inspired inter-frame image segmentation method, device and medium. Background Art

[0002] The brain-inspired inter-frame image segmentation method improves the efficiency and accuracy of image segmentation by imitating the visual processing mechanism of the human brain. The brain-inspired inter-frame image segmentation method combines the principles of neuroscience and advanced deep learning techniques, enabling it to effectively handle complex visual tasks. By using brain-like convolutional neural networks and attention mechanisms, this method can adaptively capture and analyze important features in inter-frame images. Compared with traditional methods, brain-inspired methods perform better in processing diverse and dynamically changing image scenes. Its main advantage is that it can achieve high-precision segmentation at a low computational cost, which makes it have great potential in real-time applications. In addition, the brain-inspired inter-frame image segmentation method can better handle problems such as noise and blur, and improve the robustness of image processing. The brain-inspired inter-frame image segmentation method has performed well in a variety of practical applications, especially in the fields of autonomous driving, video surveillance, and medical image analysis. By imitating the visual processing mechanism of the human brain, this method can effectively handle image changes in complex dynamic scenes and improve the accuracy of target detection and tracking. It has significant advantages in applications with high real-time requirements, and can quickly respond and accurately segment key objects in images. In addition, this method can also enhance the robustness to noisy and blurred images and improve the reliability of the system.

[0003] Brain-inspired inter-frame image segmentation methods can be classified into methods based on different structures such as convolutional neural networks, recurrent neural networks, and transformers. Methods based on convolutional neural networks usually have higher segmentation accuracy and are suitable for processing local features in static images, but they are insufficient in capturing long-distance dependencies. Recurrent neural network methods can process time series information and are more suitable for continuous frame segmentation in dynamic scenes, but the computational complexity is higher. Transformer-based methods effectively capture inter-frame correlations by introducing a self-attention mechanism, improving the processing capabilities of complex scenes, but consuming more hardware resources.

[0004] How to achieve accurate and efficient video image segmentation has become a technical problem that needs to be solved. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a brain-inspired inter-frame image segmentation method, device and medium based on 3D-CNN.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to one aspect of the present invention, a brain-inspired inter-frame image segmentation method based on 3D-CNN is provided, the method comprising the following steps:

[0008] Step S1, obtaining a video image, and using a 3D-CNN network to extract spatiotemporal features from continuous video image frames;

[0009] Step S2, applying differential geometry tools to analyze the extracted spatiotemporal features and extracting geometric features;

[0010] Step S3, fusing the spatiotemporal features and the geometric features to form fused features, wherein the feature fusion includes convolution operation and convolution smoothing on the manifold;

[0011] Step S4, using the fusion features, the accurate segmentation of the inter-frame images is achieved through the segmentation network.

[0012] Preferably, the process of extracting spatiotemporal features includes:

[0013] Assume that the input video image frame sequence is Among them I t Represents the t-th frame image, T is the total number of frames; 3D convolution is applied to the video image frame sequence to extract spatial and temporal features at the same time. The formula of the 3D convolution operation is:

[0014]

[0015] in, represents the feature map of layer l, represents the 3D convolution kernel, b (l) is the bias term, σ(·) is the activation function, and * represents the 3D convolution operation.

[0016] Preferably, the process of extracting geometric features includes:

[0017] The spatiotemporal feature map Assume that each frame in the video corresponds to a point on the manifold represented by X t ∈M, Where φ(·) is the embedding map, which embeds the feature map into the manifold;

[0018] Use the metric tensor to describe the geometric properties of the local neighborhood: ds 2 =g μv dx μ dx v , where ds is the small distance on the manifold, g μν is the metric tensor, dx μ and dx νare small changes in the x-axis and y-axis of the coordinate system, respectively.

[0019] Preferably, the convolution operation on the manifold is specifically:

[0020] On a high-dimensional manifold M, define the Laplace-Beltrami operator To perform convolution operations to capture feature maps Changes on the manifold:

[0021]

[0022] Where |g| is the metric tensor g μν The determinant of g μν is the inverse matrix of the metric tensor, is the Laplace–Beltrami operator of the high-dimensional manifold M.

[0023] Preferably, the convolution smoothing on the manifold is specifically:

[0024] The feature map is smoothed using the Laplace-Beltrami operator:

[0025]

[0026] Where η is the smoothing coefficient, It is the feature map after smoothing.

[0027] Preferably, the loss function of the segmentation network in step S4 combines segmentation accuracy and temporal consistency, specifically:

[0028]

[0029]

[0030] in, is the total loss function; is the cross entropy loss of segmentation; λ temporal The weight coefficient for controlling the timing consistency loss; is the timing consistency loss; P t and P t+1 are the segmentation results of the t-th frame and the t+1-th frame, respectively, indicating the probability that each pixel belongs to a different category; T is the total number of frames.

[0031] Preferably, the feature F processed in step S3 t smooth Input to the segmentation head, and generate the classification probability of each pixel through the Softmax function:

[0032] P t =Softmax(Conv(Ft smooth ))

[0033] Among them, P t is the segmentation result of the tth frame, indicating the probability that each pixel belongs to a different category.

[0034] Preferably, the geodesic in geometry is used to describe the evolution path of the feature map on the manifold to capture the continuous changes of features in the video:

[0035]

[0036] Where D represents the covariant derivative, which ensures that the evolution of the feature on the manifold remains consistent; X t Represents a feature point on the manifold corresponding to the tth frame.

[0037] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0038] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described above when executed by a processor.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1) The present invention uses 3D-CNN to extract spatiotemporal information (change and motion information) from continuous image frames to improve the dynamic responsiveness of segmentation, and then uses differential geometry to analyze the surfaces and shapes in the image to capture the local geometric shapes of the image and help understand complex spatial relationships. The fusion of the two can more accurately capture the spatiotemporal features in complex scenes and achieve high-precision image segmentation. It effectively combines the advantages of spatial shape analysis and temporal change detection, making the segmentation results more accurate and robust, and having better adaptability to scenes with dynamic changes and complex shapes, especially in applications with high real-time requirements.

[0041] 2) The feature fusion of the present invention includes convolution operations and convolution smoothing on the manifold, which reduce noise and enhance structural features, and introduce geodesics to describe the evolution path of the feature map on the manifold, capture the continuous changes of the feature map on the manifold, help identify the spatial relationship and spatiotemporal information of complex scenes, can accurately model in high-dimensional space, and improve dynamic responsiveness, preparing for subsequent segmentation.

[0042] 3) The present invention combines segmentation accuracy and temporal consistency loss when training the segmentation network to ensure the consistency of the results in time and space, thereby outputting more stable and accurate segmentation results.

[0043] 4) The brain-inspired inter-frame image segmentation method combining 3D-CNN and differential geometry analysis of the present invention has low computational complexity and can process complex images more quickly, thereby improving the efficiency of image segmentation and being suitable for applications with high real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the process of the brain-inspired inter-frame image segmentation method based on 3D-CNN in the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0046] Based on the shortcomings of the above brain-inspired inter-frame image segmentation method, the present invention proposes a brain-inspired inter-frame image segmentation method based on differential geometry and 3D-CNN. The brain-inspired inter-frame image segmentation method is a technology that segments continuous images by simulating the visual processing of the human brain. It can intelligently identify and separate different objects in videos or continuous images, allowing machines to understand the content of the picture like humans. Compared with traditional methods, this technology can process complex images more quickly and has better performance on noisy and blurred images. Because it draws on the working principle of the brain, it has great advantages in applications that require real-time analysis such as autonomous driving and monitoring systems. 3D-CNN is a convolutional neural network that can process video or three-dimensional data. Unlike ordinary 2D-CNN, 3D-CNN not only calculates on the width and height of the image, but also operates on the time or depth dimension, enabling it to extract dynamically changing information from continuous frames. This network structure is good at analyzing video, medical imaging and other three-dimensional data, and can recognize complex spatiotemporal features. 3D-CNN performs well in tasks such as action recognition, video classification and 3D object detection because it can more comprehensively understand the motion and shape information in the data.

[0047] The present embodiment relates to a brain-inspired inter-frame image segmentation method based on 3D-CNN, which combines the spatial structure analysis capability of differential geometry with the spatiotemporal feature extraction capability of 3D convolutional neural network (3D-CNN). Differential geometry helps analyze surfaces and shapes in images, is used to capture the local geometry of images, and helps understand complex spatial relationships, while 3D-CNN can extract spatiotemporal information (change and motion information) from continuous image frames to improve the dynamic responsiveness of segmentation. By combining the two, the method can not only capture the detailed features in the image, but also handle the temporal changes in dynamic scenes, so it can accurately model and segment complex dynamic scenes in high-dimensional space. On the whole, this method can effectively improve the accuracy and robustness of inter-frame image segmentation, as well as improve the adaptability to complex dynamic scenes.

[0048] like Figure 1 , the method can be divided into four steps:

[0049] Step S1, image feature extraction: Use 3D-CNN network to extract spatiotemporal features from continuous image frames.

[0050] Capture motion information and changing patterns in the picture.

[0051] Step S2, geometric analysis: Apply differential geometry tools to analyze the extracted spatiotemporal features, understand the surface and shape features of objects in the image, extract geometric features, and help identify complex spatial structures.

[0052] Step S3, feature fusion: the spatiotemporal features and geometric features are fused to form a more comprehensive feature representation, i.e., fused features, so as to better understand the objects and background in the image.

[0053] Step S4, image segmentation: using the fused features, accurately segment different object regions through a classifier or segmentation network to achieve accurate segmentation of images between frames.

[0054] In step S1, it is assumed that the input video image frame sequence is Among them I t Represents the t-th frame image, and T is the total number of frames. 3D convolution is applied to the video image frame sequence to extract spatial and temporal features simultaneously. The formula for the 3D convolution operation is:

[0055]

[0056] in, represents the feature map of layer l, represents the 3D convolution kernel, b (l) is the bias term, σ(·) is the activation function, and * represents the 3D convolution operation.

[0057] In step S2, differential geometry is introduced, and the manifold representation is: the feature map is considered as a set of points embedded in a high-dimensional manifold M. Assume that each frame in the video corresponds to a point on the manifold, denoted as X t ∈M,X t =φ(F t (l) ), where φ(·) is the embedding map that embeds the feature map onto the manifold.

[0058] Metric tensor: On a high-dimensional manifold M, define a metric tensor g μν to measure the distance between features.

[0059] The metric tensor is used to describe the geometric properties of a local neighborhood: ds 2 =g μv dx μ dx v , where ds is a small distance on the manifold, g μν is the metric tensor, dx μ and dx ν are small changes in the x-axis and y-axis of the coordinate system, respectively.

[0060] In step S3, feature fusion includes convolution operation on the manifold and convolution smoothing.

[0061] Convolution operation on manifold: On the high-dimensional manifold M, define the Laplace-Beltrami operator To perform convolution operations to capture feature maps Changes on the manifold:

[0062]

[0063] Where |g| is the metric tensor g μν The determinant of g μν is the inverse matrix of the metric tensor, is the Laplace–Beltrami operator of the high-dimensional manifold M.

[0064] Convolution smoothing: The feature map is smoothed by the Laplace-Beltrami operator to reduce noise and enhance structural features:

[0065]

[0066] Where η is the smoothing coefficient, It is the feature map after smoothing.

[0067] Evolution of feature maps on manifolds: Geodesic in Riemann geometry is used to describe the evolution path of feature maps on manifolds to capture the continuous changes of features in the video:

[0068]

[0069] Where D represents the covariant derivative, which ensures that the evolution of the feature on the manifold (in space) remains consistent; X t Represents a feature point on the manifold corresponding to each frame.

[0070] In step S4, the features processed by the manifold are input into the segmentation head, and the classification probability of each pixel is generated by the Softmax function:

[0071] P t =Softmax(Conv(F t smooth ))

[0072] Among them, P t is the segmentation result of the tth frame, indicating the probability that each pixel belongs to a different category.

[0073] Temporal consistency loss: In order to ensure the smoothness and consistency of the segmentation results in the time dimension, temporal consistency loss is added:

[0074]

[0075] Total loss function combined loss: The final loss function combines segmentation accuracy and temporal consistency:

[0076]

[0077] in, is the cross entropy loss for segmentation, λ temporal is the weight coefficient that controls the timing consistency loss.

[0078] Training and reasoning: By optimizing the loss function, the gradient descent method is used to train the weights and differential geometry related parameters of 3D-CNN (such as the metric tensor g μν and its related parameters).

[0079] In the inference stage, a video frame sequence is input, and the segmentation result of each frame is generated through feature processing on the manifold and 3D-CNN to ensure the consistency of the results in time and space.

[0080] Through these steps, the brain-inspired video segmentation method based on differential geometry and 3D-CNN can better handle the complex geometric structures and continuous dynamic changes in videos and generate stable and accurate segmentation results.

[0081] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0082] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0083] The processing unit performs the various methods and processes described above. For example, in some embodiments, the method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method described above can be performed. Alternatively, in other embodiments, the CPU can be configured to perform the method in any other appropriate manner (e.g., by means of firmware).

[0084] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0085] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0086] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0087] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A brain-inspired inter-frame image segmentation method based on 3D-CNN, characterized in that: The method comprises the following steps: Step S1, obtaining a video image, and using a 3D-CNN network to extract spatiotemporal features from continuous video image frames; Step S2, applying differential geometry tools to analyze the extracted spatiotemporal features and extracting geometric features; Step S3, fusing the spatiotemporal features and the geometric features to form fused features, wherein the feature fusion includes convolution operation and convolution smoothing on the manifold; Step S4, using the fusion features, the accurate segmentation of the inter-frame images is achieved through the segmentation network.

2. The method for brain-inspired inter-frame image segmentation based on 3D-CNN according to claim 1, characterized in that: The process of extracting spatiotemporal features includes: Assume that the input video image frame sequence is Among them I t Represents the t-th frame image, T is the total number of frames; 3D convolution is applied to the video image frame sequence to extract spatial and temporal features at the same time. The formula of the 3D convolution operation is: in, represents the feature map of layer l, represents the 3D convolution kernel, b (l) is the bias term, σ(·) is the activation function, and * represents the 3D convolution operation.

3. The method for brain-inspired inter-frame image segmentation based on 3D-CNN according to claim 1, characterized in that: The process of extracting geometric features includes: The spatiotemporal feature map Assume that each frame in the video corresponds to a point on the manifold represented by X t ∈M, Where φ(·) is the embedding map, which embeds the feature map into the manifold; Use the metric tensor to describe the geometric properties of the local neighborhood: ds 2 =g μv dx μ dx v , where ds is the small distance on the manifold, g μν is the metric tensor, dx μ and dx ν are small changes in the x-axis and y-axis of the coordinate system, respectively.

4. The method for brain-inspired inter-frame image segmentation based on 3D-CNN according to claim 1, characterized in that: The convolution operation on the manifold is specifically: On a high-dimensional manifold M, define the Laplace-Beltrami operator To perform convolution operations to capture feature maps Changes on the manifold: Where |g| is the metric tensor g μν The determinant of g μν is the inverse matrix of the metric tensor, is the Laplace–Beltrami operator of the high-dimensional manifold M.

5. The method for brain-inspired inter-frame image segmentation based on 3D-CNN according to claim 1, characterized in that: The convolution smoothing on the manifold is as follows: The feature map is smoothed using the Laplace-Beltrami operator: Where η is the smoothing coefficient, It is the feature map after smoothing.

6. The method for brain-inspired inter-frame image segmentation based on 3D-CNN according to claim 1, characterized in that: The loss function of the segmentation network in step S4 combines segmentation accuracy and temporal consistency, specifically: in, is the total loss function; is the cross entropy loss of segmentation; λ temporal The weight coefficient for controlling the timing consistency loss; is the timing consistency loss; P t and P t+1 are the segmentation results of the t-th frame and the t+1-th frame, respectively, indicating the probability that each pixel belongs to a different category; T is the total number of frames.

7. The method for brain-inspired inter-frame image segmentation based on 3D-CNN according to claim 1, characterized in that: The features processed in step S3 Input to the segmentation head, and generate the classification probability of each pixel through the Softmax function: Among them, P t is the segmentation result of the tth frame, indicating the probability that each pixel belongs to a different category.

8. The method for brain-inspired inter-frame image segmentation based on 3D-CNN according to claim 1, characterized in that: The geodesic in geometry is used to describe the evolution path of the feature map on the manifold to capture the continuous changes of features in the video: Where D represents the covariant derivative, which ensures that the evolution of the feature on the manifold remains consistent; X t Represents a feature point on the manifold corresponding to the tth frame.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images