Image segmentation method of simplified continuous coupling neural network based on saliency fusion

Abstract

The invention relates to the field of image segmentation, and provides an image segmentation method of a simplified continuous coupled neural network model (SF-SCCNN) based on saliency fusion, which is used for simulating a real neuron, and a simulation experiment is realized in a matrix laboratory (MATLAB). According to the constructed neural network, the fused visual saliency serves as external stimulation of the continuous coupling neural network, so that the neural network has dynamic behaviors more similar to real neurons. It is proved that the SF-SCCNN is more sensitive to a region with high significance, and pulse output is easier to generate. The SF-SCCNN simulates the perception of a human visual system to significant features, and the recognition capability of important regions in the image is enhanced. An MATLAB segmentation experiment result shows that the segmentation method of the simplified continuous coupling neural network based on the significance fusion can simulate an HVS to realize identification and segmentation of an object.

Description

[0001] The present invention relates to the technical field of image processing, and in particular to an image segmentation method based on a continuous coupled neural network of saliency fusion. Background Art

[0002] A neural network is a computational model inspired by the structure and function of the human brain. It uses a structure similar to synaptic connections in the brain to simulate the behavior of the animal nervous system and achieve decentralized information processing. The Continuous Coupled Neural Network (CCNN) is a new artificial neural network model designed based on the workings of the primary visual cortex. This model treats the excitation of neurons as a random process, which is consistent with primary visual cortical neurons and enables it to better simulate real neurons.

[0003] Image segmentation is a key technology in the fields of computer vision and image processing and is considered one of the most important problems in image processing. Its purpose is to divide a digital image into multiple regions or objects based on the image's static and dynamic characteristics. These regions typically represent different objects or components within the image. Image segmentation is crucial for further image analysis and processing. CCNN is able to simulate the synchronous oscillations, refractory periods, and exponential decay characteristics of real neurons. In image processing tasks, the model encodes pixel intensity as the frequency of the output signal, allowing it to group pixels with similar grayscale values. This image processing method can reduce local grayscale differences in an image and compensate for small local discontinuities within the image, making it well-suited for application in the field of image segmentation.

[0004] Visual saliency refers to the ability of the human visual system to selectively focus on certain specific areas or objects when faced with a rich and diverse range of visual information. These areas or objects typically possess highly salient features, such as color contrast, unique shape, or motion changes, which make them stand out in the visual scene, attract the observer's attention, and thus be processed preferentially. Visual saliency detection technology simulates the characteristics of the human visual system, predicts human visual gaze point and eye movements, and extracts salient areas in the image, i.e., areas of human interest. It is a key image analysis technology in the field of computer vision. Continuously coupled neural networks based on saliency fusion use the fused visual saliency as external stimuli for the continuously coupled neural network, giving the neural network dynamics that are more similar to real neurons. Visual saliency is consistent with the requirements of computer vision and is of great significance to the field of image processing. Summary of the Invention

[0005] Traditional CCNNs only use pixel intensity as an external stimulus to activate neurons when processing images, without considering other salient features of the image. To make segmentation results closer to the human visual system, this paper proposes a saliency fusion-based simplified continuous coupled neural network (SF-SCCNN) image segmentation method. The main part of the neural network consists of three modules: internal activity U, dynamic threshold E, and output module Y. The mathematical model of SF-SCCNN is as follows:

[0006]

[0007] where α u and α e U ij and E ij The exponential decay factor is β, which is the connection strength between neurons to determine the output Y of adjacent neurons in the previous iteration. kl [n-1] Internal activities U in this iteration ij The share in [n] is generally between 0 and 1 (0 means no coupling connection, 1 means strong connection), V U and V E Is to adjust U ij and E ij The weighting factor of the external stimulus S' ij By fusion significance Saliecny ij and image grayscale value s ij The significance calculation method is as follows:

[0008] Given an image I, perform a 2D discrete Fourier transform on it, converting it from the spatial domain to the frequency domain, and calculate its amplitude A(f) and phase P(f). After taking the logarithm of the amplitude, we get the logarithmic amplitude spectrum L(f), i.e., L(f) = log(A(f)). Since the logarithmic curve satisfies the local linear condition, it is smoothed with a local average filter M to obtain the average spectrum AL(f), i.e., AL(f) = L(f) * M. Where * represents the convolution operation, and M is a 3*3 matrix defined as:

[0009]

[0010] Then the logarithmic spectrum of the image is subtracted from the average logarithmic spectrum to obtain the spectrum residual R(f), that is, R(f) = L(f) - AL(f), and then the spectrum residual R(f) and phase P(f) are subjected to a 2D discrete Fourier inverse transform to obtain the frequency domain saliency map S F (I). S FThe calculation method of (I) is as follows, wherein is the inverse Fourier transform.

[0011]

[0012] For spatial features, the original image I is converted from RGB color space to CIELab color space to better represent the human eye's perception of color. The point I(i,j) in the original image can be regarded as a vector in Lab color space. On this basis, the arithmetic mean of L, a, and b spaces is calculated respectively, which is recorded as Then, a Gaussian kernel with a window size of 5*5 is used to smooth the original image to remove high-frequency noise. The color feature C(x,y) at coordinate I(i,j) is calculated as follows:

[0013] C(i,j)=||I μ -I G (i,j)||

[0014] Among them I G (i, j) is the pixel value vector after Gaussian filtering of the point, and ||·|| is the Euclidean norm. The color features of each point in the image can be calculated to form the color feature map C(I) of image I. The color feature map is imported into the frequency saliency map to obtain the final saliency map saliency(I), that is,

[0015] saliency(I)=S F (I)+C(I). BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is the network structure diagram of SF-SCCNN;

[0017] Figure 2 is the image segmentation result of SF-SCCNN;

[0018] Figure 3 It is the comparison of segmentation results of natural images;

[0019] Figure 4 This is the contrast segmentation result of a fat gland image;

[0020] Figure 5 It is the segmentation result of the lesion area in the breast X-ray image. DETAILED DESCRIPTION

[0021] The present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0022] Due to the complexity of the CCNN model and the number of parameters, the main part of the neural network proposed in this paper simplifies the CCNN and introduces visual saliency, which is fused with the original image as an external stimulus to make salient areas more likely to generate pulses. The SF-SCCNN mainly consists of three modules: internal activity U, dynamic threshold E, and output module Y:

[0023]

[0024] Internal activity U of neuron (i, j) ij Affected by the connection strength β and the weight matrix W ijkl Impact. ij and dynamic threshold E ij Compare and decide output Y ij In the continuous iterative process, the internal activity U in each iteration ij (n) will be affected by the internal activities of the previous iteration U ij (n-1) and output Y ij (n-1) and the threshold E ij (n) will be based on the output Y in the previous iteration ij (n-1) is dynamically adjusted, Figure 1 The network structure diagram of SF-SCCNN is shown.

[0025] During the iteration process of SF-SCCNN, when the neurons do not receive external stimulation, there is no internal activity in the entire system, and Y ij is 0. When neuron (i, j) receives external stimulus S' ij Afterwards, it is internally modulated to generate internal activity U ij .Y ij The value range of Y is between 0 and 1. ij The value of is determined by the sigmoid activation function and is affected by U ij and E ij Impact, when U ij Greater than E ij When U ij The larger the Y ij The closer to 1; when U ij Less than E ij When U ij The smaller the Y ij The closer to 0.

[0026] In the first iteration (n=1), due to U ij 、Y ij 、E ij The initial value is 0, and the internal activity U of the entire neural network ij Only by external stimulation S' ijComposition, namely U ij [1] = S' ij Output At this time, the output Y ij The value of is between 0.5 and 1. The more significant the point, the closer the output is to 1, and the greater the probability of generating a pulse in the first iteration. For the point (i, j) that generates a pulse in the first iteration, the threshold E ij It will increase adaptively, making the output smaller in the next iteration and reducing the probability of generating a pulse in the next iteration; for the point (i, j) that did not generate a pulse in the first iteration, the threshold E ij It will adaptively decrease, causing the output to increase in the next iteration, increasing the probability of generating a pulse in the next iteration. After n iterations, the final pulse output result is generated.

[0027] Regarding frequency domain feature analysis, after many studies on image statistics, it was found that natural images are not random, but obey highly predictable distributions, among which scale invariance is the most important feature. This property indicates that after Fourier transform, the amplitude spectrum of the image obeys the following law: Where E{·} represents the average value of ·, f represents the frequency, and A(f) represents the amplitude at that frequency. This means that the average amplitude of a large number of natural images at a certain frequency is inversely proportional to that frequency. Based on this principle, further analysis of the spectral residual information contained in the image frequency-domain logarithmic spectrum reveals that the frequency-domain logarithmic amplitude spectra of different images have similar distribution trends, and the average logarithmic amplitude spectrum satisfies the local linearity condition. This similarity reflects the presence of redundant information in the image. Removing this redundant information reveals salient information that attracts the attention of the human visual system.

[0028] In a large number of different logarithmic spectra with similar shapes, smooth curves often represent redundant information, and what is noteworthy is the information that stands out from the smooth curve. Therefore, the logarithmic amplitude spectrum is used to represent the image, and the average logarithmic amplitude spectrum represents the redundant information in the image. Then, the logarithmic amplitude spectrum minus the average logarithmic amplitude spectrum can be used to obtain significant information. The discrete Fourier transform of the image can calculate its amplitude and phase. After taking the logarithm of the amplitude, the logarithmic amplitude spectrum L(f) can be obtained. At the same time, the average logarithmic amplitude spectrum AL(f) can also be calculated. The spectral residual R(f) is obtained by subtracting the average logarithmic amplitude spectrum from the logarithmic amplitude spectrum, that is, R(f) = L(f) - AL(f). The spectral residual can be used as significant information in the frequency domain, so an inverse Fourier transform is performed to calculate the frequency domain significant map. Then calculate the color feature map C(I) of the image, import the color feature map into the frequency domain saliency map, and obtain the final saliency map saliency(I), that is, saliency(I) = S F (I)+C(I).

[0029] The specific steps of image segmentation are as follows:

[0030] 1) Calculating saliency: The visual saliency of the image is calculated using the saliency detection algorithm proposed in the present invention to obtain a saliency map.

[0031] 2) Build the network model: Build the SF-SCCNN model according to the above formula. Each pixel in the image corresponds to a neuron.

[0032] 3) Set model parameters: Set system parameters α, α according to the basic properties of the image u , β, V U 、W ijkl , α e and V E . To external stimulus S' ij Perform normalization and initialize U ij 、Y ij 、E ij The value of is 0.

[0033] 4) Iteration: Iterate once and update U ij 、Y ij 、E ij value.

[0034] 5) Output pulse: Set a threshold th to obtain the pulse data of neuron (i, j) If Y ij ≥th, then otherwise, The threshold is set to th = a*max(Y), where a is a constant that can be adjusted according to the image, and max(Y) is the maximum value in Y.

[0035] 6) Generate segmentation results: Set the number of iterations and repeat steps 4 and 5 until the specified number of iterations is reached. The composed image is the segmentation result.

[0036] The present invention uses MATLAB to model the network and conduct segmentation experiments. Figure 2 The following table shows the segmentation results of the present invention. The first column shows five natural images from the Berkeley segmentation dataset and the PASCAL VOC dataset, the second column shows their grayscale images, the third and fourth columns show the saliency map and saliency fusion map calculated by the saliency detection algorithm proposed in this invention, and the last column shows the segmentation result of SF-SCCNN. To further verify the segmentation effect of the present invention, the present invention conducted comparative experiments on natural and medical images using the above image segmentation method. The experimental results are as follows:

[0037] 1) Natural Image Segmentation

[0038] The present invention conducts natural image segmentation experiments on the BSDS500 dataset, and selects one image from each of the training, validation, and test images for comparison with some advanced primary visual cortical neural networks. The experimental results are shown in the attached Figure 3 , where (a) is the original sample image; (b) is the segmentation result of OTSU; (c) is the segmentation result of SCM; (d) is the segmentation result of SPCNN; (e) is the segmentation result of CCNN; and (f) is the segmentation result of the present invention. Through visual comparison, it can be concluded that the result of the present invention retains more details and less noise while segmenting the target.

[0039] 2) Medical Image Segmentation

[0040] For medical images, the present invention conducts segmentation experiments on the DDSM dataset. The present invention uses Otsu, SCM, SPCNN, CCNN, Dense-Unet and Selective Kernel U-Net as comparison algorithms to evaluate the effectiveness of SF-SCCNN. The segmentation results of a fat gland image are shown in the attached figure. Figure 4 , where (a) is the original sample image in DDSM; (b) is the segmentation result of OTSU; (c) is the segmentation result of SCM; (d) is the segmentation result of SPCNN; (e) is the segmentation result of Selective Kernel U-Net; (f) is the segmentation result of Dense-Unet; (g) is the segmentation result of CCNN; (h) is the segmentation result of the present invention; and (i) is the segmentation result of a professional doctor. Compared with other image segmentation methods, the segmentation result of the present invention is closest to the segmentation result of a professional doctor.

[0041] In order to further evaluate the effectiveness of SF-SCCNN, the present invention uses SF-SCCNN to automatically detect and segment the lesion area in mammograms. The segmentation results are shown in the attached Figure 5 , (a)-(h) are eight lesion images and segmentation results from DDSM mammograms. The first row shows the original images, the second row shows the segmentation results from SF-SCCNN, and the last row shows the manual annotations. This paper further introduces pixel accuracy (PA), intersection over union (IoU), dice similarity coefficient (Dice), sensitivity (Sen), specificity (Spe), and precision (Pre) to qualitatively evaluate the segmentation results. The evaluation results are shown in the table below.

[0042]

[0043] Based on the above evaluation results, it can be concluded that the model has high accuracy and precision in segmenting lesions in mammograms and has great application value in medical image segmentation.

Claims

1. An image segmentation method based on simplified continuously coupled neural network with saliency fusion, characterized in that: The following procedures are included: A new Saliency Fusion-Based Simplified Continuous Coupled Neural Network (SF-SCCNN) model is proposed. It introduces the real biological feature of visual saliency, simulates the human visual system (HVS) extraction of regions of interest and the pulse behavior of biological neurons, and uses this network model for automatic image segmentation.

2. The image segmentation method based on simplified continuously coupled neural network with saliency fusion according to claim 1, characterized in that: SF-SCCNN discards the feed input F and link input L of the continuously coupled neural network, retains only the internal activity U, dynamic threshold E and output function Y to simplify the model, and uses the fused visual saliency map instead of the image grayscale value as the external stimulus S of the neural network. ' , simulate the image with low HVS contrast for information extraction.

3. According to the image segmentation method based on saliency fusion of simplified continuously coupled neural network according to claim 2, the discrete mathematical model of SF-SCCNN is as follows: where α u and α e U ij and E ij The exponential decay factor is β, which is the connection strength between neurons to determine the output Y of adjacent neurons in the previous iteration. kl [n-1] Internal activities U in this iteration ij The share in [n] is generally between 0 and 1 (0 means no coupling connection, 1 means strong connection), V U and V E Is to adjust U ij and E ij The weighting factor of the external stimulus S' ij By fusion significance Saliecny ij And the image gray value S ij composition.

4. The image segmentation method based on simplified continuous coupled neural network based on saliency fusion according to claim 2, which combines spatial domain and frequency domain features to calculate image saliency, uses the following saliency detection method: Given an image I, perform a 2D discrete Fourier transform on it, convert it from the spatial domain to the frequency domain, and calculate its amplitude A(f) and phase P(f). After taking the logarithm of the amplitude, we get the logarithmic amplitude spectrum L(f), that is, L(f) = log(A(f)). Since the logarithmic curve meets the local linear condition, it is smoothed with a local average filter M to obtain the average spectrum AL(f), that is, AL(f) = L(f)*M, where * represents the convolution operation and M is a 3*3 matrix defined as: Then the logarithmic spectrum of the image is subtracted from the average logarithmic spectrum to obtain the spectrum residual R(f), that is, R(f) = L(f) - AL(f), and then the spectrum residual R(f) and phase P(f) are subjected to a 2D discrete Fourier inverse transform to obtain the frequency domain saliency map S F (I), S F The calculation method of (I) is as follows, wherein is the inverse Fourier transform, For spatial features, the original image I is converted from RGB color space to CIELab color space to better represent the human eye's perception of color. The point I(i,j) in the original image can be regarded as a vector in Lab color space. On this basis, the arithmetic mean of L, a, and b spaces is calculated respectively, which is recorded as Then, a Gaussian kernel with a window size of 5*5 is used to smooth the original image to eliminate high-frequency noise. The color feature C(i, j) at the coordinate I(i, j) is calculated as follows: C(i,j)=||I μ -I G (i,j)||, Among them I G (i, j) is the pixel value vector after Gaussian filtering of the point, ||·|| is the Euclidean norm, and the color features of each point in the image can be calculated to form the color feature map C(I) of image I. The color feature map is imported into the frequency saliency map to obtain the final saliency map saliency(I), that is, saliency(I)=S F (I)+C(I)。 5. The image segmentation method based on simplified continuously coupled neural network with saliency fusion according to claim 2, wherein the specific steps for image segmentation tasks are as follows: 1) Calculating saliency: Calculating the visual saliency of the image according to the saliency detection algorithm described in claim 4 to obtain a saliency map; 2) Build the network model: Build the SF-SCCNN model according to the above formula, where each pixel in the image corresponds to a neuron; 3) Set model parameters: Set system parameters α, α according to the basic properties of the image u , β, V U 、W ijkl , α e and V E , to external stimulus S' ij Perform normalization and initialize U ij 、Y ij 、E ij The value of is 0; 4) Iteration: Iterate once and update U ij 、Y ij 、E ij The value of 5) Output pulse: Set a threshold th to obtain the pulse data of neuron (i, j) If Y ij ≥th, then otherwise, The threshold is set as th = a*max(Y), where a is a constant that can be adjusted according to the image, and max(Y) is the maximum value in Y; 6) Generate segmentation results: Set the number of iterations and repeat steps 4 and 5 until the specified number of iterations is reached. The composed image is the segmentation result.

Images