Adaptive layer segmentation method based on deep neural network driving

Abstract

The invention discloses an adaptive layer segmentation method based on deep neural network driving, and the method comprises the steps: obtaining multi-scene image data in a database, carrying out the preprocessing of the multi-scene image data, and obtaining initial multi-scene image data; establishing a basic neural network model, wherein the basic neural network model at least comprises a plurality of convolutional layers, pooling layers and full connection layers; designing a multi-modal feature fusion module in a middle layer of the basic neural network model, and inserting an attention module behind a convolutional layer of the basic neural network model; constructing a pyramid structure by using convolution kernels of different sizes to obtain a target neural network model; inputting the initial multi-scene image data into a target neural network model for training; analyzing the features of the image through the target neural network model, carrying out the layer segmentation of the image, carrying out the morphological processing of a preliminary image segmentation result, removing the noise and small holes in the segmentation result, and obtaining a self-adaptive layer segmentation result. And the segmentation accuracy is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an adaptive layer segmentation method driven by a deep neural network. Background Art

[0002] In image processing, layer segmentation is a key task with a wide range of application scenarios, such as computer vision, image editing, and medical image analysis. Traditional layer segmentation methods often rely on manually set parameters or are based on fixed rules, which are difficult to adapt to complex and changing image content and scenarios. With the development of deep learning technology, deep neural networks have shown powerful capabilities in image processing, but existing layer segmentation methods based on deep neural networks still have certain limitations in terms of accuracy, adaptability, and efficiency. For example, some methods have unstable segmentation effects when facing images with different resolutions, lighting conditions, and texture features, and cannot be adaptively adjusted according to the specific characteristics of the image; in addition, some methods have high computational complexity, resulting in slow processing speed and cannot meet application scenarios with high real-time requirements. Therefore, a new adaptive layer segmentation method driven by deep neural networks is needed to solve these problems. Summary of the invention

[0003] The purpose of the present invention is to solve the above problems and design an adaptive layer segmentation method driven by a deep neural network.

[0004] To achieve the above object, the technical solution of the present invention is that, further, in the above-mentioned adaptive layer segmentation method based on deep neural network driving, the adaptive layer segmentation method comprises the following steps:

[0005] Acquire multi-scene image data in a database, and preprocess the multi-scene image data to obtain initial multi-scene image data;

[0006] Establishing a basic neural network model, wherein the basic neural network model includes at least a plurality of convolutional layers, a pooling layer, and a fully connected layer;

[0007] In the middle layer of the basic neural network model, a multimodal feature fusion module is designed, and an attention module is inserted after the convolution layer of the basic neural network model; a pyramid structure is constructed using convolution kernels of different sizes to obtain a target neural network model;

[0008] Inputting the initial multi-scene image data into the target neural network model for training;

[0009] Analyze the features of the image through the target neural network model, perform layer segmentation on the image, and obtain a preliminary image segmentation result;

[0010] The preliminary image segmentation result is subjected to morphological processing to remove noise and small holes in the segmentation result, thereby obtaining an adaptive layer segmentation result.

[0011] Furthermore, in the above-mentioned adaptive layer segmentation method driven by a deep neural network, the step of acquiring multi-scene image data from a database and preprocessing the multi-scene image data to obtain initial multi-scene image data includes:

[0012] Acquire multi-scene image data in a database, wherein the multi-scene image data at least includes image data sets of different scenes, objects and backgrounds;

[0013] Acquire depth information, an optical flow map, and a modal image in the image data set through an optical flow algorithm and a stereo vision algorithm to obtain processed image data;

[0014] Normalizing the pixel values ​​of the processed image data to the interval [0, 1] to obtain normalized image data;

[0015] The normalized image data is randomly rotated, scaled, cropped, and flipped to obtain initial multi-scene image data.

[0016] Furthermore, in the above-mentioned adaptive layer segmentation method driven by a deep neural network, a multimodal feature fusion module is designed in the middle layer of the basic neural network model, and an attention module is inserted after the convolution layer of the basic neural network model; a pyramid structure is constructed using convolution kernels of different sizes to obtain a target neural network model, including:

[0017] A multimodal feature fusion module is designed at the starting layer of the network. The RGB preliminary visual features of the RGB image features are extracted through the convolutional layer;

[0018] Use convolutional layers to map the depth features to spatial dimensions that match the RGB preliminary visual features to obtain depth information;

[0019] Use the fusion layer to perform weighted summation of RGB preliminary visual features, depth information, and optical flow map features;

[0020] An attention module is established based on the combination of channel attention and spatial attention, and the attention module is inserted after the convolutional layer;

[0021] Channel attention compresses the feature map through global average pooling and global maximum pooling, and calculates the weight of each channel through the MLP multi-layer perceptron;

[0022] Spatial attention generates a spatial weight map by performing convolution and pooling operations on the feature map;

[0023] Introducing deformable convolutional layers at appropriate locations in the network, the network automatically adjusts the offset of the convolution kernel according to the content of the input image and captures changes in the target shape;

[0024] Convolution kernels of different sizes are used to construct a pyramid structure to obtain feature maps at different resolutions. An upsampling fusion strategy is used to combine multi-scale features to obtain the target neural network model.

[0025] Furthermore, in the above-mentioned adaptive layer segmentation method driven by a deep neural network, the inputting of the initial multi-scene image data into the target neural network model for training includes:

[0026] An adaptive loss function is established by combining cross entropy loss and Dice loss. Dice loss is used to measure the degree of regional overlap between the predicted result and the true label.

[0027] According to the different difficulties of the segmentation task, the weight factors of the cross entropy loss and Dice loss are dynamically adjusted.

[0028] Using SGD stochastic gradient descent as the optimization algorithm of the target neural network model, the parameters of the network are continuously adjusted to minimize the loss function;

[0029] Save the network parameters of the model, evaluate the network using the validation set, and select the best model parameters based on the evaluation results.

[0030] Furthermore, in the above-mentioned adaptive layer segmentation method driven by a deep neural network, the target neural network model is used to analyze the characteristics of the image, and the image is segmented by layers to obtain a preliminary image segmentation result, including:

[0031] Automatically adjusting its internal structure according to input image characteristics, wherein the input image characteristics include at least resolution, complexity, and object size;

[0032] The input image is judged by an evaluation network with pre-set rules. If the image resolution is high and contains complex target structures, the number of channels of the convolution layer is increased. If the image is simple and has high real-time requirements, the number of channels of the convolution layer is reduced.

[0033] The image features are analyzed through an adaptive attention mechanism, and the image is segmented layer by layer based on the learned knowledge to obtain preliminary image segmentation results.

[0034] Furthermore, in the above-mentioned adaptive layer segmentation method driven by a deep neural network, the preliminary image segmentation result is subjected to morphological processing to remove noise and small holes in the segmentation result to obtain an adaptive layer segmentation result, including:

[0035] Performing morphological processing on the preliminary image segmentation result, wherein the morphological processing at least includes corrosion, expansion, etc.;

[0036] Remove noise and small holes in the segmentation results, and merge adjacent areas with similar semantics but mistakenly divided into different layers in the initial segmentation into one layer based on semantic information;

[0037] For the case where the objects semantically belong to the same object but the segmentation boundary is inaccurate, the boundary is adjusted according to the semantic contour to obtain the adaptive layer segmentation result.

[0038] Furthermore, a system for an adaptive layer segmentation method driven by a deep neural network is implemented, and the system includes the following modules:

[0039] A data processing module, used to obtain multi-scene image data in a database, pre-process the multi-scene image data, and obtain initial multi-scene image data;

[0040] A model building module, used to build a basic neural network model, wherein the basic neural network model includes at least a plurality of convolutional layers, a pooling layer and a fully connected layer;

[0041] A model improvement module is used to design a multimodal feature fusion module in the middle layer of the basic neural network model, insert an attention module after the convolution layer of the basic neural network model; and construct a pyramid structure using convolution kernels of different sizes to obtain a target neural network model;

[0042] A data input module, used for inputting the initial multi-scene image data into the target neural network model for training;

[0043] A preliminary segmentation module is used to analyze the characteristics of the image through the target neural network model, perform layer segmentation on the image, and obtain a preliminary image segmentation result;

[0044] The target segmentation module is used to perform morphological processing on the preliminary image segmentation result, remove noise and small holes in the segmentation result, and obtain an adaptive layer segmentation result.

[0045] Furthermore, a system for an adaptive layer segmentation method driven by a deep neural network is implemented, and the system includes the following submodules:

[0046] An acquisition submodule, used to acquire multi-scene image data in a database, wherein the multi-scene image data at least includes image data sets of different scenes, objects and backgrounds;

[0047] A processing submodule, used to obtain depth information, an optical flow map, and a modal image in the image data set through an optical flow algorithm and a stereo vision algorithm to obtain processed image data;

[0048] A normalization submodule, used to normalize the pixel values ​​of the processed image data to the interval [0, 1] to obtain normalized image data;

[0049] A submodule is obtained, which is used to randomly rotate, scale, crop and flip the normalized image data to obtain initial multi-scene image data.

[0050] Furthermore, a system for an adaptive layer segmentation method driven by a deep neural network is implemented, and the system includes the following submodules:

[0051] An adjustment submodule, configured to automatically adjust its internal structure according to input image characteristics, wherein the input image characteristics include at least resolution, complexity, and object size;

[0052] The evaluation submodule is used to judge the input image through the evaluation network with pre-set rules. If the image resolution is high and contains complex target structures, the number of channels of the convolution layer is increased. If the image is simple and has high real-time requirements, the number of channels of the convolution layer is reduced.

[0053] The segmentation submodule is used to analyze the features of the image through an adaptive attention mechanism and perform layer segmentation on the image based on the learned knowledge to obtain preliminary image segmentation results.

[0054] The beneficial effects are as follows: by acquiring multi-scene image data in a database, the multi-scene image data is preprocessed to obtain initial multi-scene image data; a basic neural network model is established, and the basic neural network model includes at least multiple convolution layers, pooling layers and fully connected layers; in the middle layer of the basic neural network model, a multimodal feature fusion module is designed, and an attention module is inserted after the convolution layer of the basic neural network model; a pyramid structure is constructed using convolution kernels of different sizes to obtain a target neural network model; the initial multi-scene image data is input into the target neural network model for training; the image features are analyzed by the target neural network model, and the image is layer segmented to obtain a preliminary image segmentation result; the preliminary image segmentation result is morphologically processed to remove noise and small holes in the segmentation result to obtain an adaptive layer segmentation result. The segmentation accuracy is greatly improved, especially when complex scenes and objects change. In terms of robustness, multimodal complementarity and adaptive technology enable it to adapt to various environments and image changes. In terms of efficiency, dynamic adjustment of network structure, algorithm optimization and online learning mechanism improve processing speed and system operation efficiency. It is widely applicable to multiple fields and has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0056] Figure 1 Schematic diagram of a first embodiment of an adaptive layer segmentation method driven by a deep neural network in an embodiment of the present invention;

[0057] Figure 2 Schematic diagram of a second embodiment of an adaptive layer segmentation method driven by a deep neural network in an embodiment of the present invention;

[0058] Figure 3 Schematic diagram of a third embodiment of an adaptive layer segmentation method driven by a deep neural network in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0061] The present invention will be described in detail below in conjunction with the accompanying drawings. Figure 1 As shown, an adaptive layer segmentation method based on deep neural network driving includes the following steps:

[0062] Step 101: Acquire multi-scene image data in a database, pre-process the multi-scene image data, and obtain initial multi-scene image data;

[0063] Specifically, in this embodiment, multi-scene image data is obtained from a database, and the multi-scene image data includes at least image data sets of different scenes, objects and backgrounds; depth information, optical flow maps and modal images in the image data set are obtained through an optical flow algorithm and a stereo vision algorithm to obtain processed image data; pixel values ​​of the processed image data are normalized to the [0,1] interval to obtain normalized image data; the normalized image data is randomly rotated, scaled, cropped and flipped to obtain initial multi-scene image data.

[0064] Step 102: Establish a basic neural network model, which includes at least a plurality of convolutional layers, a pooling layer, and a fully connected layer;

[0065] Step 103: In the middle layer of the basic neural network model, a multimodal feature fusion module is designed, and an attention module is inserted after the convolution layer of the basic neural network model; a pyramid structure is constructed using convolution kernels of different sizes to obtain a target neural network model;

[0066] Specifically, in this embodiment, a multimodal feature fusion module is designed at the starting layer of the network. The RGB preliminary visual features of the RGB image features are extracted through the convolution layer; the depth features are mapped to the spatial dimension matching the RGB preliminary visual features using the convolution layer to obtain the depth information; the fusion layer is used to weight the sum of the RGB preliminary visual features, the depth information and the optical flow map features; the attention module is established based on the combination of channel attention and spatial attention, and the attention module is inserted after the convolution layer; the channel attention compresses the feature map through global average pooling and global maximum pooling, and the weight of each channel is calculated through the MLP multi-layer perceptron; the spatial attention generates a spatial weight map by performing convolution and pooling operations on the feature map; a deformable convolution layer is introduced at an appropriate position of the network, and the network automatically adjusts the offset of the convolution kernel according to the content of the input image and captures the changes in the target shape; convolution kernels of different sizes are used to construct a pyramid structure, and feature maps at different resolutions are obtained. The multi-scale features are combined using an upsampling fusion strategy to obtain the target neural network model.

[0067] Step 104: input the initial multi-scene image data into the target neural network model for training;

[0068] Specifically, in this embodiment, an adaptive loss function is established by combining cross entropy loss and Dice loss, and Dice loss is used to measure the degree of regional overlap between the predicted result and the true label; according to the different difficulties of the segmentation task, the weight factors of the cross entropy loss and the Dice loss are dynamically adjusted, and SGD stochastic gradient descent is used as the optimization algorithm of the target neural network model to continuously adjust the parameters of the network to minimize the loss function; the network parameters of the model are saved, and the network is evaluated using a validation set, and the optimal model parameters are selected according to the evaluation results.

[0069] Step 105: Analyze the features of the image through the target neural network model, perform layer segmentation on the image, and obtain a preliminary image segmentation result;

[0070] Specifically, in this embodiment, the internal structure is automatically adjusted according to the input image characteristics, and the input image characteristics include at least resolution, complexity, and target size; the input image is judged by an evaluation network with pre-set rules, if the image resolution is high and contains a complex target structure, the number of channels of the convolution layer is increased, if the image is simple and has high real-time requirements, the number of channels of the convolution layer is reduced; the image features are analyzed through an adaptive attention mechanism, and the image is segmented into layers according to the learned knowledge to obtain a preliminary image segmentation result.

[0071] Step 106: Perform morphological processing on the preliminary image segmentation result to remove noise and small holes in the segmentation result to obtain an adaptive layer segmentation result.

[0072] Specifically, in this embodiment, morphological processing is performed on the preliminary image segmentation result, and the morphological processing includes at least corrosion, dilation, etc.; noise and small holes in the segmentation result are removed, and adjacent areas with similar semantics but mistakenly divided into different layers in the preliminary segmentation are merged into one layer according to the semantic information; for situations where they belong to the same object semantically but the segmentation boundary is inaccurate, the boundary is adjusted according to the semantic contour to obtain an adaptive layer segmentation result.

[0073] The beneficial effect is that by obtaining multi-scene image data in the database, the multi-scene image data is preprocessed to obtain initial multi-scene image data; a basic neural network model is established, and the basic neural network model includes at least multiple convolution layers, pooling layers and fully connected layers; in the middle layer of the basic neural network model, a multimodal feature fusion module is designed, and an attention module is inserted after the convolution layer of the basic neural network model; a pyramid structure is constructed using convolution kernels of different sizes to obtain a target neural network model; the initial multi-scene image data is input into the target neural network model for training; the image features are analyzed by the target neural network model, and the image is layer segmented to obtain a preliminary image segmentation result; the preliminary image segmentation result is morphologically processed to remove noise and small holes in the segmentation result to obtain an adaptive layer segmentation result. The segmentation accuracy is greatly improved, especially when complex scenes and objects change. In terms of robustness, multimodal complementarity and adaptive technology enable it to adapt to various environments and image changes. In terms of efficiency, dynamic adjustment of network structure, algorithm optimization and online learning mechanism improve processing speed and system operation efficiency. It is widely applicable to multiple fields and has strong versatility.

[0074] Specifically, the specific processing flow in this embodiment is as follows:

[0075] Data preprocessing

[0076] Collect a large number of image datasets containing different scenes, objects, and backgrounds, and simultaneously obtain the corresponding depth information, optical flow maps, and other related modal data (if applicable). For example, for some scenes containing moving objects, the optical flow map can be calculated by the optical flow algorithm to capture the motion characteristics of the object; for images with depth information, a depth camera or a stereo vision algorithm can be used to obtain a depth map to describe the spatial position relationship of the object.

[0077] The RGB image is normalized to normalize the pixel values ​​to the interval [0,1] to improve the training efficiency and stability of the deep neural network. The depth information and other modal data are also normalized accordingly to make their value range meet the requirements of the network input.

[0078] Data enhancement techniques such as random rotation, scaling, cropping and flipping are used to increase data diversity and prevent overfitting. In addition, image color space transformation enhancement is innovatively introduced, such as converting RGB images to HSV color space and then performing random adjustments, which enriches the feature representation of the image, enables the network to learn more comprehensive image information, and improves the adaptability to images under different lighting and color conditions.

[0079] Deep neural network architecture design

[0080] Build a deep neural network infrastructure with multiple convolutional layers, pooling layers, and fully connected layers.

[0081] Multimodal feature fusion: Design a multimodal feature fusion module at the starting layer or appropriate intermediate layer of the network. For RGB image features, preliminary visual features are extracted through the convolution layer. For depth information, a dedicated convolution layer or other appropriate processing method can be used to map the depth features to a spatial dimension that matches the RGB features. Optical flow map features can be extracted and fused in a manner similar to that of processing time series data. For example, a fusion layer is used to combine RGB features, depth features, and optical flow map features in a certain way, such as concatenation or weighted summation, to make full use of the complementary information of different modal data and improve the robustness and accuracy of segmentation.

[0082] Adaptive attention mechanism: Introduce adaptive attention modules in multiple layers of deep neural networks, such as inserting attention modules after convolutional layers or before pooling layers. This module can adopt a combination of channel attention and spatial attention. Channel attention compresses the feature map through global average pooling and global maximum pooling, and then calculates the weight of each channel through a multi-layer perceptron (MLP) to highlight important channel features. Spatial attention generates a spatial weight map by performing convolution and pooling operations on the feature map, enabling the network to adaptively adjust the weights according to different regions of the input image, highlighting important features and suppressing irrelevant background interference.

[0083] Deformable convolution: Deformable convolution layers are introduced at appropriate locations in the network to address challenges caused by target deformation. Deformable convolution allows the convolution kernel to shift in space, and its offset is learned. During the training process, the network automatically adjusts the offset of the convolution kernel according to the content of the input image, thereby better capturing changes in target shape and improving the ability to segment targets of different shapes.

[0084] Multi-scale feature extraction and fusion: Use pyramid structures or multi-scale convolution modules to extract features of different scales. For example, use convolution kernels of different sizes or pooling operations of different step lengths to build a pyramid structure to obtain feature maps at different resolutions. Then use effective fusion strategies to combine these multi-scale features, such as upsampling and splicing operations, to more comprehensively capture the information of the target object and adapt to the segmentation requirements of objects of different sizes.

[0085] Use residual connections and dense skip connections to alleviate the gradient vanishing problem during deep neural network training, strengthen feature propagation and reuse, and improve network performance and its ability to capture details.

[0086] Network training

[0087] The deep neural network is trained using the preprocessed multimodal image dataset.

[0088] Adaptive loss function: Design an adaptive loss function, such as a combination of cross entropy loss and Dice loss. Dice loss is used to measure the degree of regional overlap between the predicted result and the true label, and has a good effect on unbalanced segmentation tasks. According to the different difficulties of the segmentation task, the weight factors of cross entropy loss and Dice loss are dynamically adjusted. For categories or regions that are difficult to segment, the corresponding loss weights are increased, so as to balance the segmentation performance of each category during the training process, so that the network pays more attention to the learning of difficult samples.

[0089] Using optimization algorithms, such as stochastic gradient descent (SGD) or Adam, the network parameters are continuously adjusted to minimize the loss function, so that the network can learn effective layer segmentation features.

[0090] During the training process, the network parameters are saved regularly, and the network is evaluated using the validation set, and the best model parameters are selected based on the evaluation results. At the same time, a dynamic learning rate adjustment strategy is adopted to automatically adjust the learning rate according to the change of loss during the training process to improve the efficiency and stability of training.

[0091] Adaptive layer segmentation

[0092] For the input image to be segmented (including the corresponding multimodal data, if any), it is first input into the trained deep neural network.

[0093] Dynamically adjust the network structure: Develop a method to dynamically adjust the network structure, and automatically adjust its internal structure according to the characteristics of the input image (such as resolution, complexity, object size, etc.) when the network is running. For example, through pre-set rules or a small evaluation network, the input image is judged. If the image resolution is high and contains complex target structures, some convolutional layers can be added or the number of channels of the existing convolutional layers can be increased to extract richer features; if the image is relatively simple and has high real-time requirements, some unnecessary convolutional layers can be reduced or the number of channels can be reduced to optimize resource utilization and improve segmentation speed.

[0094] The network automatically analyzes the features of the image through adaptive mechanisms (including adaptive attention mechanisms, deformable convolutions, etc.), and segments the image layer by layer based on the learned knowledge to obtain preliminary segmentation results.

[0095] The preliminary segmentation results are post-processed, such as morphological operations (erosion, dilation, etc.), to remove noise and small holes in the segmentation results, further optimize the segmentation effect, and obtain the final adaptive layer segmentation results. An innovative post-processing optimization method based on image semantics is applied. This method uses a pre-trained image semantic segmentation model to obtain the semantic information of the input image, and then corrects and refines the segmentation results based on the semantic information. For example, for adjacent areas with similar semantics but mistakenly divided into different layers in the preliminary segmentation, they are merged into one layer based on the semantic information; for situations where the segmentation boundaries are inaccurate but belong to the same object semantically, the boundaries are adjusted based on the semantic contours to make them more consistent with the actual shape of the object.

[0096] See also Figure 2 In an adaptive layer segmentation method driven by a deep neural network, obtaining multi-scene image data in a database, preprocessing the multi-scene image data, and obtaining initial multi-scene image data include the following steps:

[0097] Step 201: Acquire multi-scene image data in a database, where the multi-scene image data at least includes image data sets of different scenes, objects, and backgrounds;

[0098] Step 202: Obtain depth information, an optical flow map, and a modal image in the image data set through an optical flow algorithm and a stereo vision algorithm to obtain processed image data;

[0099] Step 203, normalizing the pixel values ​​of the processed image data to the interval [0, 1] to obtain normalized image data;

[0100] Step 204: randomly rotate, scale, crop and flip the normalized image data to obtain initial multi-scene image data.

[0101] See also Figure 3 In an adaptive layer segmentation method driven by a deep neural network, the initial multi-scene image data is input into a target neural network model for training, which includes the following steps:

[0102] Step 301: Establish an adaptive loss function by combining cross entropy loss and Dice loss. Dice loss is used to measure the degree of regional overlap between the predicted result and the true label.

[0103] Step 302: dynamically adjust the weight factors of the cross entropy loss and the Dice loss according to the different difficulties of the segmentation task.

[0104] Step 303: Use SGD stochastic gradient descent as the optimization algorithm of the target neural network model, and continuously adjust the parameters of the network to minimize the loss function;

[0105] Step 304: Save the network parameters of the model, evaluate the network using the validation set, and select the best model parameters based on the evaluation results.

[0106] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An adaptive layer segmentation method driven by deep neural network, characterized in that: The adaptive layer segmentation method comprises the following steps: Acquire multi-scene image data from a database, and preprocess the multi-scene image data to obtain initial multi-scene image data; Establishing a basic neural network model, wherein the basic neural network model includes at least a plurality of convolutional layers, pooling layers, and fully connected layers; A multimodal feature fusion module is designed in the middle layer of the basic neural network model, and an attention module is inserted after the convolution layer of the basic neural network model; a pyramid structure is constructed using convolution kernels of different sizes to obtain a target neural network model; Inputting the initial multi-scene image data into the target neural network model for training; Analyze the characteristics of the image through the target neural network model, perform layer segmentation on the image, and obtain preliminary image segmentation results; Morphological processing is performed on the preliminary image segmentation result to remove noise and small holes in the segmentation result, thereby obtaining an adaptive layer segmentation result.

2. The method for adaptive layer segmentation based on deep neural network driving according to claim 1, characterized in that: The acquiring of multi-scene image data from a database and preprocessing of the multi-scene image data to obtain initial multi-scene image data include: Acquire multi-scene image data from a database, wherein the multi-scene image data includes at least image datasets of different scenes, objects, and backgrounds; Obtaining depth information, an optical flow map, and a modal image in the image data set through an optical flow algorithm and a stereo vision algorithm to obtain processed image data; Normalizing pixel values ​​of the processed image data to the interval [0, 1] to obtain normalized image data; The normalized image data is randomly rotated, scaled, cropped, and flipped to obtain initial multi-scene image data.

3. The method for adaptive layer segmentation based on deep neural network driving according to claim 1, characterized in that: In the middle layer of the basic neural network model, a multimodal feature fusion module is designed, and an attention module is inserted after the convolution layer of the basic neural network model; Convolution kernels of different sizes are used to build a pyramid structure to obtain the target neural network model, including: A multimodal feature fusion module is designed at the starting layer of the network. The RGB preliminary visual features of the RGB image features are extracted through the convolutional layer; Use the convolutional layer to map the depth features to the spatial dimension that matches the RGB preliminary visual features to obtain depth information; Use the fusion layer to perform weighted summation of RGB preliminary visual features, depth information, and optical flow map features; An attention module is established based on the combination of channel attention and spatial attention, and the attention module is inserted after the convolutional layer; Channel attention compresses the feature map through global average pooling and global maximum pooling, and calculates the weight of each channel through the MLP multi-layer perceptron; Spatial attention generates a spatial weight map by performing convolution and pooling operations on the feature map; By introducing a deformable convolution layer at the appropriate position of the network, the network automatically adjusts the offset of the convolution kernel according to the content of the input image and captures the changes in the target shape; Convolution kernels of different sizes are used to construct a pyramid structure to obtain feature maps at different resolutions. An upsampling fusion strategy is used to combine multi-scale features to obtain the target neural network model.

4. The method for adaptive layer segmentation based on deep neural network driving according to claim 1, characterized in that: Inputting the initial multi-scene image data into the target neural network model for training includes: An adaptive loss function is established by combining cross entropy loss and Dice loss. Dice loss is used to measure the degree of regional overlap between the predicted results and the true labels. According to the different difficulties of the segmentation task, the weight factors of cross entropy loss and Dice loss are dynamically adjusted. SGD stochastic gradient descent is used as the optimization algorithm of the target neural network model, and the parameters of the network are continuously adjusted to minimize the loss function; Save the network parameters of the model, evaluate the network using the validation set, and select the best model parameters based on the evaluation results.

5. The method for adaptive layer segmentation based on deep neural network driving according to claim 1, characterized in that: The step of analyzing the features of the image by the target neural network model and performing layer segmentation on the image to obtain a preliminary image segmentation result includes: Automatically adjust its internal structure based on input image characteristics, wherein the input image characteristics include at least resolution, complexity, and object size; The input image is judged by an evaluation network with pre-set rules. If the image resolution is high and contains complex target structures, the number of channels in the convolution layer is increased. If the image is simple and has high real-time requirements, the number of channels in the convolution layer is reduced. The image features are analyzed through the adaptive attention mechanism, and the image is segmented layer by layer based on the learned knowledge to obtain preliminary image segmentation results.

6. The method for adaptive layer segmentation based on deep neural network driving according to claim 1, characterized in that: The step of performing morphological processing on the preliminary image segmentation result to remove noise and small holes in the segmentation result to obtain an adaptive layer segmentation result includes: Performing morphological processing on the preliminary image segmentation result, wherein the morphological processing at least includes corrosion, expansion, etc.; Remove noise and small holes in the segmentation results, and merge adjacent areas with similar semantics but mistakenly divided into different layers in the initial segmentation into one layer based on semantic information; For the cases where the objects belong to the same semantics but the segmentation boundaries are inaccurate, the boundaries are adjusted according to the semantic contours to obtain the adaptive layer segmentation results.

7. A system for implementing the adaptive layer segmentation method based on deep neural network driving as claimed in claim 1, characterized in that: The system includes the following modules: A data processing module is used to obtain multi-scene image data from a database, pre-process the multi-scene image data, and obtain initial multi-scene image data; A model building module is used to build a basic neural network model, wherein the basic neural network model includes at least multiple convolutional layers, pooling layers and fully connected layers; A model improvement module is used to design a multimodal feature fusion module in the middle layer of the basic neural network model and insert an attention module after the convolution layer of the basic neural network model; Use convolution kernels of different sizes to build a pyramid structure and obtain the target neural network model; A data input module, configured to input the initial multi-scene image data into the target neural network model for training; A preliminary segmentation module is used to analyze the characteristics of the image through the target neural network model, perform layer segmentation on the image, and obtain a preliminary image segmentation result; The target segmentation module is used to perform morphological processing on the preliminary image segmentation result, remove noise and small holes in the segmentation result, and obtain an adaptive layer segmentation result.

8. A system for implementing the deep neural network-driven adaptive layer segmentation method according to claim 1, characterized in that: The system includes the following submodules: An acquisition submodule, configured to acquire multi-scene image data from a database, wherein the multi-scene image data includes at least image datasets of different scenes, objects, and backgrounds; a processing submodule, configured to obtain depth information, an optical flow map, and a modal image from the image dataset through an optical flow algorithm and a stereo vision algorithm to obtain processed image data; a normalization submodule, configured to normalize the pixel values ​​of the processed image data to the interval [0, 1] to obtain normalized image data; A submodule is obtained, which is used to randomly rotate, scale, crop and flip the normalized image data to obtain initial multi-scene image data.

9. A system for implementing the deep neural network-driven adaptive layer segmentation method according to claim 1, characterized in that: The system includes the following submodules: an adjustment submodule, configured to automatically adjust its internal structure according to input image characteristics, wherein the input image characteristics include at least resolution, complexity, and object size; The evaluation submodule is used to judge the input image through an evaluation network with pre-set rules. If the image resolution is high and contains complex target structures, the number of channels in the convolution layer is increased. If the image is simple and has high real-time requirements, the number of channels in the convolution layer is reduced. The segmentation submodule is used to analyze the features of the image through an adaptive attention mechanism and perform layer segmentation on the image based on the learned knowledge to obtain preliminary image segmentation results.

Images