Image segmentation method and device, equipment and storage medium
By using a combination of multi-feature extraction modules, attention networks and state space modules in the image segmentation method, the deficiencies in multi-level information integration and different-scale object processing in the existing technology are solved, and more stable and accurate image segmentation is achieved. Effect.
Patent Information
- Application Number
- CN202411343168.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2025-01-21
AI Technical Summary
The existing image segmentation methods have shortcomings in integrating multi-level information and processing objects at different scales, resulting in unstable segmentation effect and deviation in accuracy.
An image segmentation method is adopted, by obtaining the sample image data and the image to be processed, inputting the first feature extraction module and the second feature extraction module respectively for feature extraction, combining the attention network for feature fusion and global feature extraction, and using the state space module Extract hidden state features and output the target feature image through the fully connected layer to train and segment the image segmentation model.
It improves the image segmentation model's ability to understand complex scenes, enhances its ability to capture the global context information of the image, and improves the stability and accuracy of the segmentation effect.
Smart Images
Figure CN119339375A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent driving, and in particular to an image segmentation method, device, equipment and storage medium. Background Art
[0002] Image segmentation is a core technology in the field of computer vision and image processing. It uses complex algorithms to accurately separate the target object from the image background. Its goal is to identify and mark specific objects or boundaries from the image for further image analysis or decision-making. It is now widely used in medical imaging, autonomous driving, video surveillance, and geographic information systems. It usually involves methods such as feature extraction, edge detection, cluster analysis, and threshold processing. With the development of deep learning, machine learning methods based on convolutional neural networks are also used to improve segmentation accuracy and achieve more accurate and robust image segmentation effects.
[0003] However, in the existing image segmentation process, although the convolutional neural network can achieve excellent local feature extraction, excessive focus on local details may lead to a lack of understanding of the overall object, and when faced with objects of different scales, its ability to integrate multi-level information is insufficient, resulting in accuracy deviations in the segmented image and unstable segmentation effects in the image segmentation process. Summary of the invention
[0004] The purpose of the embodiments of the present invention is to provide an image segmentation method, apparatus, device and storage medium to solve the above technical problems.
[0005] The present invention provides an image segmentation method, which includes: acquiring sample image data and an image to be processed; inputting the sample image data into a first feature extraction module for first feature extraction to obtain first feature information, and inputting the sample image data into a second feature extraction module for second feature extraction to obtain second feature information; inputting the first feature information and the second feature information into an attention network for third feature extraction to obtain a feature image to be processed; performing hidden state feature extraction on the feature image to be processed based on a state space module, and outputting a target feature image through a fully connected layer; training an image segmentation model based on the target feature image to obtain a trained image segmentation model, the image segmentation model including the first feature extraction module, the second feature extraction module, the attention network, the state space module and the fully connected layer; inputting the image to be processed into the trained image segmentation model to obtain a target segmented image.
[0006] In one embodiment of the present invention, the first feature extraction module includes a pyramid simplification submodule and a multi-head attention submodule. The sample image data is input into the first feature extraction module for first feature extraction, and obtaining the first feature information includes: inputting the sample image data into the pyramid simplification submodule for multi-level feature extraction to obtain a first initial feature image set, and the pyramid simplification submodule includes a plurality of stepped convolution subnetworks with different expansion rates; inputting the first initial feature image into the multi-head attention submodule for hierarchical feature enhancement to obtain a plurality of hierarchical enhanced feature images, and splicing the plurality of hierarchical enhanced feature images to obtain the first feature information, and the output end of the pyramid simplification submodule is connected to the input end of the multi-head attention submodule.
[0007] In one embodiment of the present invention, the process of inputting the sample image data into the pyramid reduction submodule for multi-level feature extraction is characterized as follows:
[0008]
[0009] Among them, f i m is the mth result after being processed by the i-th pyramid reduction module, PRM i is the ith pyramid reduction module, f i is the i-th image input, Cat() is the fusion function, Conv ij () is the jth convolutional layer in the i-th pyramid reduction module, S i is the i-th expansion rate set, s ij is the jth expansion rate in the i-th expansion rate set, R is the reduction ratio set, r i is the i-th reduction ratio in the reduction ratio set;
[0010] In one embodiment of the present invention, the second feature extraction module includes a stacked convolution submodule and an image sequence conversion submodule. The sample image data is input into the second feature extraction module for second feature extraction to obtain the second feature information, including: inputting the sample image data into the stacked convolution submodule for local feature extraction to obtain multiple local feature maps; inputting the multiple local feature maps into the image sequence conversion submodule for feature sequence conversion to obtain the second feature information, and the output end of the stacked convolution submodule is connected to the input end of the image sequence conversion submodule.
[0011] In one embodiment of the present invention, the process of inputting the sample image data into the second feature extraction module to extract the second feature can be characterized as follows:
[0012] f i l =f i g+PCM(f i )
[0013] f i g =MHSA i (Img2Seq(f i m ))
[0014]
[0015] Among them, f i l is the lth result after the local branching of the ith input, f i g is the g-th result of the i-th input processed by multi-head attention (MHSA), f i is the i-th input, PCM() is the triple convolution module, Attension is the attention formula, Q is the projection matrix, K is the key, V is the value, softmax is the activation function, K T represents the transpose of K.
[0016] In one embodiment of the present invention, the first feature information and the second feature information are input into the attention network for third feature extraction to obtain a feature image to be processed, including: fusing the first feature information and the second feature information to obtain a fused feature image; and performing global feature extraction on the fused feature image based on the attention network to obtain a feature image to be processed.
[0017] In one embodiment of the present invention, the process of extracting hidden state features from the feature image to be processed based on the state space module is characterized as follows:
[0018] x′(t)=Ax(t)+Bu(t)
[0019] y(t)=Cx(t)+Du(t)
[0020] Among them, A is the parameter of the linear ordinary differential equation, A and B are projection parameters, t is the time, x′(t) is the result after the input passes through the projection layer, x(t) is the state vector of the model, u(t) is the input of the model, C is the parameter for the conversion from input to output, D is the factor parameter affecting the output, and y(t) is the output vector of the model.
[0021] An embodiment of the present invention also provides an image segmentation device, characterized in that the image segmentation device includes: an image data acquisition module, used to acquire sample image data and an image to be processed; a model construction module, used to input the sample image data into a first feature extraction module for first feature extraction to obtain first feature information, and input the sample image data into a second feature extraction module for second feature extraction to obtain second feature information; input the first feature information and the second feature information into an attention network for third feature extraction to obtain a feature image to be processed; based on a state space module, hidden state feature extraction is performed on the feature image to be processed, and a target feature image is output through a fully connected layer; based on the target feature image, an image segmentation model is trained to obtain a trained image segmentation model, and the image segmentation model includes the first feature extraction module, the second feature extraction module, the attention network, the state space module and the fully connected layer; an image segmentation processing module, used to input the image to be processed into the trained image segmentation model to obtain a target segmented image.
[0022] An embodiment of the present invention also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the image segmentation method as described in any one of the above embodiments.
[0023] An embodiment of the present invention further provides a computer-readable storage medium on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the image segmentation method as described in any one of the above embodiments.
[0024] The present invention provides an image segmentation method, device, equipment and storage medium. By acquiring sample image data and an image to be processed, the sample image data is respectively input into a first feature extraction module and a second feature extraction module to obtain first feature information and second feature information, and after extraction, the sample image data is input into an attention network for third feature extraction to obtain a feature image to be processed, hidden state feature extraction is performed on the feature image to be processed based on a state space module, a target feature image is output through a fully connected layer, an image segmentation model is trained based on the target feature image to obtain a trained image segmentation model, and the image to be processed is segmented to obtain a target segmented image; the method uses an attention network to emphasize important features of the feature image after convolution processing, suppresses irrelevant background information, better captures the global context information in the image, and uses a state space module to extract hidden state features, establishes better long-distance dependencies, helps to learn the potential dynamic change laws in the image, thereby enhancing the model's ability to understand complex scenes, improves the shortcomings of the attention mechanism in processing long-distance dependencies in the network structure, and provides a better image segmentation application technology effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0026] Figure 1 is a schematic diagram of an exemplary system architecture shown in an exemplary embodiment of the present application;
[0027] Figure 2 is a flow chart of an image segmentation method shown in an exemplary embodiment of the present application;
[0028] Figure 3 is a schematic diagram of a specific image segmentation method flow chart shown in an exemplary embodiment of the present application;
[0029] Figure 4 is a schematic diagram of an image segmentation device shown in an exemplary embodiment of the present application;
[0030] Figure 5 It is a structural diagram of a computer system of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0031] The following will describe the embodiments of the present invention with reference to the accompanying drawings and specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.
[0032] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic manner, and thus the illustrations only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0033] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.
[0034] The "and / or" mentioned in this application describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.
[0035] First of all, it should be explained that the present application is applicable to intelligent driving or assisted driving scenarios of vehicles. Image acquisition is performed based on the image acquisition device set up in the vehicle, and image segmentation is performed based on this scheme to obtain high-precision and feature-clear segmented images, so that the vehicle can perform autonomous driving or assisted driving decision-making applications.
[0036] Figure 1 It is a schematic diagram of an exemplary system architecture shown in an exemplary embodiment of the present application.
[0037] Reference Figure 1 As shown, the system architecture may include an image acquisition device 110 and a computer device 120. The computer device 120 acquires sample image data and an image to be processed through the image acquisition device 110, inputs the sample image data into a first feature extraction module for first feature extraction, obtains first feature information, and inputs the sample image data into a second feature extraction module for second feature extraction, obtains second feature information, inputs the first feature information and the second feature information into an attention network for third feature extraction, obtains a feature image to be processed, performs hidden state feature extraction on the feature image to be processed based on a state space module, and outputs a target feature image through a fully connected layer, trains an image segmentation model based on the target feature image, obtains a trained image segmentation model, and the image segmentation model includes a first feature extraction module, a second feature extraction module, an attention network, a state space module, and a fully connected layer, and inputs the image to be processed into the trained image segmentation model to obtain a target segmented image. The above-mentioned computer equipment 120 can be at least one of a microcomputer, an embedded computer, a network computer, a single-chip microcomputer, etc.; the above-mentioned track door sensor device 110 refers to a vibration sensor used to collect vibration signals of track shielding doors. The above-mentioned vibration sensor is installed on the guide rail above the shielding door, and then connected to the opening and closing control signal of the shielding door. The vibration signal collector collects vibration signals after receiving the switch signal of the slide rail movement.
[0038] Illustratively, the computer device 120 acquires sample image data and an image to be processed through the image acquisition device 110, inputs the sample image data into the first feature extraction module and the second feature extraction module respectively, obtains the first feature information and the second feature information, and inputs the extracted data into the attention network for the third feature extraction to obtain the feature image to be processed, performs hidden state feature extraction on the feature image to be processed based on the state space module, outputs the target feature image through the fully connected layer, trains the image segmentation model based on the target feature image to obtain the trained image segmentation model, and segments the image to be processed to obtain the target segmented image; the method utilizes the attention network to emphasize important features of the feature image after convolution processing, suppresses irrelevant background information, better captures the global context information in the image, and utilizes the state space module to extract hidden state features, establishes better long-distance dependencies, helps to learn the potential dynamic change laws in the image, thereby enhancing the model's ability to understand complex scenes, improves the shortcomings of the attention mechanism in handling long-distance dependencies in the network structure, and provides better image segmentation application technology effects.
[0039] Figure 2 is a flowchart of an image segmentation method shown in an exemplary embodiment of the present application. The image segmentation method can be Figure 1 The implementation environment may be implemented in other implementation environments, and the above implementation environments are not specifically limited here. Figure 2 As shown, the flowchart of the image segmentation method includes at least steps S210 to S260, which are described in detail as follows:
[0040] In step S210, sample image data and an image to be processed are obtained.
[0041] In one embodiment of the present application, the above-mentioned image to be processed is acquired by an image acquisition device, which is arranged in the vehicle and includes at least one image acquisition device, including but not limited to a camera, a surround view camera, an infrared camera, a night vision camera, a thermal imager, a three-dimensional camera, a stereo camera, etc.
[0042] In one embodiment of the present application, the method for acquiring sample image data includes selecting from a historical image library acquired by an image acquisition device, selecting images from Internet information, experimental shooting data, virtually constructing digital images, etc. In an embodiment of this scheme, the method for acquiring sample image data is selecting from a historical image library acquired by an image acquisition device, which includes at least one of a single image, a continuous frame image, and a key frame image.
[0043] After the image is acquired, it is first preprocessed, including data cleaning to reduce the blur of the image, and then data enhancement operations such as geometric transformation to enhance the robustness of the data, adjust the size to a fixed size, and normalize the pixel values in the image for subsequent image processing steps.
[0044] In step S220, the sample image data is input into a first feature extraction module to perform first feature extraction to obtain first feature information, and the sample image data is input into a second feature extraction module to perform second feature extraction to obtain second feature information.
[0045] In one embodiment of the present application, the first feature extraction module includes a pyramid reduction submodule and a multi-head attention submodule. Pyramid reduction is an image processing technology that generates a multi-layer image structure by gradually reducing the image size and performing filtering and smoothing. Each layer is a reduced version of the previous layer, which is used for image analysis and processing at different resolutions. Multi-head attention is an improved attention mechanism used to process sequence data in deep learning models. It decomposes the input information into multiple heads, each head performs attention calculations independently, thereby capturing different subspace features in the input data, and finally merging the results of these heads to enhance the model's understanding and representation of the multi-faceted dependencies of the input data.
[0046] In one embodiment of the present application, sample image data is input into a first feature extraction module for first feature extraction, including inputting the sample image data into a pyramid simplification submodule for multi-level feature extraction to obtain a first initial feature image set, the pyramid simplification submodule includes multiple stepped convolution subnetworks with different expansion rates, the first initial feature image is input into a multi-head attention submodule for hierarchical feature enhancement to obtain multiple hierarchical enhanced feature images, and the multiple hierarchical enhanced feature images are spliced to obtain first feature information, and the output end of the pyramid simplification submodule is connected to the input end of the multi-head attention submodule.
[0047] In one embodiment of the present application, the process of inputting sample image data into the pyramid reduction submodule for multi-level feature extraction is characterized as follows:
[0048]
[0049] Among them, f i m is the mth result after being processed by the i-th pyramid reduction module, PRM i is the ith pyramid reduction module, f i is the i-th image input, Cat() is the fusion function, Conv ij() is the jth convolutional layer in the i-th pyramid reduction module, S i is the i-th expansion rate set, s ij is the jth expansion rate in the ith expansion rate set, R is the reduction ratio set, and ri is the ith reduction ratio in the reduction ratio set.
[0050] In one embodiment of the present application, the second feature extraction module includes a stacked convolution submodule and an image-to-sequence submodule. Stacked convolutions refer to the continuous use of multiple convolution layers in a convolutional neural network (CNN) without inserting a pooling layer or a nonlinear activation function, and deepening the network by stacking layer by layer to gradually extract high-level features in the input data, enhance the model's feature expression capabilities and the ability to capture complex patterns. The image-to-sequence module is a mechanism for converting a two-dimensional image into a one-dimensional sequence, by expanding each pixel or feature map in the image one by one to form a sequence arranged in order, so that the Transformer model can directly process image information in subsequent processing, and is suitable for tasks such as image classification and image description generation.
[0051] In one embodiment of the present application, inputting the sample image data into the second feature extraction module for second feature extraction includes inputting the sample image data into the stacked convolution submodule for local feature extraction to obtain multiple local feature maps, inputting the multiple local feature maps into the image conversion sequence submodule for feature sequence conversion to obtain second feature information, and the output end of the stacked convolution submodule is connected to the input end of the image conversion sequence submodule.
[0052] In one embodiment of the present application, the process of inputting the sample image data into the second feature extraction module for second feature extraction can be characterized as follows:
[0053] f i l =f i g +PCM(f i )
[0054] f i g =MHSA i (Img2Seq(f i m ))
[0055]
[0056] Among them, f i l is the lth result after the local branching of the ith input, f i gis the g-th result of the i-th input processed by multi-head attention (MHSA), f i is the i-th input, PCM() is the triple convolution module, Attension is the attention formula, Q is the projection matrix, K is the key, V is the value, softmax is the activation function, K T represents the transpose of K.
[0057] In one embodiment of the present application, the sample image data is sent to the first feature extraction module of multi-scale context extraction, and the features of the multi-scale context are extracted using stepped convolution with multiple dilation rates. At the same time, the local correlation of the sample image data is modeled in the second feature extraction module, so that the model can make full use of local information features while maintaining the ability to extract global information.
[0058] In step S230, the first feature information and the second feature information are input into the attention network to extract the third feature to obtain a feature image to be processed.
[0059] In one embodiment of the present application, the first feature information and the second feature information are fused to obtain a fused feature image, and the fused feature image is subjected to global feature extraction based on the attention network to obtain a feature image to be processed. The attention network selects the Transformer network for feature processing. The Transformer network is a deep learning model based on the self-attention mechanism, which is used to process sequence data. By calculating the relationship between each position in the sequence, it realizes parallel processing of input data while capturing long-distance dependencies. It is widely used in natural language processing, image processing and other fields, and significantly improves processing efficiency and model performance.
[0060] The process of extracting global features from the fused feature image based on the attention network realizes feature processing through the attention calculation expression in the above formula, which is represented as:
[0061]
[0062] Among them, Attension is the attention formula, Q is the projection matrix, K is the key, V is the value, softmax is the activation function, K T represents the transpose of K.
[0063] In one embodiment of the present application, the process of fusing the first feature information and the second feature information includes feature addition or feature concatenation to meet feature fusion requirements.
[0064] In step S240, hidden state features are extracted from the feature image to be processed based on the state space module, and the target feature image is output through the fully connected layer.
[0065] In one embodiment of the present application, the process of extracting hidden state features from the feature image to be processed based on the state space module is characterized as follows:
[0066] x′(t)=Ax(t)+Bu(t)
[0067] y(t)=Cx(t)+Du(t)
[0068] Among them, A is the parameter of the linear ordinary differential equation, A and B are projection parameters, t is the time, x′(t) is the result after the input passes through the projection layer, x(t) is the state vector of the model, u(t) is the input of the model, C is the parameter for the conversion from input to output, D is the factor parameter affecting the output, and y(t) is the output vector of the model.
[0069] In one embodiment of the present application, the state space module refers to the feature space represented by the hidden layer in the neural network, which captures the inherent complex patterns and structures of the data by learning the abstract representation of the input data, and provides more meaningful feature representations for downstream tasks such as classification and regression. Specifically, the Mamba network is selected. The Mamba network is a neural network architecture designed to optimize the efficiency of processing long sequence data. By adopting a design with linear time complexity, it can effectively capture long-term dependencies in the sequence while maintaining computational efficiency.
[0070] In step S250, the image segmentation model is trained based on the target feature image to obtain a trained image segmentation model.
[0071] In one embodiment of the present application, the image segmentation model includes a first feature extraction module, a second feature extraction module, an attention network, a state space module and a fully connected layer. In order to improve the training effect, the training process introduces a loss function to further perform shrinkage training, and the target feature image loss value can be calculated. Based on the loss weight set based on the training target during the training process, further model training is performed to improve the accuracy of the model.
[0072] In step S260, the image to be processed is input into the trained image segmentation model to obtain a target segmented image.
[0073] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of a specific image segmentation method shown in an exemplary embodiment of the present application. Figure 3As shown in the flowchart of the specific image segmentation method, a monitoring image is acquired by an image acquisition device, including sample image data and an image to be processed. After image preprocessing, the monitoring image is input into a global modeling branch (consistent with the first feature extraction module in the above embodiment) and a local modeling branch (consistent with the second feature extraction module in the above embodiment) respectively. In the global modeling branch, the pyramid approximation is first performed using a predefined step convolution with different expansion rates to obtain multi-scale context information, and then input into a multi-head attention module to expand the single-head self-attention for different projection matrices; in the local modeling branch, the image is passed through a multiple convolution module, and the local correlation of the image is modeled by three groups of convolution units to fully extract the local feature information. Finally, the results of the two branches are fused, and the fused feature map is first processed by a Transformer (consistent with the attention network in the above embodiment), and the image is first processed using its core attention mechanism to establish long-distance dependencies so that invalid information in the image is eliminated, and then the state space model Mamba (consistent with the state space module in the above embodiment) is used to perform longer-distance dependency modeling on the input to achieve accurate segmentation of the vehicle.
[0074] The present invention provides an image segmentation method, device, equipment and storage medium. By acquiring sample image data and an image to be processed, the sample image data is respectively input into a first feature extraction module and a second feature extraction module to obtain first feature information and second feature information, and after extraction, the sample image data is input into an attention network for third feature extraction to obtain a feature image to be processed, hidden state feature extraction is performed on the feature image to be processed based on a state space module, a target feature image is output through a fully connected layer, an image segmentation model is trained based on the target feature image to obtain a trained image segmentation model, and the image to be processed is segmented to obtain a target segmented image; the method uses an attention network to emphasize important features of the feature image after convolution processing, suppresses irrelevant background information, better captures the global context information in the image, and uses a state space module to extract hidden state features, establishes better long-distance dependencies, helps to learn the potential dynamic change laws in the image, thereby enhancing the model's ability to understand complex scenes, improves the shortcomings of the attention mechanism in processing long-distance dependencies in the network structure, and provides a better image segmentation application technology effect.
[0075] The following describes an apparatus embodiment of the present application, which can be used to execute the image segmentation method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the image segmentation method of the present application.
[0076] Figure 4 is a schematic diagram of an image segmentation device shown in an exemplary embodiment of the present application. The device can be applied to Figure 2The method implementation process shown in the figure can be based on Figure 1 The present invention is executed in the implementation environment shown in the figure, and may also be applicable to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the apparatus is applicable.
[0077] like Figure 4 As shown, the exemplary image segmentation device includes: an image data acquisition module 401, a model building module 402 and an image segmentation processing module 403.
[0078] Among them, the image data acquisition module 401 is used to obtain sample image data and an image to be processed; the model construction module 402 is used to input the sample image data into the first feature extraction module for first feature extraction to obtain first feature information, and input the sample image data into the second feature extraction module for second feature extraction to obtain second feature information; the first feature information and the second feature information are input into the attention network for third feature extraction to obtain a feature image to be processed; based on the state space module, hidden state feature extraction is performed on the feature image to be processed, and a target feature image is output through a fully connected layer; based on the target feature image, an image segmentation model is trained to obtain a trained image segmentation model, and the image segmentation model includes a first feature extraction module, a second feature extraction module, an attention network, a state space module and a fully connected layer; the image segmentation processing module 403 is used to input the image to be processed into the trained image segmentation model to obtain a target segmented image.
[0079] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the electronic device implements the image segmentation method provided in the above-mentioned embodiments.
[0080] Figure 5 is a schematic diagram of the structure of a computer system of an electronic device according to an exemplary embodiment of the present application. It should be noted that: Figure 5 The computer system 500 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0081] like Figure 5As shown, computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in read-only memory (ROM) 502 or the program loaded from the storage part to random access memory (RAM) 503, such as executing the method in the above embodiment. In RAM 503, various programs and data required for system operation are also stored. CPU 501, ROM 502 and RAM 503 are connected to each other through a bus. Input / output (I / O) interface 505 is also connected to bus 504.
[0082] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.
[0083] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 509, and / or installed from a removable medium 511. When the computer program is executed by a central processing unit (CPU) 501, various functions defined in the system of the present application are executed.
[0084] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0086] In the corresponding drawings of the above embodiments, connecting lines may indicate the connection relationship between the various components to indicate more constituent signal paths (constituent_signalpath) and / or one or more ends of some lines may have arrows to indicate the main information flow direction. Connecting lines, as a mark, are not a limitation to the scheme itself, but the use of these lines in combination with one or more exemplary embodiments helps to connect circuits or logic units more easily. Any represented signal (determined by design requirements or preferences) may actually include one or more signals that can be transmitted in either direction and implemented with any appropriate type of signal scheme.
[0087] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0088] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.
[0089] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.
[0090] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.
[0091] It should be noted that the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.
[0092] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.
[0093] It should be understood that the above content of the present application is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. Ordinary technicians in this field can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application should be the scope of protection required by the claims.
Claims
1. An image segmentation method, characterized in that: The image segmentation method comprises: Obtain sample image data and images to be processed; Inputting the sample image data into a first feature extraction module to perform first feature extraction to obtain first feature information, and inputting the sample image data into a second feature extraction module to perform second feature extraction to obtain second feature information; Inputting the first feature information and the second feature information into the attention network to extract the third feature, thereby obtaining a feature image to be processed; Extract hidden state features of the feature image to be processed based on the state space module, and output the target feature image through a fully connected layer; Training an image segmentation model based on the target feature image to obtain a trained image segmentation model, wherein the image segmentation model includes the first feature extraction module, the second feature extraction module, an attention network, a state space module, and a fully connected layer; The image to be processed is input into the trained image segmentation model to obtain a target segmented image.
2. The image segmentation method according to claim 1, characterized in that: The first feature extraction module includes a pyramid reduction submodule and a multi-head attention submodule. The sample image data is input into the first feature extraction module to perform first feature extraction, and the first feature information obtained includes: Inputting the sample image data into the pyramid reduction submodule for multi-level feature extraction to obtain a first initial feature image set, wherein the pyramid reduction submodule includes a plurality of stepped convolution subnetworks with different dilation rates; The first initial feature image is input into the multi-head attention submodule for hierarchical feature enhancement to obtain multiple hierarchical enhanced feature images, and the multiple hierarchical enhanced feature images are spliced to obtain first feature information. The output end of the pyramid simplification submodule is connected to the input end of the multi-head attention submodule.
3. The image segmentation method according to claim 2, characterized in that: The process of inputting the sample image data into the pyramid reduction submodule for multi-level feature extraction is characterized as follows: in, is the mth result after being processed by the i-th pyramid reduction module, PRM i is the ith pyramid reduction module, f i is the i-th image input, Cat() is the fusion function, Conv ij () is the jth convolutional layer in the i-th pyramid reduction module, S i is the i-th expansion rate set, s ij is the jth expansion rate in the i-th expansion rate set, R is the reduction ratio set, r i is the i-th reduction ratio in the reduction ratio set.
4. The image segmentation method according to claim 1, characterized in that: The second feature extraction module includes a stacked convolution submodule and an image conversion sequence submodule. The sample image data is input into the second feature extraction module for second feature extraction, and the second feature information obtained includes: Inputting the sample image data into the stacked convolution submodule to extract local features to obtain multiple local feature maps; The multiple local feature maps are input into the image-to-sequence submodule for feature sequence conversion to obtain second feature information, and the output end of the stacked convolution submodule is connected to the input end of the image-to-sequence submodule.
5. The image segmentation method according to claim 4, characterized in that: The process of inputting the sample image data into the second feature extraction module for second feature extraction can be characterized as follows: in, is the lth result after the local branching of the ith input, is the g-th result of the i-th input processed by multi-head attention (MHSA), f i is the i-th input, PCM() is the triple convolution module, Attension is the attention formula, Q is the projection matrix, K is the key, V is the value, softmax is the activation function, K T represents the transpose of K.
6. The image segmentation method according to claim 1, characterized in that: Inputting the first feature information and the second feature information into the attention network to extract the third feature, and obtaining the feature image to be processed includes: Fusing the first feature information and the second feature information to obtain a fused feature image; Based on the attention network, global features are extracted from the fused feature image to obtain a feature image to be processed.
7. The image segmentation method according to claim 1, characterized in that: The process of extracting hidden state features of the feature image to be processed based on the state space module is characterized as follows: x′(t)=Ax(t)+Bu(t) y(t)=Cx(t)+Du(t) Among them, A is the parameter of the linear ordinary differential equation, A and B are projection parameters, t is the time, x′(t) is the result after the input passes through the projection layer, x(t) is the state vector of the model, u(t) is the input of the model, C is the parameter for the conversion from input to output, D is the factor parameter affecting the output, and y(t) is the output vector of the model.
8. An image segmentation device, characterized in that: The image segmentation device comprises: An image data acquisition module, used to acquire sample image data and images to be processed; A model building module is used to input the sample image data into a first feature extraction module for first feature extraction to obtain first feature information, and input the sample image data into a second feature extraction module for second feature extraction to obtain second feature information; input the first feature information and the second feature information into an attention network for third feature extraction to obtain a feature image to be processed; perform hidden state feature extraction on the feature image to be processed based on a state space module, and output a target feature image through a fully connected layer; train an image segmentation model based on the target feature image to obtain a trained image segmentation model, wherein the image segmentation model includes the first feature extraction module, the second feature extraction module, the attention network, the state space module, and the fully connected layer; The image segmentation processing module is used to input the image to be processed into the trained image segmentation model to obtain a target segmented image.
9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the electronic device to implement the image segmentation method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the image segmentation method according to any one of claims 1 to 7.