Image segmentation method and device, storage medium and electronic device
By extracting horizontal and vertical vector features and using a strip attention mechanism to calculate global attention features, the redundant computation problem of global context modeling in deep convolutional neural networks is solved, thus improving the speed and efficiency of image segmentation.
Patent Information
- Application Number
- CN202210130539.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-11
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2042-02-11
AI Technical Summary
Existing image segmentation methods based on deep convolutional neural networks struggle to model global contextual relationships, resulting in high computational and storage costs, as well as redundant computation issues.
By extracting horizontal and vertical vector features, and using a bar-shaped attention mechanism to calculate global attention features, redundant computation is reduced and efficiency is improved.
This approach improves image segmentation speed, reduces computational and storage overhead, and solves the redundant computation problem of global context modeling in traditional methods.
Smart Images
Figure CN114463342B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to computer vision technology, and in particular to an image segmentation method and apparatus, storage medium, and electronic device. Background Technology
[0002] In image segmentation based on deep convolutional neural networks, context modeling has always been a key focus. Convolutional neural networks, with their good local connectivity, can handle local relationship modeling well. However, it is difficult for convolutional neural networks to achieve global context relationship modeling. In image segmentation, context relationship modeling can better help the model analyze and understand the scene, distinguish different types of regions, and achieve segmentation. Summary of the Invention
[0003] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide an image segmentation method and apparatus, a storage medium, and an electronic device.
[0004] According to one aspect of the present disclosure, an image segmentation method is provided, comprising:
[0005] The features corresponding to the image to be processed are processed to obtain horizontal vector features and vertical vector features;
[0006] Based on the features to be processed, the horizontal vector features, and the vertical vector features, global attention features are obtained;
[0007] Based on the global attention features, the segmentation result of the image to be processed is determined.
[0008] According to another aspect of the embodiments of this disclosure, an image segmentation apparatus is provided, comprising:
[0009] The feature extraction module is used to process the features corresponding to the image to be processed, and obtain horizontal vector features and vertical vector features;
[0010] An attention determination module is used to obtain global attention features based on the features to be processed, the horizontal vector features, and the vertical vector features determined by the feature extraction module.
[0011] The image segmentation module is used to determine the segmentation result of the image to be processed based on the global attention features determined by the attention determination module.
[0012] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the image segmentation method described in any of the above embodiments.
[0013] According to another aspect of the embodiments of this disclosure, an electronic device is provided, the electronic device comprising:
[0014] processor;
[0015] Memory used to store the processor's executable instructions;
[0016] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image segmentation method described in any of the above embodiments.
[0017] Based on the image segmentation method, apparatus, storage medium, and electronic device provided in the above embodiments of this disclosure, the features to be processed corresponding to the image to be processed are processed to obtain horizontal vector features and vertical vector features; based on the features to be processed, the horizontal vector features, and the vertical vector features, global attention features are obtained; and based on the global attention features, the segmentation result of the image to be processed is determined. The embodiments of this disclosure accelerate image segmentation by extracting horizontal and vertical vector features, solve the problem of high computational and storage overhead caused by directly calculating attention mechanisms based on image features, and overcome the problem of large amounts of redundant computation in the context modeling of attention mechanisms in traditional technologies, thereby eliminating redundancy and reducing resource consumption.
[0018] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0019] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.
[0020] Figure 1a This is a schematic diagram of the structure of an image segmentation network for semantic segmentation of an image, provided in an exemplary embodiment of this disclosure.
[0021] Figure 1b This is a schematic diagram of the structure of an application attention mechanism network for processing features provided in an exemplary embodiment of this disclosure.
[0022] Figure 2 This is a schematic flowchart of an image segmentation method provided in an exemplary embodiment of this disclosure.
[0023] Figure 3 This is a public announcement Figure 2 A flowchart illustrating step 201 in the illustrated embodiment.
[0024] Figure 4 This is a public announcement Figure 2 A flowchart illustrating step 202 in the illustrated embodiment.
[0025] Figure 5 This is a public announcement Figure 2 A flowchart illustrating step 203 in the illustrated embodiment.
[0026] Figure 6 This is a schematic diagram of the structure of an image segmentation apparatus provided in an exemplary embodiment of the present disclosure.
[0027] Figure 7 This is a schematic diagram of the structure of an image segmentation apparatus provided in another exemplary embodiment of this disclosure.
[0028] Figure 8 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation
[0029] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.
[0030] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of this disclosure.
[0031] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.
[0032] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.
[0033] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.
[0034] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.
[0035] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.
[0036] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.
[0037] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.
[0038] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0039] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0040] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.
[0041] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.
[0042] Application Overview
[0043] In the process of realizing this disclosure, the inventors discovered that the global context relationship modeling in related technologies, which is usually based on dilated convolution and downsampling, often has the following problems: it is impossible to obtain global context modeling in practice, and it is difficult to handle the relationship between fine-grained pixels.
[0044] Exemplary System
[0045] Figure 1a This is a schematic diagram illustrating the structure of an image segmentation network for semantic segmentation of an image, provided in an exemplary embodiment of this disclosure. For example... Figure 1a As shown, the image to be processed is input into the convolutional neural network 101 for feature extraction to obtain the features to be processed corresponding to the image to be processed; the features to be processed are then input into the attention mechanism network 102.
[0046] Figure 1b This is a schematic diagram illustrating the structure of an application attention mechanism network for processing features according to an exemplary embodiment of this disclosure. The attention mechanism network 102 performs reduction operations on the feature F to be processed in the horizontal and vertical directions respectively (e.g., taking the maximum value, i.e., taking the maximum value in each column of feature F as the value in the horizontal bar feature, and taking the maximum value in each row of feature F as the value in the vertical bar feature), resulting in horizontal bar features (horizontal vector features) and vertical bar features (vertical vector features). Each of the two reduction network branches is connected to a bar attention network branch, and the bar attention network branch (connected to the output horizontal...) A reduction network branch (connected to the reduction network branch of the output vertical vector feature) performs an attention operation on the feature to be processed and the horizontal vector feature to obtain the first attention feature; another strip-shaped attention network branch (connected to the reduction network branch of the output vertical vector feature) performs an attention operation on the feature to be processed and the vertical vector feature to obtain the second attention feature. The first and second attention features are then summed to obtain the global attention feature SA(F). The global attention feature SA(F) is then processed using the prediction network branch 103, which can perform a convolution operation on the global attention feature to obtain, as shown below. Figure 1a The image segmentation results shown (different colors can be used to indicate image regions corresponding to different objects in the image segmentation results; due to the limitation that the attached image can only display grayscale images, ...) Figure 1a (The image regions corresponding to different objects are distinguished only by different gray levels).
[0047] The strip attention mechanism proposed in this embodiment reduces the time complexity by an order of magnitude compared to existing attention mechanisms, and also reduces the space complexity by an order of magnitude. Furthermore, compared to existing image segmentation networks (e.g., CCNet), this embodiment does not require iterative approximation steps, resulting in a significant reduction in computational and storage overhead compared to image segmentation networks.
[0048] Furthermore, this embodiment is simple to implement and highly scalable. The bar attention mechanism is easy to implement, requires no additional Compute Unified Device Architecture (CUDA) computation, supports native computation implementation of any framework, and involves simple computations, such as matrix addition, matrix multiplication, matrix mean / maximum calculation, Softmax operation, etc.
[0049] Optionally, before training the image segmentation network proposed in the above embodiments of this disclosure, the training process of the image segmentation network may also include training the image segmentation network. The training process of the image segmentation network is similar to the training process of deep neural networks in the prior art. The image segmentation network is jointly trained using sample images with known segmentation results. The known segmentation results are used as supervision information to determine the network loss. The gradient descent method can be used to train the image segmentation network to obtain the trained image segmentation network.
[0050] Exemplary methods
[0051] Figure 2 This is a schematic flowchart of an image segmentation method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 2 As shown, it includes the following steps:
[0052] Step 201: Process the features corresponding to the image to be processed to obtain horizontal vector features and vertical vector features.
[0053] Optionally, the corresponding features to be processed can be obtained from the image to be processed through methods such as deep learning. For example, a convolutional neural network (CNN) can be used to extract features from the image to be processed to obtain the features to be processed; the features to be processed can be feature maps, feature vectors, etc.
[0054] Step 202: Based on the features to be processed, the horizontal vector features, and the vertical vector features, obtain the global attention features.
[0055] In one embodiment, attention can be calculated by combining the features to be processed with horizontal vector features and vertical vector features respectively to obtain global attention features corresponding to the image to be processed. Since the horizontal vector features and vertical vector features are strip features, a strip attention mechanism is realized. Furthermore, since a large number of redundant features are reduced in the strip features, the speed of attention calculation is improved.
[0056] Step 203: Determine the segmentation result of the image to be processed based on global attention features.
[0057] Optionally, the segmentation result can be a semantic segmentation result, which means recognizing the different semantic content and their locations in the image (by finding all pixels belonging to each semantic content).
[0058] The image segmentation method provided in the above embodiments of this disclosure processes the features to be processed corresponding to the image to be processed, obtaining horizontal vector features and vertical vector features; based on the features to be processed, the horizontal vector features, and the vertical vector features, a global attention feature is obtained; and based on the global attention feature, the segmentation result of the image to be processed is determined. The embodiments of this disclosure accelerate the image segmentation processing speed by extracting horizontal and vertical vector features, solve the problem of high computational and storage overhead caused by directly calculating the attention mechanism based on image features, and solve the problem of a large amount of redundant computation in the context modeling of the attention mechanism, thereby eliminating redundancy and reducing resource consumption.
[0059] like Figure 3 As shown above, in the above Figure 2 Based on the illustrated embodiment, step 201 may include the following steps:
[0060] Step 2011: Perform the first preset reduction operation on the feature to be processed in the vertical direction to obtain the horizontal vector feature.
[0061] In this embodiment, the first preset reduction operation can be an operation to find the maximum value or the mean value. By using the first preset reduction operation, the feature to be processed is reduced to a smaller value (e.g., 1) in the vertical direction. The amount of data of the resulting horizontal vector feature is greatly reduced compared to the amount of data of the feature to be processed, which reduces the complexity of subsequent calculations.
[0062] Step 2012: Perform a second preset reduction operation on the feature to be processed in the horizontal direction to obtain the vertical vector feature.
[0063] Optionally, the second preset reduction operation is similar to the first preset reduction operation, and can also be an operation to find the maximum value or the mean value. The only difference is that the second preset reduction operation reduces the feature to be processed in the horizontal direction. Furthermore, the first preset reduction operation and the second preset reduction operation can be the same or different. When the first preset reduction operation and the second preset reduction operation are the same, a more accurate segmentation result can be achieved.
[0064] In this embodiment, the order of steps 2011 and 2012 can be arbitrarily changed. For example, step 2012 can be executed first, followed by step 2011; or step 2011 can be executed first, followed by step 2012. The execution order of the two steps does not affect the technical effect of this embodiment. It is only necessary to perform the first preset reduction operation and the second preset reduction operation on the features to be processed in the vertical and horizontal directions, respectively. This embodiment removes redundant features in attention calculation by performing reduction operations in the vertical and horizontal directions, thereby improving the speed and effect of image segmentation.
[0065] Optionally, based on the above embodiments, step 2011 may further include:
[0066] The feature to be processed is convolved with the first preset vertical vector to obtain the first convolutional feature.
[0067] Perform a first preset reduction operation on the first convolutional feature in the vertical direction to obtain the horizontal vector feature.
[0068] In this embodiment, a convolution operation can be performed between the feature to be processed and a first preset vertical vector, wherein the first preset vertical vector can be trained (in... Figure 1a The image segmentation network training process shown in the figure obtains an n*1 vertical vector. Optionally, to improve efficiency, the n*1 vertical vector can be compressed into a 5*1 vertical vector. In this embodiment, the vertical features of the features to be processed are extracted through convolution operation. Due to the use of a large-size convolution kernel (e.g., 5*1), the first convolution feature has a larger receptive field in the vertical direction. Furthermore, due to the larger receptive field, a larger range of contextual features is obtained, so that the obtained horizontal vector features supplement the vertical contextual features.
[0069] Optionally, based on the above embodiments, step 2012 may further include:
[0070] The features to be processed are convolved with the first preset horizontal vector to obtain the second convolutional features.
[0071] Perform a second preset reduction operation on the second convolutional feature in the horizontal direction to obtain the vertical vector feature.
[0072] In this embodiment, a convolution operation can be performed between the feature to be processed and a first preset horizontal vector, wherein the first preset horizontal vector can be... Figure 1a During the training process of the image segmentation network shown, a 1*n horizontal vector is determined (the first preset horizontal vector is adjusted as a training parameter during training). Optionally, to improve efficiency, the 1*n horizontal vector can be compressed into a 1*5 horizontal vector. In this embodiment, the horizontal features of the features to be processed are extracted through convolution operation. Due to the use of a large-size convolution kernel (e.g., 5*1), the first convolution feature has a larger receptive field in the horizontal direction. Furthermore, due to the larger receptive field, a larger range of contextual features is obtained, which supplements the horizontal contextual features with the obtained vertical vector features.
[0073] like Figure 4 As shown above, in the above Figure 2 Based on the illustrated embodiment, step 202 may include the following steps:
[0074] Step 2021: Perform attention operation on the feature to be processed and the horizontal vector feature to obtain the first attention feature.
[0075] The attention operation performed in this embodiment is similar to the self-attention in the prior art. First, the first similarity between the feature to be processed and the horizontal vector feature is determined. Optionally, the first similarity can be determined using the following formula (1):
[0076]
[0077] Among them, A v F represents the first similarity; F represents the feature to be processed; F v Represents the characteristics of the horizontal vector; and The query and key weights represent horizontal attention; these two weights can be implemented using linear layers of the neural network (specific network parameters are provided in the original text). Figure 1a The image segmentation network training shown is determined); it can be understood that the formula (1) provided in this embodiment can be implemented by a neural network, for example, by processing the feature F to be processed by the linear layer corresponding to the query weight of the horizontal attention, and by processing the horizontal vector feature F by the linear layer corresponding to the key weight of the horizontal attention. v The process involves processing the results of the two linear layers using a softmax layer to obtain the first similarity. The first attention feature is the output feature aggregated from the feature to be processed and the horizontal vector feature based on the first similarity, i.e., each pixel feature in the feature to be processed F and the horizontal vector feature F. vThe similarity of each pixel feature is calculated to obtain the first similarity. Then, based on the first similarity, the horizontal vector feature F is analyzed. v The first attention feature is obtained by extracting features from the first attention feature and filling them into the corresponding positions of the feature to be processed, F. Therefore, the first attention feature has the same size as the feature to be processed. Alternatively, the first attention feature can be determined using the following formula (2):
[0078]
[0079] Among them, O v Indicates the primary attentional feature; A v Indicates the first similarity; The value weights representing horizontal attention can be implemented using linear layers of a neural network (specific network parameters are specified through...). Figure 1a The image segmentation network training shown is determined); F v This represents the characteristics of the horizontal vector.
[0080] Step 2022: Perform attention calculation on the feature to be processed and the vertical vector feature to obtain the second attention feature.
[0081] The attention operation performed in this embodiment is similar to the self-attention in the prior art. First, the first similarity between the feature to be processed and the horizontal vector feature is determined. Optionally, the second similarity can be determined using the following formula (3):
[0082]
[0083] Among them, A h F represents the second similarity; F represents the feature to be processed; F h Represents the characteristics of vertical vectors; and The query and key weights represent vertical attention; these two weights can be implemented using linear layers of the neural network (specific network parameters are provided in the original text). Figure 1a The image segmentation network training shown is determined); it can be understood that the formula (3) provided in this embodiment can be implemented by a neural network. For example, the linear layer corresponding to the query weight of the vertical attention processes the feature F to be processed, and the linear layer corresponding to the key weight of the vertical attention processes the horizontal vector feature F. v The process involves processing the results of the two linear layers using a softmax layer to obtain a second similarity. The second attention feature is the output feature aggregated from the feature to be processed and the vertical vector feature based on the second similarity, i.e., each point in the feature to be processed F and the vertical vector feature F. hCalculate the similarity of each point in the vector to obtain a second similarity, and then use the second similarity to analyze the vertical vector features F. h The features are extracted from the middle and filled into the corresponding positions of the features to be processed, thus obtaining the second attention features. Therefore, the second attention features have the same size as the features to be processed. Alternatively, the second attention features can be determined using the following formula (4):
[0084]
[0085] Among them, O h Indicates the second attention feature; A h Indicates the second similarity; The value weights representing vertical attention can be implemented using linear layers of a neural network (specific network parameters are specified through...). Figure 1a The image segmentation network training shown is determined); F h This represents the characteristics of a vertical vector.
[0086] Step 2023: Determine global attention features based on the first attention features and the second attention features.
[0087] In this embodiment, the first attention feature and the second attention feature represent the attention output in the horizontal and vertical directions, respectively. During the attention calculation process, only the attention operation between each pixel and the two horizontal and vertical strip features (horizontal vector features and vertical vector features) is calculated, which pays more attention to global context information and reduces redundant calculations. Furthermore, by combining the first attention feature and the second attention feature, a global attention feature that pays more attention to global context information is obtained. Optionally, the first attention feature and the second attention feature are matrix summed to determine the global attention feature.
[0088] Since the first attention feature and the feature to be processed have the same size, and the second attention feature also has the same size, the first attention feature and the second attention feature have the same size. In this embodiment, by performing matrix summation on the first attention feature and the second attention feature, the horizontal attention output and the vertical attention output are integrated to obtain a global attention feature that pays more attention to global context information.
[0089] Optionally, in the above Figure 2 Based on the illustrated embodiment, step 103 may include:
[0090] The global attention features are convolved with a convolution kernel of a preset size to obtain the segmentation result of the image to be processed.
[0091] In this embodiment, a convolution operation is performed on the global attention features using a convolution kernel of a preset size. The parameter values in the convolution kernel in this embodiment can be determined by combining... Figure 1a The image segmentation network training method shown is determined, for example, by using a 1*1 convolution kernel. The resulting segmentation is the semantic segmentation of the image to be processed. This method distinguishes pixel regions belonging to different objects in the image and labels the categories corresponding to different objects, achieving pixel-level segmentation. Because the global attention feature focuses more on global contextual information, the segmentation results can achieve better technical performance in specific application scenarios (e.g., urban street scene segmentation tasks), and the segmentation speed is faster.
[0092] like Figure 5 As shown above, in the above Figure 2 Based on the illustrated embodiment, step 203 may further include the following steps:
[0093] Step 2031: Process the global attention features and the features to be processed to obtain combined features.
[0094] Optionally, the global attention features and the features to be processed can be superimposed or concatenated by matrices. The resulting combined features can reflect the global context while also integrating the overall image features corresponding to the image to be processed.
[0095] Step 2032: Perform a convolution operation between the combined features and a convolution kernel of a preset size to determine the segmentation result of the image to be processed.
[0096] In this embodiment, the process of performing convolution operation on the combined features is similar to the process of performing convolution operation on the global attention features in the above embodiments. The only difference is that the object of the convolution operation is different. By performing convolution operation on the combined features, the obtained segmentation result incorporates more overall features in the image to be processed, thereby improving the accuracy of the segmentation result.
[0097] Any of the image segmentation methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the image segmentation methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the image segmentation methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.
[0098] Exemplary device
[0099] Figure 6 This is a schematic diagram of the structure of an image segmentation apparatus provided in an exemplary embodiment of this disclosure. Figure 6 As shown, the apparatus provided in this embodiment includes:
[0100] The feature extraction module 61 is used to process the features to be processed corresponding to the image to be processed, and obtain horizontal vector features and vertical vector features.
[0101] The attention determination module 62 is used to obtain global attention features based on the features to be processed, horizontal vector features, and vertical vector features determined by the feature extraction module 61.
[0102] The image segmentation module 63 is used to determine the segmentation result of the image to be processed based on the global attention features determined by the attention determination module 62.
[0103] The image segmentation apparatus provided in the above embodiments of this disclosure processes the features to be processed corresponding to the image to be processed to obtain horizontal vector features and vertical vector features; based on the features to be processed, the horizontal vector features, and the vertical vector features, global attention features are obtained; and the segmentation result of the image to be processed is determined based on the global attention features. The embodiments of this disclosure accelerate the image segmentation speed by extracting horizontal vector features and vertical vector features, solve the problems of large computational and storage overhead caused by directly calculating the attention mechanism based on image features, and the problem of a large amount of redundant computation in the context modeling of the attention mechanism, thereby achieving the elimination of redundancy and the reduction of resource consumption.
[0104] Figure 7 This is a schematic diagram of the structure of an image segmentation apparatus provided in another exemplary embodiment of this disclosure. For example... Figure 7 As shown, in the device provided in this embodiment, the feature extraction module 61 includes:
[0105] The first reduction unit 611 is used to perform a first preset reduction operation on the feature to be processed in the vertical direction to obtain the horizontal vector feature.
[0106] The second reduction unit 612 is used to perform a second preset reduction operation on the feature to be processed in the horizontal direction to obtain a vertical vector feature.
[0107] Optionally, the first reduction unit 611 is specifically used to perform a convolution operation on the feature to be processed and a first preset vertical vector to obtain a first convolution feature; and to perform a first preset reduction operation on the first convolution feature in the vertical direction to obtain a horizontal vector feature.
[0108] Optionally, the second reduction unit 612 is specifically used to perform a convolution operation on the feature to be processed and a first preset horizontal vector to obtain a second convolution feature; and to perform a second preset reduction operation on the second convolution feature in the horizontal direction to obtain a vertical vector feature.
[0109] In some optional embodiments, the attention determination module 62 includes:
[0110] The first attention unit 621 is used to perform attention operations on the feature to be processed and the horizontal vector feature to obtain the first attention feature.
[0111] The second attention unit 622 is used to perform attention calculation on the feature to be processed and the vertical vector feature to obtain the second attention feature.
[0112] The global attention unit 623 is used to determine global attention features based on the first attention features and the second attention features.
[0113] Optionally, the global attention unit 623 is specifically used to perform matrix summation on the first attention feature and the second attention feature to determine the global attention feature.
[0114] In this embodiment, the image segmentation module 63 includes:
[0115] The feature combination unit 631 is used to process the global attention features and the features to be processed to obtain combined features.
[0116] The segmentation result unit 632 is used to perform a convolution operation on the combined features and a convolution kernel of a preset size to determine the segmentation result of the image to be processed.
[0117] In some alternative embodiments, the image segmentation module 63 can also be used to perform a convolution operation between the global attention features and a convolution kernel of a preset size to obtain the segmentation result of the image to be processed.
[0118] Exemplary electronic devices
[0119] Below, for reference Figure 8 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device 100 and a second device 200, or a standalone device independent of them, which may communicate with the first and second devices to receive acquired input signals from them.
[0120] Figure 8 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.
[0121] like Figure 8 As shown, the electronic device 80 includes one or more processors 81 and memory 82.
[0122] The processor 81 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 80 to perform desired functions.
[0123] The memory 82 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 81 may execute the program instructions to implement the image segmentation methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.
[0124] In one example, the electronic device 80 may also include an input device 83 and an output device 84, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).
[0125] For example, when the electronic device is a first device 100 or a second device 200, the input device 83 can be the aforementioned microphone or microphone array for capturing the input signal from the sound source. When the electronic device is a standalone device, the input device 83 can be a communication network connector for receiving the acquired input signals from the first device 100 and the second device 200.
[0126] In addition, the input device 83 may also include, for example, a keyboard, a mouse, etc.
[0127] The output device 84 can output various information to the outside, including determined distance information, direction information, etc. The output device 84 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0128] Of course, for the sake of simplicity, Figure 8 Only some of the components of the electronic device 80 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 80 may include any other suitable components depending on the specific application.
[0129] Exemplary computer program products and computer-readable storage media
[0130] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image segmentation methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.
[0131] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this disclosure. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0132] Furthermore, embodiments of this disclosure may also be computer-readable storage media having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the image segmentation methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.
[0133] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0134] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0135] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0136] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0137] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.
[0138] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.
[0139] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0140] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
Claims
1. An image segmentation method, comprising: The image to be processed is processed to obtain horizontal vector features and vertical vector features, including: performing reduction operations on the features to be processed in the horizontal and vertical directions respectively to obtain the horizontal vector features and the vertical vector features; Attention is calculated based on the features to be processed and the horizontal vector features and the vertical vector features respectively to obtain global attention features; Based on the global attention features, the segmentation result of the image to be processed is determined; The process of performing attention calculations based on the features to be processed and the horizontal vector features and the vertical vector features respectively to obtain global attention features includes: The feature to be processed is combined with the horizontal vector feature to perform an attention operation to obtain the first attention feature; The feature to be processed and the vertical vector feature are subjected to attention calculation to obtain the second attention feature; Perform matrix summation on the first attention feature and the second attention feature to determine the global attention feature.
2. The method according to claim 1, wherein, The process involves processing the features corresponding to the image to be processed to obtain horizontal vector features and vertical vector features, including: Perform a first preset reduction operation on the feature to be processed in the vertical direction to obtain the horizontal vector feature; A second preset reduction operation is performed on the feature to be processed in the horizontal direction to obtain the vertical vector feature.
3. The method according to claim 2, wherein, The step of performing a first preset reduction operation on the feature to be processed in the vertical direction to obtain the horizontal vector feature includes: The feature to be processed is convolved with a first preset vertical vector to obtain the first convolutional feature; The first preset reduction operation is performed on the first convolutional feature in the vertical direction to obtain the horizontal vector feature.
4. The method according to claim 2, wherein, The step of performing a second preset reduction operation on the feature to be processed in the horizontal direction to obtain the vertical vector feature includes: The feature to be processed is convolved with the first preset horizontal vector to obtain the second convolutional feature; The second preset reduction operation is performed on the second convolutional feature in the horizontal direction to obtain the vertical vector feature.
5. The method according to any one of claims 1-4, wherein, Determining the segmentation result of the image to be processed based on the global attention features includes: The global attention features are convolved with a convolution kernel of a preset size to obtain the segmentation result of the image to be processed.
6. The method according to any one of claims 1-4, wherein, Determining the segmentation result of the image to be processed based on the global attention features includes: The global attention features and the features to be processed are combined to obtain the combined features; The combined features are convolved with a convolution kernel of a preset size to determine the segmentation result of the image to be processed.
7. An image segmentation apparatus, comprising: The feature extraction module is used to process the features corresponding to the image to be processed, and obtain horizontal vector features and vertical vector features; The feature extraction module is specifically used to perform reduction operations on the features to be processed in the horizontal and vertical directions respectively to obtain the horizontal vector features and the vertical vector features; An attention determination module is used to obtain global attention features based on the features to be processed, the horizontal vector features, and the vertical vector features determined by the feature extraction module. An image segmentation module is used to determine the segmentation result of the image to be processed based on the global attention features determined by the attention determination module. The attention determination module includes: The first attention unit is used to perform an attention operation on the feature to be processed and the horizontal vector feature to obtain the first attention feature; The second attention unit is used to perform attention calculation on the feature to be processed and the vertical vector feature to obtain the second attention feature; A global attention unit is used to perform matrix summation on the first attention feature and the second attention feature to determine the global attention feature.
8. A computer-readable storage medium storing a computer program for performing the image segmentation method according to any one of claims 1-6.
9. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image segmentation method according to any one of claims 1-6.