Image segmentation method, device, equipment and storage medium
By preprocessing the image and using the setting neural network model to obtain feature vectors and segmentation reference point offsets, the problem of inefficient image segmentation in the prior art is solved, and an efficient image segmentation effect is achieved, which is particularly suitable for mobile terminals.
Patent Information
- Application Number
- CN202111320252.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-09
- Publication Date
- 2025-08-26
- Estimated Expiration
- 2041-11-09
AI Technical Summary
The existing image segmentation technology is inefficient, especially the superpixels generated by simple linear iterative clustering algorithms do not contain semantic information, and deep learning methods are time-consuming.
By preprocessing the image to be segmented, the initial segmentation reference point position and basic information are obtained, the feature vector and segmentation reference point offset are obtained by setting the neural network model, image segmentation is performed according to the similarity, and the deep segmentable convolutional network is used to improve the computing efficiency.
The efficiency and accuracy of image segmentation are improved and are suitable for the image segmentation requirements of mobile terminals.
Smart Images
Figure CN114037715B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to an image segmentation method, apparatus, device, and storage medium. Background Art
[0002] Image segmentation technology divides an image into several image blocks. It can be applied in image editing to automatically segment images uploaded by users to facilitate subsequent editing.
[0003] Existing image segmentation techniques, such as the simple linear iterative clustering (SLIC) algorithm, segment images using underlying color features. The resulting superpixels lack semantic information and require parameter adjustment, resulting in low efficiency. Other image segmentation techniques that use deep learning are time-consuming. Summary of the Invention
[0004] The embodiments of the present disclosure provide an image segmentation method, apparatus, device, and storage medium, which can improve the efficiency of image segmentation.
[0005] In a first aspect, an embodiment of the present disclosure provides an image segmentation method, comprising:
[0006] Preprocessing the image to be segmented to obtain the position of the initial segmentation reference point and basic information of each pixel; wherein the basic information includes color RGB information, color space LAB information and coordinate information; the number of the segmentation reference points is multiple;
[0007] Input the basic information into a set neural network model to obtain the feature vector of each pixel and the offset of the segmentation reference point;
[0008] Determine the target segmentation reference point position according to the segmentation reference point offset and the initial segmentation reference point position;
[0009] Determine the similarity between each pixel and the target segmentation reference point according to the feature vector;
[0010] The image to be segmented is segmented based on the similarity to obtain a segmented image.
[0011] In a second aspect, the present disclosure further provides an image segmentation apparatus, comprising:
[0012] An initial segmentation reference point position acquisition module is used to pre-process the image to be segmented to obtain the initial segmentation reference point position and basic information of each pixel; wherein the basic information includes color RGB information, color space LAB information and coordinate information; the number of the segmentation reference points is multiple;
[0013] A segmentation reference point offset acquisition module is used to input the basic information into a set neural network model to obtain the feature vector of each pixel point and the segmentation reference point offset;
[0014] a target segmentation reference point position determination module, configured to determine the target segmentation reference point position according to the segmentation reference point offset and the initial segmentation reference point position;
[0015] A similarity determination module is used to determine the similarity between each pixel point and the target segmentation reference point according to the feature vector;
[0016] The image segmentation module is used to segment the image to be segmented based on the similarity to obtain a segmented image.
[0017] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:
[0018] one or more processing devices;
[0019] a storage device for storing one or more programs;
[0020] When the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the image segmentation method as described in the embodiment of the present disclosure.
[0021] In a fourth aspect, the present disclosure further provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the image segmentation method as described in the embodiment of the present disclosure is implemented.
[0022] The embodiments of the present disclosure disclose an image segmentation method, apparatus, device and storage medium. The image to be segmented is pre-processed to obtain the initial segmentation reference point position and basic information of each pixel point; wherein the basic information includes color RGB information, color space LAB information and coordinate information; the number of segmentation reference points is multiple; the basic information is input into a set neural network model to obtain the feature vector of each pixel point and the segmentation reference point offset; the target segmentation reference point position is determined according to the segmentation reference point offset and the initial segmentation reference point position; the similarity between each pixel point and the target segmentation reference point is determined according to the feature vector; the image to be segmented is segmented based on the similarity to obtain a segmented image. The image segmentation method provided by the embodiments of the present disclosure segments an image based on each similarity and the similarity between the segmentation reference points, which can improve the efficiency of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 is a flow chart of an image segmentation method according to an embodiment of the present disclosure;
[0024] Figure 2 is a structural diagram of a neural network model set in an embodiment of the present disclosure;
[0025] Figure 3 is a structural diagram of an image segmentation device in an embodiment of the present disclosure;
[0026] Figure 4 It is a structural diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0027] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.
[0028] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0029] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.
[0030] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0031] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".
[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0033] Figure 1This is a flowchart of an image segmentation method provided in the first embodiment of the present disclosure. This embodiment is applicable to the case of image segmentation. The method can be performed by an image segmentation device, which can be composed of hardware and / or software and can generally be integrated into a device with image segmentation function, which can be an electronic device such as a server, mobile terminal or server cluster. Figure 1 As shown, the method specifically includes the following steps:
[0034] Step 110 : pre-process the image to be segmented to obtain the position of the initial segmentation reference point and basic information of each pixel.
[0035] The basic information may include color (Red, Green, Blue, RGB) information, color space (Lab colorspace, LAB) information, and coordinate information. The number of segmentation reference points is multiple. In LAB information, L represents brightness, and A and B represent color opposition dimensions, which are color space coordinates based on nonlinear compression. The coordinate information may be the row and column numbers of the pixel.
[0036] The image to be segmented can be any color image or grayscale image. The initial segmentation reference point positions can be pre-set segmentation reference point positions, and the initial segmentation reference points are evenly distributed in the entire image.
[0037] Optionally, the image to be segmented is preprocessed to obtain the initial segmentation reference point position by: gridding the segmented region image to obtain multiple grid regions; and determining the center point position of the grid region as the initial segmentation reference point position.
[0038] The grid division can be performed according to a preset size. For example, if the size is 10*10, the image to be segmented can be divided into 100 grid areas. If the size is 20*20, the image to be segmented can be divided into 400 grid areas. Specifically, after the image to be segmented is divided into multiple grid areas, the center point position of each grid area is determined, and the position of each center point is determined as the initial segmentation reference point position. In this embodiment, determining the grid center as the initial segmentation reference point can quickly determine the initial segmentation reference point, thereby helping to improve the speed of the entire image segmentation.
[0039] Step 120: Input the basic information into the set neural network model to obtain the feature vector of each pixel and the offset of the segmentation reference point.
[0040] In this embodiment, the image to be segmented needs to be resized before being input into the set neural network model. The size of the image to be segmented is adjusted to a size that the set neural network model can recognize, such as 256*256 or 512*512. The basic information is 8-channel data consisting of RGB information, LAB information, and coordinate information, and this 8-channel data is input into the set neural network.
[0041] The segmentation reference point offset may include a horizontal offset and a vertical offset.
[0042] In the disclosed embodiment, in order for the model to be deployed on a mobile terminal, the model needs to be computationally efficient, simple, and small. In the disclosed embodiment, the convolutional network is a depthwise separable convolutional network. Due to the advantages of a small structure and low computational complexity, the depthwise separable convolutional network can be easily deployed on a mobile terminal. In this embodiment, since the set vector field prediction model includes a depthwise separable convolutional network, basic pixel information such as RGB information, LAB information, and coordinate information is input into the set vector field prediction model. A smaller computational load can be used to determine the feature vector of each pixel and the offset of the segmentation reference point, thereby improving the efficiency of image segmentation.
[0043] Optional, Figure 2 This is a schematic diagram of a structure of a neural network model in an embodiment of the present disclosure. Figure 2 As shown, the neural network model is set to include: a channel exchange network, a channel splitting network and a depth-wise separable convolutional network. Among them, the depth-wise separable convolutional network includes a first channel convolutional subnetwork, a depth-wise convolutional subnetwork, a second channel convolutional subnetwork and a channel merging layer; the channel exchange network, the channel splitting network, the first channel convolutional subnetwork, the depth-wise convolutional subnetwork, the second channel convolutional subnetwork and the channel merging layer are connected in sequence; and the output of the channel splitting network is jump-connected to the input of the channel merging layer. Among them, the depth-wise convolutional subnetwork can improve the feature extraction capability of the set vector field prediction model, and the channel convolutional subnetwork and the depth-wise convolutional subnetwork have the advantages of small structure and small computational complexity. In this embodiment, since the depth-wise convolutional subnetwork has a high feature extraction capability, the basic information of the pixel points such as RGB information, LAB information and coordinate information is input into the set vector field prediction model, and the feature vector of each pixel point and the offset of the segmentation reference point can be accurately obtained, thereby improving the accuracy of image segmentation.
[0044] like Figure 2As shown in the figure, the first channel convolution subnetwork includes the first channel convolution layer, the nonlinear activation layer and the linear transformation layer; the depth convolution subnetwork includes the depth convolution layer (Depthwise Convolution), the nonlinear activation layer and the linear transformation layer; the second channel convolution subnetwork includes the second channel convolution layer (Pointwise Convolution), the nonlinear activation layer and the linear transformation layer; the depth convolution layer consists of multiple parallel convolution kernels.
[0045] Among them, the first channel convolution layer and the second channel convolution layer can both be composed of 1×1 convolution kernels. The depth convolution layer can be composed of 3×3 convolution kernels, and the 3×3 convolution kernel is composed of three parallel convolution kernels. The sizes of the three parallel convolution kernels are divided into 3×3, 3×1 and 1×3. The 3×3 convolution kernel is implemented by three parallel convolution kernels, which can improve the calculation speed of the model. The channel exchange network can be implemented by the channel shuffle method, the nonlinear activation layer can be implemented by the linear rectification function (Rectified Linear Unit, ReLU), and the linear transformation layer can be implemented by the batch normalization (BatchNormalization, BN) algorithm. The setting of the neural network model provided in this embodiment has low working time consumption and can be applied to mobile terminals with high time consumption requirements.
[0046] In the embodiment of the present disclosure, the training method of the neural network model is set as follows: clustering the sample image using a superpixel algorithm to obtain the actual segmentation reference point position; gridding the sample image, and determining the center position of the grid after division as the initial segmentation reference point position; determining the actual offset based on the actual segmentation reference point position and the initial segmentation reference point position, and extracting the actual feature vector of each pixel in the sample image; and training the set neural network model based on the actual offset and the actual feature vector.
[0047] Among them, the superpixel algorithm can be a simple linear iterative cluster (SLIC) algorithm. The principle of the SLIC algorithm can be found in the prior art and will not be repeated here. Among them, the grid division can be divided according to a preset size. For example, if the size is 10*10, the image to be segmented can be divided into 100 grid areas. If the size is 20*20, the image to be segmented can be divided into 400 grid areas. Specifically, after dividing the image to be segmented into multiple grid areas, the center point position of each grid area is determined, and the position of each center point is determined as the initial segmentation reference point position.
[0048] The actual feature vector may be extracted by analyzing the sample image using a set feature extraction algorithm to obtain the actual feature vector of each pixel.
[0049] In this embodiment, the process of training the set neural network model based on the actual offset and the actual feature vector can be: obtaining 8-channel information (including RGB information, LAB information and coordinate information) of each pixel of the sample image, inputting the 8-channel information into the set neural network model, obtaining the predicted offset and the predicted feature vector, determining the first loss function according to the predicted offset and the actual offset, determining the second loss function according to the predicted feature vector and the actual feature vector, and training the set neural network model based on the first loss function and the second loss function. Among them, the loss function can be a mean square error (MSE) function or a mean absolute error (MAE) function. This embodiment does not limit the choice of loss function. By training the set neural network model, the set neural network accurately determines the feature vector of each pixel and the offset of the segmentation reference point.
[0050] Step 130: Determine the target segmentation reference point position according to the segmentation reference point offset and the initial segmentation reference point position.
[0051] Specifically, after obtaining the horizontal offset and vertical offset of the segmentation reference point, the horizontal component of the initial segmentation reference point position is added to the horizontal offset, and the vertical component of the initial segmentation reference point position is added to the vertical offset to obtain the final target segmentation reference point position.
[0052] Step 140: Determine the similarity between each pixel and the target segmentation reference point based on the feature vector.
[0053] The similarity can be represented by the Euclidean distance or Mahalanobis distance between feature vectors.
[0054] Specifically, the process of determining the similarity between each pixel point and the target segmentation reference point based on the feature vector can be: traversing the pixel points in the image to be segmented, determining the distance between the traversed pixel points and each target segmentation reference point based on the coordinate information and the position of the target segmentation reference point; determining the candidate segmentation reference point from the target segmentation reference point whose distance to the pixel point is less than a set threshold; and determining the similarity between the pixel point and the candidate segmentation reference point based on the feature vector.
[0055] In this embodiment, there are multiple target segmentation reference points. The distances between each traversed pixel and each target segmentation reference point are calculated based on the coordinate information and the target segmentation reference point locations. Target segmentation reference points with distances less than a set threshold are then identified as candidate segmentation reference points. This approach has the advantage of only calculating the similarity between a pixel and a candidate segmentation reference point. If the distance between a target segmentation reference point and a pixel exceeds the set threshold, the pixel will not be assigned to the region where the target segmentation reference point is located, even if the target segmentation reference point has the highest similarity with the pixel. This improves image segmentation accuracy.
[0056] Optionally, the method of determining the similarity between the pixel point and the candidate segmentation reference point based on the feature vector may be: calculating the distance between the feature vector of the pixel point and the feature vector of the candidate segmentation reference point; and determining the distance as the similarity.
[0057] The distance between the feature vectors may be a Euclidean distance or a Mahalanobis distance. Specifically, after obtaining the candidate segmentation reference point, the Euclidean distance or the Mahalanobis distance between the feature vector of the pixel point and the feature vector of the candidate segmentation reference point is calculated, and the Euclidean distance or the Mahalanobis distance is determined as the similarity between the two. In this embodiment, the similarity between the pixels is represented by the distance between the feature vectors of the pixel points, that is, image segmentation is achieved through the feature vector, which can further improve the accuracy of image segmentation.
[0058] Step 150 : Segment the image to be segmented based on the similarity to obtain a segmented image.
[0059] In this embodiment, the similarity between the pixel and each candidate segmentation reference point can be calculated based on the feature vector, and the similarities are compared to divide the pixel into the area where the candidate segmentation reference point with the greatest similarity is located, thereby improving the accuracy of image segmentation.
[0060] In this example, the number of segmented regions is the same as the number of segmentation reference points. Assuming that there are N segmentation reference points, the image can be segmented into N regions.
[0061] The technical solution of the embodiment of the present disclosure is to pre-process the image to be segmented to obtain the initial segmentation reference point position and basic information of each pixel point; wherein the basic information includes color RGB information, color space LAB information and coordinate information; the number of segmentation reference points is multiple; the basic information is input into a set neural network model to obtain the feature vector of each pixel point and the segmentation reference point offset; the target segmentation reference point position is determined based on the segmentation reference point offset and the initial segmentation reference point position; the similarity between each pixel point and the target segmentation reference point is determined based on the feature vector; the image to be segmented is segmented based on the similarity to obtain a segmented image. The image segmentation method provided by the embodiment of the present disclosure segments an image based on the similarity between each similarity and the similarity between the segmentation reference points, which can improve the efficiency of image segmentation.
[0062] Figure 3 This is a schematic diagram of the structure of an image segmentation device disclosed in an embodiment of the present disclosure. Figure 3 As shown, the device includes:
[0063] The initial segmentation reference point position acquisition module 210 is used to pre-process the image to be segmented to obtain the initial segmentation reference point position and basic information of each pixel; wherein the basic information includes color RGB information, color space LAB information and coordinate information; the number of segmentation reference points is multiple;
[0064] The segmentation reference point offset acquisition module 220 is used to input basic information into the set neural network model to obtain the feature vector of each pixel point and the segmentation reference point offset;
[0065] A target segmentation reference point position determination module 230 is configured to determine the target segmentation reference point position according to the segmentation reference point offset and the initial segmentation reference point position;
[0066] A similarity determination module 240 is used to determine the similarity between each pixel and the target segmentation reference point based on the feature vector;
[0067] The image segmentation module 250 is configured to segment the image to be segmented based on similarity to obtain a segmented image.
[0068] Optionally, the initial segmentation reference point position acquisition module 210 is further configured to:
[0069] Perform grid division on the image to be segmented to obtain multiple grid areas;
[0070] The center point of the grid area is determined as the initial segmentation reference point.
[0071] Optionally, the neural network model includes: channel exchange network, channel segmentation network and depthwise separable convolutional network;
[0072] Among them, the depth-wise separable convolutional network includes the first channel convolutional subnetwork, the depth-wise convolutional subnetwork, the second channel convolutional subnetwork and the channel merging layer;
[0073] The channel exchange network, channel splitting network, first channel convolution subnetwork, deep convolution subnetwork, second channel convolution subnetwork and channel merging layer are connected in sequence; and the output of the channel splitting network is skip-connected to the input of the channel merging layer;
[0074] The first channel convolution subnetwork includes the first channel convolution layer, the nonlinear activation layer and the linear transformation layer; the deep convolution subnetwork includes the deep convolution layer, the nonlinear activation layer and the linear transformation layer; the second channel convolution subnetwork includes the second channel convolution layer, the nonlinear activation layer and the linear transformation layer; the deep convolution layer consists of multiple parallel convolution kernels.
[0075] Optionally, the similarity determination module 240 is further configured to:
[0076] Traverse the pixel points in the image to be segmented, and determine the distance between the traversed pixel points and each target segmentation reference point based on the coordinate information and the position of the target segmentation reference point;
[0077] The target segmentation reference point whose distance from the pixel point is less than the set threshold is determined as a candidate segmentation reference point;
[0078] Determine the similarity between the pixel point and the candidate segmentation reference point based on the feature vector.
[0079] Optionally, the similarity determination module 240 is further configured to:
[0080] Calculate the distance between the feature vector of the pixel point and the feature vector of the candidate segmentation reference point;
[0081] The distance is determined as the similarity.
[0082] Optionally, the image segmentation module 250 is further configured to:
[0083] The pixels are divided into the region where the candidate segmentation reference point with the greatest similarity is located to obtain the segmented image.
[0084] Optionally, it also includes: setting a neural network model training module for:
[0085] Use superpixel algorithm to cluster sample images and obtain the actual segmentation reference point position;
[0086] Divide the sample image into grids, and determine the center position of the divided grid as the initial segmentation reference point position;
[0087] Determine the actual offset based on the actual segmentation reference point position and the actual segmentation reference point position, and extract the actual feature vector of each pixel in the sample image;
[0088] The set neural network model is trained based on the actual offset and the actual feature vector.
[0089] The above device can execute the methods provided by all the above embodiments of the present disclosure, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not fully described in this embodiment, please refer to the methods provided by all the above embodiments of the present disclosure.
[0090] Reference below Figure 4 , which shows a schematic structural diagram of an electronic device 300 suitable for implementing the embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), etc., fixed terminals such as digital TVs, desktop computers, etc., or various forms of servers, such as independent servers or server clusters. Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.
[0091] like Figure 4 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory device (ROM) 302 or a program loaded from a storage device 305 into a random access memory device (RAM) 303. Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0092] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 4 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.
[0093] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing a word recommendation method. In such an embodiment, the computer program can be downloaded and installed from a network via the communication device 309, or installed from the storage device 305, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
[0094] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.
[0095] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0096] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0097] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device is enabled to: pre-process the image to be segmented to obtain the initial segmentation reference point position and basic information of each pixel point; wherein, the basic information includes color RGB information, color space LAB information and coordinate information; the number of the segmentation reference points is multiple; input the basic information into the set neural network model to obtain the feature vector of each pixel point and the segmentation reference point offset; determine the target segmentation reference point position according to the segmentation reference point offset and the initial segmentation reference point position; determine the similarity between each pixel point and the target segmentation reference point according to the feature vector; segment the image to be segmented based on the similarity to obtain the segmented image.
[0098] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0100] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.
[0101] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.
[0102] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0103] According to one or more embodiments of the present disclosure, the present disclosure discloses an image segmentation method, including:
[0104] Preprocessing the image to be segmented to obtain the position of the initial segmentation reference point and basic information of each pixel; wherein the basic information includes color RGB information, color space LAB information and coordinate information; the number of the segmentation reference points is multiple;
[0105] Input the basic information into a set neural network model to obtain the feature vector of each pixel and the offset of the segmentation reference point;
[0106] Determine the target segmentation reference point position according to the segmentation reference point offset and the initial segmentation reference point position;
[0107] Determine the similarity between each pixel and the target segmentation reference point according to the feature vector;
[0108] The image to be segmented is segmented based on the similarity to obtain a segmented image.
[0109] Furthermore, the image to be segmented is preprocessed to obtain the initial segmentation reference point position, including:
[0110] Performing grid division on the image to be segmented to obtain a plurality of grid areas;
[0111] The center point position of the grid area is determined as the initial segmentation reference point position.
[0112] Furthermore, the set neural network model includes: a channel exchange network, a channel segmentation network and a depth-separable convolutional network;
[0113] The depth-wise separable convolutional network includes a first channel convolutional subnetwork, a depth-wise convolutional subnetwork, a second channel convolutional subnetwork, and a channel merging layer.
[0114] The channel exchange network, the channel splitting network, the first channel convolution subnetwork, the deep convolution subnetwork, the second channel convolution subnetwork and the channel merging layer are connected in sequence; and the output of the channel splitting network is jump-connected to the input of the channel merging layer;
[0115] The first channel convolution subnetwork includes a first channel convolution layer, a nonlinear activation layer and a linear transformation layer; the deep convolution subnetwork includes a deep convolution layer, a nonlinear activation layer and a linear transformation layer; the second channel convolution subnetwork includes a second channel convolution layer, a nonlinear activation layer and a linear transformation layer; the deep convolution layer is composed of multiple parallel convolution kernels.
[0116] Furthermore, determining the similarity between each pixel point and the target segmentation reference point according to the feature vector includes:
[0117] Traversing the pixel points in the image to be segmented, and determining the distance between the traversed pixel points and each target segmentation reference point according to the coordinate information and the position of the target segmentation reference point;
[0118] Determine a candidate segmentation reference point based on a target segmentation reference point whose distance from the pixel point is less than a set threshold;
[0119] The similarity between the pixel point and the candidate segmentation reference point is determined according to the feature vector.
[0120] Furthermore, determining the similarity between the pixel point and the candidate segmentation reference point according to the feature vector includes:
[0121] Calculating the distance between the feature vector of the pixel point and the feature vector of the candidate segmentation reference point;
[0122] The distance is determined as the similarity.
[0123] Furthermore, segmenting the image to be segmented based on the similarity to obtain a segmented image includes:
[0124] The pixel points are divided into the region where the candidate segmentation reference point with the greatest similarity is located to obtain a segmented image.
[0125] Furthermore, the training method of the neural network model is set as follows:
[0126] Use superpixel algorithm to cluster sample images and obtain the actual segmentation reference point position;
[0127] Dividing the sample image into grids, and determining the center position of the divided grid as the initial segmentation reference point position;
[0128] Determining an actual offset according to the actual segmentation reference point position and the initial segmentation reference point position, and extracting an actual feature vector of each pixel in the sample image;
[0129] The set neural network model is trained based on the actual offset and the actual feature vector.
[0130] Note that the above are only preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the present disclosure is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure has been described in more detail through the above embodiments, the present disclosure is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the appended claims.
Claims
1. An image segmentation method, characterized in that: include: Preprocessing the image to be segmented to obtain the position of the initial segmentation reference point and basic information of each pixel; wherein the basic information is 8-channel data consisting of color RGB information, color space LAB information and coordinate information; the number of the segmentation reference points is multiple; Input the basic information into a set neural network model to obtain the feature vector of each pixel and the offset of the segmentation reference point; Determine the target segmentation reference point position according to the segmentation reference point offset and the initial segmentation reference point position; Determine the similarity between each pixel and the target segmentation reference point according to the feature vector; Segmenting the image to be segmented based on the similarity to obtain a segmented image; The set neural network model includes: a channel exchange network, a channel splitting network and a depth-separable convolutional network; wherein the depth-separable convolutional network includes a first channel convolutional subnetwork, a depth-separable convolutional subnetwork, a second channel convolutional subnetwork and a channel merging layer; The initial segmentation reference points are evenly distributed throughout the image to be segmented.
2. The method according to claim 1, characterized in that Preprocess the image to be segmented to obtain the initial segmentation reference point position, including: Performing grid division on the image to be segmented to obtain a plurality of grid areas; The center point position of the grid area is determined as the initial segmentation reference point position.
3. The method according to claim 1, characterized in that The channel exchange network, the channel splitting network, the first channel convolution subnetwork, the deep convolution subnetwork, the second channel convolution subnetwork and the channel merging layer are connected in sequence; and the output of the channel splitting network is jump-connected to the input of the channel merging layer.
4. The method according to claim 1, wherein Determining the similarity between each pixel point and the target segmentation reference point according to the feature vector includes: Traversing the pixel points in the image to be segmented, and determining the distance between the traversed pixel points and each target segmentation reference point according to the coordinate information and the position of the target segmentation reference point; Determine a candidate segmentation reference point based on a target segmentation reference point whose distance from the pixel point is less than a set threshold; The similarity between the pixel point and the candidate segmentation reference point is determined according to the feature vector.
5. The method according to claim 4, characterized in that Determining the similarity between the pixel point and the candidate segmentation reference point according to the feature vector includes: Calculating the distance between the feature vector of the pixel point and the feature vector of the candidate segmentation reference point; The distance is determined as the similarity.
6. The method according to claim 4, characterized in that Segmenting the image to be segmented based on the similarity to obtain a segmented image, including: The pixel points are divided into the region where the candidate segmentation reference point with the greatest similarity is located to obtain a segmented image.
7. The method according to claim 1, characterized in that The training method of the neural network model is set as follows: Use superpixel algorithm to cluster sample images and obtain the actual segmentation reference point position; Dividing the sample image into grids, and determining the center position of the divided grid as the initial segmentation reference point position; Determining an actual offset according to the actual segmentation reference point position and the initial segmentation reference point position, and extracting an actual feature vector of each pixel in the sample image; The set neural network model is trained based on the actual offset and the actual feature vector.
8. An image segmentation device, characterized in that: include: An initial segmentation reference point position acquisition module is used to pre-process the image to be segmented to obtain the initial segmentation reference point position and basic information of each pixel; wherein the basic information is 8-channel data consisting of color RGB information, color space LAB information and coordinate information; the number of segmentation reference points is multiple; A segmentation reference point offset acquisition module is used to input the basic information into a set neural network model to obtain the feature vector of each pixel point and the segmentation reference point offset; a target segmentation reference point position determination module, configured to determine the target segmentation reference point position according to the segmentation reference point offset and the initial segmentation reference point position; A similarity determination module is used to determine the similarity between each pixel point and the target segmentation reference point according to the feature vector; An image segmentation module, configured to segment the image to be segmented based on the similarity to obtain a segmented image; The set neural network model includes: a channel exchange network, a channel splitting network and a depth-separable convolutional network; wherein the depth-separable convolutional network includes a first channel convolutional subnetwork, a depth-separable convolutional subnetwork, a second channel convolutional subnetwork and a channel merging layer; The initial segmentation reference points are evenly distributed throughout the image to be segmented.
9. An electronic device, characterized in that: The electronic device comprises: one or more processing devices; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processing devices, the one or more processing devices implement the image segmentation method according to any one of claims 1 to 7.
10. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the image segmentation method according to any one of claims 1 to 7 is implemented.