Method and device for segmenting image

Abstract

The embodiment of the invention discloses an image segmentation method and device. Specifically, the image segmentation method comprises the following steps: obtaining a to-be-segmented image and first indication information, the first indication information being used for indicating a first object in the to-be-segmented image; performing feature extraction on the to-be-segmented image to obtain a first feature map; according to the first feature map, a first clustering category is determined, the first clustering category is a clustering category in at least one clustering category, and the at least one clustering category is in one-to-one correspondence with at least one network branch; and segmenting the to-be-segmented image through a first network branch corresponding to the first clustering category to obtain the first object, the first network branch being a network branch in the at least one network branch. According to the invention, based on the feature map of the to-be-segmented image, the to-be-segmented image is allocated to the corresponding first network branch for segmentation, so that the efficiency and precision of image segmentation can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a method and device for segmenting an image. Background Art

[0002] With the development of artificial intelligence technology, various image processing methods based on artificial intelligence have been widely used. Among them, the image segmentation method can divide the image to be segmented into several specific regions with unique properties, or segment one or more objects in the image to be segmented. In order to achieve accurate image segmentation, the image to be segmented can be manually annotated.

[0003] However, in order to improve the efficiency and accuracy of image segmentation, a large number of labeled images are often required for training, which requires a lot of time and effort. Based on this, how to improve the efficiency and accuracy of image segmentation needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide a method and device for segmenting an image, which can improve the efficiency and accuracy of image segmentation.

[0005] In a first aspect, a method for segmenting an image is provided, comprising: obtaining an image to be segmented and first indication information, wherein the first indication information is used to indicate a first object in the image to be segmented; performing feature extraction on the image to be segmented to obtain a first feature map; determining a first clustering category based on the first feature map, wherein the first clustering category is a clustering category in at least one clustering category, and the at least one clustering category corresponds one-to-one to at least one network branch; segmenting the image to be segmented through a first network branch corresponding to the first clustering category to obtain the first object, wherein the first network branch is a network branch in the at least one network branch.

[0006] In an embodiment of the present application, an image to be segmented is segmented based on a feature map of the image to be segmented, and at least one clustering category and at least one network branch are set, so that a suitable network branch is selected based on information about features in the image, thereby improving the efficiency and accuracy of image segmentation.

[0007] In some embodiments, determining the first cluster category based on the first feature graph includes: obtaining the distance between the first feature graph and the cluster center of at least one cluster category; and taking the cluster category corresponding to the case where the distance is the smallest as the first cluster category.

[0008] In an embodiment of the present application, the cluster category corresponding to the time when the distance between the first feature map and at least one cluster center is the smallest is used as the first cluster category, so that the image to be segmented is divided into the first cluster category, and the image is segmented using a suitable network branch, which can improve the accuracy of image segmentation.

[0009] In some embodiments, the first feature map includes multiple groups of data, and the distance between the first feature map and the cluster center includes: an average value of the distances between the multiple groups of data of the first feature map and the cluster center.

[0010] In an embodiment of the present application, when the first feature map includes multiple groups of data, the average value of the distances between the multiple groups of data and the cluster centers is used as the distance between the first feature map and the cluster centers, so that the first feature map corresponds to a more matching cluster category, thereby using a suitable network branch to segment the image, which can improve the accuracy of image segmentation, and the calculation process is simple, easy to implement, and short in time, which can also improve the efficiency of image segmentation.

[0011] In some embodiments, the at least one clustering category is obtained based on K-means clustering algorithm training.

[0012] In the embodiment of the present application, at least one clustering category is obtained by training based on the K-means clustering algorithm. The calculation process is simple and easy to implement, and the efficiency of image segmentation can be improved.

[0013] In some embodiments, the first indication information includes pixel information of the first object.

[0014] In the embodiment of the present application, the first indication information includes pixel information of the first object, so that the neural network can quickly identify the user's needs, which can improve the efficiency of image segmentation and the user experience at the same time.

[0015] In some embodiments, extracting features from the image to be segmented to obtain a first feature map includes: extracting at least one of the color features, texture features, shape features, and spatial relationship features of the image to be segmented through a backbone network to obtain the first feature map.

[0016] In the embodiment of the present application, various features of the image to be segmented are extracted through a backbone network to obtain a first feature map, so that a network branch can be selected based on the feature distribution of the image to be segmented to perform image segmentation, thereby improving the accuracy of image segmentation.

[0017] In some embodiments, the backbone network includes at least one of a residual network ResNet, a visual self-attention model VIT, and a high-resolution network HRNet.

[0018] In the embodiment of the present application, the backbone network includes a residual network, a visual self-attention model, and a high-resolution network, so that the above networks or models can be used for feature extraction, which can improve the efficiency and accuracy of image segmentation.

[0019] In some embodiments, the first object includes one or more objects.

[0020] In the embodiment of the present application, by setting the first object to include one or more objects, one or more objects can be segmented out at the same time, thereby improving the efficiency of image segmentation.

[0021] In some embodiments, the distance comprises a similarity.

[0022] In the embodiment of the present application, the above-mentioned distance includes similarity, so that cluster categories can be assigned based on similarity, and images with high similarity are assigned to the same cluster category, so that suitable network branches are used for image segmentation, which can improve the accuracy and efficiency of image segmentation.

[0023] In a second aspect, a device for segmenting an image is provided, comprising: an acquisition unit, used to acquire an image to be segmented and first indication information, wherein the first indication information is used to indicate a first object in the image to be segmented; a processing unit, used to perform feature extraction on the image to be segmented and acquire a first feature map; the processing unit is also used to determine a first clustering category based on the first feature map, the first clustering category is a clustering category in at least one clustering category, and the at least one clustering category corresponds one-to-one to at least one network branch; the processing unit is also used to segment the image to be segmented through a first network branch corresponding to the first clustering category to acquire the first object, and the first network branch is a network branch in the at least one network branch.

[0024] In an embodiment of the present application, an image to be segmented is segmented based on a feature map of the image to be segmented, and at least one clustering category and at least one network branch are set, so that a suitable network branch is selected based on information about features in the image, thereby improving the efficiency and accuracy of image segmentation.

[0025] In some embodiments, the processing unit is further used to: obtain the distances between the first feature graph and the cluster centers of at least one cluster category; and use the cluster category corresponding to the case where the distance is the smallest as the first cluster category.

[0026] In an embodiment of the present application, the cluster category corresponding to the time when the distance between the first feature map and at least one cluster center is the smallest is used as the first cluster category, so that the image to be segmented is divided into the first cluster category, and the image is segmented using a suitable network branch, which can improve the accuracy of image segmentation.

[0027] In some embodiments, the first feature map includes multiple groups of data, and the distance between the first feature map and the cluster center includes: an average value of the distances between the multiple groups of data of the first feature map and the cluster center.

[0028] In an embodiment of the present application, when the first feature map includes multiple groups of data, the average value of the distances between the multiple groups of data and the cluster centers is used as the distance between the first feature map and the cluster centers, so that the first feature map corresponds to a more matching cluster category, thereby using a suitable network branch to segment the image, which can improve the accuracy of image segmentation, and the calculation process is simple, easy to implement, and short in time, which can also improve the efficiency of image segmentation.

[0029] In some embodiments, the at least one clustering category is obtained based on K-means clustering algorithm training.

[0030] In the embodiment of the present application, at least one clustering category is obtained by training based on the K-means clustering algorithm. The calculation process is simple and easy to implement, and the efficiency of image segmentation can be improved.

[0031] In some embodiments, the first indication information includes pixel information of the first object.

[0032] In the embodiment of the present application, the first indication information includes pixel information of the first object, so that the neural network can quickly identify the user's needs, which can improve the efficiency of image segmentation and the user experience at the same time.

[0033] In some embodiments, the processing unit is further used to: extract at least one feature of the color feature, texture feature, shape feature and spatial relationship feature of the image to be segmented through a backbone network to obtain the first feature map.

[0034] In the embodiment of the present application, various features of the image to be segmented are extracted through a backbone network to obtain a first feature map, so that a network branch can be selected based on the feature distribution of the image to be segmented to perform image segmentation, thereby improving the accuracy of image segmentation.

[0035] In some embodiments, the backbone network includes at least one of a residual network ResNet, a visual self-attention model VIT, and a high-resolution network HRNet.

[0036] In the embodiment of the present application, the backbone network includes a residual network, a visual self-attention model, and a high-resolution network, so that the above networks or models can be used for feature extraction, which can improve the efficiency and accuracy of image segmentation.

[0037] In some embodiments, the first object includes one or more objects.

[0038] In the embodiment of the present application, by setting the first object to include one or more objects, one or more objects can be segmented out at the same time, which can improve the efficiency of image segmentation.

[0039] In some embodiments, the distance comprises a similarity.

[0040] In the embodiment of the present application, the above-mentioned distance includes similarity, so that cluster categories can be assigned based on similarity, and images with high similarity are assigned to the same cluster category, so that suitable network branches are used for image segmentation, which can improve the accuracy and efficiency of image segmentation.

[0041] In a third aspect, a device for segmenting an image is provided, comprising a processor and a memory, wherein the memory is used to store a program, and the processor is used to call and run the program from the memory to execute the method for segmenting an image in the above-mentioned first aspect or any possible embodiment of the first aspect.

[0042] In a fourth aspect, a computer-readable storage medium is provided, comprising a computer program. When the computer program is run on a computer, the computer executes the method for segmenting an image in the first aspect or any possible embodiment of the first aspect.

[0043] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed by a computer, causes the computer to execute the method for segmenting an image in the first aspect or any possible embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A system architecture diagram provided by an embodiment of the present application is shown.

[0045] Figure 2 A schematic flowchart of a method for segmenting an image provided in an embodiment of the present application is shown.

[0046] Figure 3 A schematic flowchart of another method for segmenting an image provided in an embodiment of the present application is shown.

[0047] Figure 4 A schematic block diagram of a device for segmenting an image according to an embodiment of the present application is shown.

[0048] Figure 5 A schematic diagram of the hardware structure of an image segmentation device according to another embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] The following detailed description and drawings of the embodiments of the present application are used to illustrate the principles of the present application, but cannot be used to limit the scope of the present application, that is, the present application is not limited to the described embodiments.

[0050] The embodiments of the present application may be applicable to image processing systems, including but not limited to products based on infrared imaging. The image processing system may be applied to various electronic devices with image processing functions, such as personal computers, computer workstations, smart phones, tablet computers, media consumption devices, wearable devices, set-top boxes, game consoles, augmented reality (AR) AR / virtual reality (VR) devices, vehicle-mounted terminals, etc., and the embodiments disclosed in the present application do not limit this.

[0051] It should be understood that the specific examples in this article are only intended to help those skilled in the art better understand the embodiments of the present application, rather than to limit the scope of the embodiments of the present application.

[0052] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0053] It should also be understood that the various implementation methods described in this specification can be implemented individually or in combination, and the embodiments of the present application are not limited to this.

[0054] Unless otherwise stated, all technical and scientific terms used in the embodiments of the present application have the same meaning as those generally understood by those skilled in the art of the present application. The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit the scope of this application. The term "and / or" used in this application includes any and all combinations of one or more related listed items.

[0055] In order to better understand the solution of the embodiment of the present application, Figure 1 The possible application scenarios of the embodiments of the present application are briefly introduced.

[0056] like Figure 1 As shown, the embodiment of the present application provides a system architecture 100. Figure 1 In the example, the data acquisition device 160 is used to acquire a sample image. For the image segmentation method of the embodiment of the present application, the sample image may be a sample image including one or more foregrounds, and the foreground and background areas in the sample image are known.

[0057] It is worth noting that in an image, optionally, the foreground can be understood as the part of the image that needs to be segmented, and the other parts except the foreground can be collectively referred to as the background.

[0058] After collecting sample images, the data collection device 160 stores these sample images in the database 130 , and the training device 120 obtains the target model / rule 101 through training based on the sample images maintained in the database 130 .

[0059] The above-mentioned target model / rule 101 can be used to implement the method for segmenting an image in the embodiment of the present application. The target model / rule 101 in the embodiment of the present application can specifically be a neural network. It should be noted that in actual applications, the sample images maintained in the database 130 do not necessarily all come from the collection of the data acquisition device 160, and may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 entirely based on the sample images maintained by the database 130, and it is also possible to obtain sample images from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0060] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 1 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, etc., or a server or a cloud. Figure 1 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include an image to be segmented input by the client device 140 in the embodiment of the present application.

[0061] In some implementations, the client device 140 may be the same device as the execution device 110 . For example, the client device 140 and the execution device 110 may both be terminal devices.

[0062] In other embodiments, the client device 140 may be a different device from the execution device 110. For example, the client device 140 is a terminal device, and the execution device 110 is a cloud device, a server, or other device. The client device 140 may interact with the execution device 110 through a communication network with any communication mechanism / communication standard. The communication network may be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0063] The calculation module 111 of the execution device 110 is used to process the input data (such as the image to be segmented) received by the I / O interface 112. When the calculation module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0064] Finally, the I / O interface 112 returns the processing result, such as the image segmentation result obtained above, to the client device 140 to provide it to the user.

[0065] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0066] exist Figure 1 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0067] It is worth noting that Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 1 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0068] like Figure 1As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 can be a neural network in the embodiment of the present application. Specifically, the neural network in the embodiment of the present application can be a convolutional neural network (CNN), a regional convolutional neural network (RCNN) or other types of neural networks, etc., and the present application does not make specific limitations on this.

[0069] In the description of the present application, it should be noted that, unless otherwise specified, "multiple" means more than two; the terms "upper", "lower", "left", "right", "inside", "outside", etc., indicating the orientation or positional relationship, are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. "Vertical" is not strictly vertical, but is within the allowable error range. "Parallel" is not strictly parallel, but is within the allowable error range.

[0070] The directional words appearing in the following description are all directions shown in the figures, and do not limit the specific structure of this application. In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0071] In the field of deep learning and computer vision, a large amount of labeled data is the key to training high-performance models. However, obtaining this labeled data usually requires a lot of human resources, especially in tasks that require precise annotation, such as image segmentation tasks. Image segmentation tasks require that each pixel in the image be accurately divided into a specific category, such as foreground or background. In other words, image segmentation tasks require high accuracy in image segmentation, and each pixel must be accurately distinguished to which category it belongs so that it can be accurately segmented. This fine-level annotation task usually takes a lot of time and effort.

[0072] In order to improve the efficiency of image segmentation, the interactive image segmentation method was introduced. The interactive image segmentation method allows the annotator to provide some initial annotations (such as foreground seed points, background seed points), and then the model can automatically complete the image segmentation task based on these initial annotations. However, the usual interactive image segmentation method relies on a fixed neural network structure and cannot adapt to the diversity and complexity of images, which will affect the efficiency and accuracy of image segmentation.

[0073] In view of this, an embodiment of the present application provides a method for segmenting an image, the method comprising: obtaining an image to be segmented and first indication information, the first indication information being used to indicate a first object in the image to be segmented; performing feature extraction on the image to be segmented to obtain a first feature map; determining a first clustering category based on the first feature map, the first clustering category being a clustering category in at least one clustering category, and at least one clustering category corresponding to at least one network branch; segmenting the image to be segmented through a first network branch corresponding to the first clustering category to obtain a first object, and the first network branch being a network branch in at least one network branch. Therefore, by segmenting the image to be segmented based on the feature map of the image to be segmented, and setting at least one clustering category and at least one network branch, a suitable network branch is selected based on information about features in the image, thereby improving the efficiency and accuracy of image segmentation.

[0074] Figure 2 A schematic flow chart of a method for segmenting an image provided by an embodiment of the present application is shown. The method 200 for segmenting an image includes:

[0075] S201, obtaining an image to be segmented and first indication information, where the first indication information is used to indicate a first object in the image to be segmented;

[0076] S202, performing feature extraction on the image to be segmented to obtain a first feature map;

[0077] S203, determining a first clustering category according to the first feature graph, where the first clustering category is a clustering category in at least one clustering category, and at least one network branch corresponds to at least one clustering category in a one-to-one manner;

[0078] S204, segmenting the image to be segmented by using a first network branch corresponding to the first clustering category to obtain a first object, where the first network branch is a network branch in at least one network branch.

[0079] Specifically, the first indication information may be annotation information input by the user, and the annotation information is used to indicate the object that the user desires to be segmented. For example, the user may click or select the object on the image to be segmented.

[0080] Optionally, when training the algorithm corresponding to the above method, multiple clustering categories can be obtained based on multiple sample image training, and each clustering category in the multiple clustering categories corresponds to a trained network branch, that is, the clustering categories correspond to the network branches one by one.

[0081] It is worth noting that the first feature map can be understood as feature data of the image to be segmented, and the feature data can represent, for example, the color distribution and size of the image, the distance between multiple objects in the image, the number of objects in the image, etc.

[0082] It is also worth noting that at least one network branch can be a branch of a dynamic neural network. Specifically, a dynamic neural network can be understood as a neural network that can adaptively adjust its own structure or parameters according to different inputs, as opposed to a static neural network or static model. Optionally, each network branch can be an interactive segmentation head, each interactive segmentation head including a series of convolutional layers, nonlinear activation functions, and segmentation prediction layers.

[0083] In addition, the clustering category may also be referred to as a clustering cluster, etc., which is not limited in this application.

[0084] Therefore, by segmenting the image to be segmented based on the feature map of the image to be segmented, and setting at least one clustering category and at least one network branch, a suitable network branch is selected based on the feature information in the image, thereby improving the efficiency and accuracy of image segmentation.

[0085] Optionally, the first clustering category may be one or more clustering categories. In addition, the first network branch may be one or more network branches, which is not limited in the present application.

[0086] In the embodiment of the present application, the above step S203 may be implemented as follows:

[0087] S2031, obtaining the distance between the first feature graph and the cluster center of at least one cluster category;

[0088] S2032: The cluster category corresponding to the case where the distance is the smallest is used as the first cluster category.

[0089] Specifically, at least one cluster category may be multiple cluster categories, and each cluster category in the multiple cluster categories corresponds to a cluster center. Thus, multiple cluster categories correspond to multiple cluster centers. In the above step S2031, the distance between the first feature map and each cluster center can be obtained. In the case of multiple cluster centers, multiple distances can be obtained. At this time, the cluster category corresponding to the minimum value of the multiple distances is the first cluster category.

[0090] It is worth noting that the feature map of an image can represent important visual information and structural attribute information of the image. Different images have different or similar visual information and structural attribute information, based on which images can be classified into different categories.

[0091] Therefore, by taking the cluster category corresponding to the time when the distance between the first feature map and at least one cluster center is the smallest as the first cluster category, the image to be segmented is divided into the first cluster category, and the image is segmented using a suitable network branch, which can improve the accuracy of image segmentation.

[0092] In an embodiment of the present application, the first feature map may include multiple groups of data, and the distance between the first feature map and the cluster center includes: the average value of the distances between the multiple groups of data of the first feature map and the cluster center.

[0093] Specifically, when calculating the distance between the first feature map and a cluster center, since the first feature map may include multiple groups of data, such as multiple vectors, the distance between each group of data in the multiple groups of data and the cluster center can be calculated separately to obtain multiple distance data, and then the average of the multiple distance data is used as the distance between the first feature map and the cluster center.

[0094] Therefore, when the first feature map includes multiple groups of data, the average value of the distances between the multiple groups of data and the cluster centers is used as the distance between the first feature map and the cluster centers, so that the first feature map corresponds to a more matching cluster category, thereby using a suitable network branch to segment the image, which can improve the accuracy of image segmentation, and the calculation process is simple, easy to implement, and short in time, which can also improve the efficiency of image segmentation.

[0095] In an embodiment of the present application, at least one clustering category is obtained by training based on a K-means clustering algorithm.

[0096] K-means clustering algorithm is a common clustering algorithm and an iterative algorithm. Specifically, the training process of the algorithm can be: 1. First, select K data points as the initial cluster centers. This can be a random selection or a certain strategy; 2. For each data in the data set, associate it with the nearest center point according to the distance from the K center points, and group all data points associated with the same center point into one group; 3. Calculate the average value of the data points in each group, move the center point associated with the group to the position of the average value, and use the average value as the new cluster center; 4. Repeat steps 2 and 3 until the center point (i.e., cluster center) no longer changes or changes slightly, or reaches a predefined number of iterations, then the algorithm converges and stops. It should be understood that the cluster center is constantly changing, or iteratively changing, during the training process.

[0097] Therefore, at least one clustering category is obtained by training based on the K-means clustering algorithm. The calculation process is simple and easy to implement, and the efficiency of image segmentation can be improved.

[0098] Optionally, at least one clustering category can also be obtained based on other types of clustering algorithms. For example, other types of partitioning clustering algorithms, density-based clustering algorithms (such as DBSCAN clustering algorithms), hierarchical clustering methods, quantum clustering algorithms, kernel clustering algorithms, spectral clustering algorithms, mean shift clustering algorithms, expectation maximization (EM) clustering algorithms using Gaussian mixture models (GMM), etc. This application is not limited to comparison.

[0099] Optionally, when at least one cluster category is obtained through K-means clustering algorithm training, the initial cluster center can be selected based on a certain strategy. For example, when the above method is used for image segmentation of batteries, images including only battery cells and images including battery modules can be used as different initial cluster centers, so that different cluster categories can be obtained. Or the selection of the initial cluster center can be based on user needs, which is not limited in this application.

[0100] In an embodiment of the present application, the first indication information includes pixel information of the first object.

[0101] Specifically, the pixel information of the first object may be determined based on the pixel points selected by the user on the image to be segmented. For example, the user may click on some pixel points of the first object on the image to be segmented, and the clicked pixel points may be called seed points.

[0102] Optionally, the first indication information may include parameter information of the first object. Specifically, the parameter information may be determined based on a parameter value input or selected by a user. For example, it may include RGB values, size data, etc. of the first object, or may also include selection items in a predefined option menu, etc. This application does not limit this.

[0103] Therefore, by including the pixel information of the first object in the first indication information, the neural network can quickly identify the user's needs, thereby improving the efficiency of image segmentation and the user experience.

[0104] In an embodiment of the present application, the specific implementation method of the above step S202 can be: extracting at least one feature of the color feature, texture feature, shape feature and spatial relationship feature of the image to be segmented through a backbone network to obtain a first feature map.

[0105] Therefore, the backbone network is used to extract various features of the image to be segmented and obtain the first feature map, so that the network branch can be selected based on the feature distribution of the image to be segmented to perform image segmentation, which can improve the accuracy of image segmentation.

[0106] In an embodiment of the present application, the backbone network includes at least one of a residual network (Residual Network, ResNet), a vision transformer (VIT), and a high-resolution network (High-Resolution Network, HRNet).

[0107] Specifically, the backbone network can be used for feature extraction, and can specifically include multiple types of networks or models, such as the above-mentioned networks or models. Optionally, the backbone network can also include other types of networks or models, which is not limited in this application.

[0108] Therefore, through the backbone network including the residual network, the visual self-attention model, and the high-resolution network, the above networks or models can be used for feature extraction, which can improve the efficiency and accuracy of image segmentation.

[0109] In an embodiment of the present application, the first object includes one or more objects.

[0110] Specifically, the method provided in the embodiment of the present application can segment out one or more objects in the image to be segmented.

[0111] It is worth noting that the above-mentioned object can refer to the foreground in the image, such as a cat, dog, tree, person, etc. For example, when an image includes a cat, a dog, and a person at the same time, the cat, the dog, the person, etc. in the image can be segmented out at the same time, and the cat can be segmented out alone.

[0112] Therefore, by setting the first object to include one or more objects, one or more objects can be segmented out at the same time, which can improve the efficiency of image segmentation.

[0113] In the embodiment of the present application, the distance between the first feature graph and the cluster center, and the distance between multiple groups of data of the first feature graph and the cluster center include similarity.

[0114] Specifically, a cluster category has a cluster center. The distance between the cluster center and the feature map, and the distance between the cluster center and other feature data can be understood as the similarity between the feature map or other features and the cluster category. The closer the distance, the higher the similarity.

[0115] Optionally, the distance may be a Euclidean distance, or may be other distance metrics.

[0116] It is worth noting that Euclidean distance is a measure used to represent the distance between two points. In two-dimensional or three-dimensional space, it can refer to "straight-line distance" or "spatial distance". For example, for two n-dimensional vectors A = (a1, a2, ..., a n ) and B=(b1,b2,...,b n ), the formula of Euclidean distance can be expanded to:

[0117]

[0118] Therefore, by including similarity in the above distance, clustering categories can be assigned based on similarity, and images with high similarity are assigned to the same clustering category, so that suitable network branches are used for image segmentation, which can improve the accuracy and efficiency of image segmentation.

[0119] In some embodiments, the cluster center may be a cluster center determined based on a K-means clustering algorithm, or may be a cluster center determined based on other clustering algorithms. The cluster center may be simply understood as a central point or average value in a cluster category.

[0120] Figure 3 A schematic flowchart of another method for segmenting an image provided in an embodiment of the present application is shown.

[0121] like Figure 3 As shown, the image segmentation method provided in the embodiment of the present application includes a training process and a testing process. In the training process, various neural networks for image segmentation are mainly trained. Specifically, first, the training data can be input into the backbone network, and the backbone network extracts features from the training data. The training data may include multiple sample images, such as different types of battery images; then, a clustering algorithm is used to train multiple clustering categories, such as Figure 3 Three cluster categories are trained, including cluster categories 1-3; secondly, different interactive segmentation heads are trained corresponding to different cluster categories, for example, interactive segmentation heads 1-3 are trained corresponding to cluster categories 1-3. The cluster categories to the interactive segmentation heads may pass through a fully connected layer, for example, fully connected layers 1-3.

[0122] In the test process, the test data can be input into the trained backbone network for feature extraction. The test data can be the above-mentioned image to be segmented. Then, based on the extracted features, such as the above-mentioned first feature map, the cluster category to which the test data should be assigned is determined. For example, it is determined which cluster center the extracted features are closest to, and then the cluster category corresponding to the cluster center closest to the extracted features is assigned. For example, Figure 3The distance between the test data and cluster category 1 is the shortest, so the test data can be segmented through the fully connected layer 1 and the interactive segmentation head 1 corresponding to cluster category 1.

[0123] It is worth noting that Figure 3 The network structure shown can be understood as having multiple network paths, each network path corresponds to a type of image, each network path corresponds to a clustering category and an interactive segmentation head, so that different types of images can be segmented. For example, due to the different contents of the images, some images are relatively simple to segment, while some are relatively complex. By setting different network paths to separate simple and complex categories, image segmentation can be better achieved and the image segmentation accuracy can also be improved.

[0124] Figure 4 A schematic block diagram of an apparatus 400 for segmenting an image according to an embodiment of the present application is shown. The apparatus 400 can execute the method for segmenting an image according to the embodiment of the present application, for example, the apparatus 400 can be the aforementioned execution device 110.

[0125] like Figure 4 As shown, the device comprises:

[0126] An acquisition unit 410 is used to acquire an image to be segmented and first indication information, where the first indication information is used to indicate a first object in the image to be segmented;

[0127] The processing unit 420 is used to extract features from the image to be segmented to obtain a first feature map;

[0128] The processing unit 420 is further configured to determine a first clustering category according to the first feature graph, where the first clustering category is a clustering category in at least one clustering category, and the at least one clustering category corresponds one-to-one to at least one network branch;

[0129] The processing unit 420 is further configured to segment the image to be segmented by using a first network branch corresponding to the first clustering category to obtain the first object, where the first network branch is a network branch in the at least one network branch.

[0130] For more detailed functions of the above-mentioned device 400, please refer to the relevant description in the implementation of the above-mentioned method, which will not be repeated here.

[0131] Figure 5 A schematic diagram of the hardware structure of an image segmentation device according to another embodiment of the present application is shown. Figure 5The image segmentation device 500 shown includes a memory 501 , a processor 502 , a communication interface 503 and a bus 504 . The memory 501 , the processor 502 , and the communication interface 503 are connected to each other through the bus 504 .

[0132] The memory 501 may be a read-only memory (ROM), a static storage device, and a random access memory (RAM). The memory 501 may store a program. When the program stored in the memory 501 is executed by the processor 502, the processor 502 and the communication interface 503 are used to execute the various steps of the method for segmenting an image in the embodiment of the present application.

[0133] The processor 502 can adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the image segmentation device of the embodiment of the present application, or to execute the image segmentation method of the embodiment of the present application.

[0134] The processor 502 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the image segmentation method of the embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 502 or by instructions in the form of software.

[0135] The processor 502 may also be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The methods, steps, and logic diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 501, and the processor 502 reads the information in the memory 501, and completes the functions required to be performed by the units included in the device for segmenting an image in the embodiment of the present application in combination with its hardware, or executes the method for segmenting an image in the embodiment of the present application.

[0136] The communication interface 503 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 500 and other devices or a communication network. For example, the traffic data of an unknown device can be obtained through the communication interface 503.

[0137] The bus 504 may include a path for transmitting information between various components of the device 500 (eg, the memory 501 , the processor 502 , and the communication interface 503 ).

[0138] It should be noted that although the above device 500 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the device 500 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 500 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 500 may also include only the devices necessary for implementing the embodiments of the present application, and does not necessarily include Figure 5 All devices shown in .

[0139] The embodiment of the present application further provides a computer-readable storage medium storing a program code for execution by a device, wherein the program code includes instructions for executing the steps in the above-mentioned method for segmenting an image.

[0140] An embodiment of the present application further provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned method for segmenting an image.

[0141] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0142] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0144] The words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular "one" and "said" are intended to include the plural form as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprising" refers to the existence of stated features, wholes, steps, operations, elements, and / or components, but does not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or these groupings.

[0145] The various aspects, implementations, implementations or features of the described embodiments can be used alone or in any combination. The various aspects of the described embodiments can be implemented by software, hardware or a combination of software and hardware. The described embodiments can also be embodied by a computer-readable medium storing a computer-readable code, the computer-readable code comprising instructions executable by at least one computing device. The computer-readable medium can be associated with any data storage device capable of storing data that can be read by a computer system. Computer-readable media for example can include read-only memory, random access memory, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), hard disk drive (Hard Disk Drive, HDD), digital video disc (Digital Video Disc, DVD), magnetic tape, and optical data storage device, etc. The computer-readable medium can also be distributed in computer systems connected via a network so that the computer-readable code can be stored and executed in a distributed manner.

[0146] The above technical description may refer to the accompanying drawings, which form a part of the present application, and the implementation methods according to the described embodiments are shown in the accompanying drawings by description. Although these embodiments are described in sufficient detail to enable those skilled in the art to implement these embodiments, these embodiments are non-restrictive; other embodiments can be used in this way, and changes can be made without departing from the scope of the described embodiments. For example, the order of operations described in the flowchart is non-restrictive, so the order of two or more operations illustrated in the flowchart and described according to the flowchart can be changed according to several embodiments. As another example, in several embodiments, one or more operations illustrated in the flowchart and described according to the flowchart are optional or deletable. In addition, certain steps or functions can be added to the disclosed embodiments, or the order of two or more steps can be replaced. All these changes are considered to be included in the disclosed embodiments and claims.

[0147] In addition, terms are used in the above technical description to provide a thorough understanding of the described embodiments. However, overly detailed details are not required to implement the described embodiments. Therefore, the above description of the embodiments is presented for the purpose of explanation and description. The embodiments presented in the above description and the examples disclosed according to these embodiments are provided separately to add context and help understand the described embodiments. The above description is not intended to be exhaustive or to limit the described embodiments to the precise form of the present application. According to the above teachings, several modifications, selective applications and changes are feasible. In some cases, well-known processing steps are not described in detail to avoid unnecessarily affecting the described embodiments.

[0148] Although the present application has been described with reference to the above embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, the various technical features mentioned in the various embodiments may be combined in any manner as long as there is no structural conflict. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for segmenting an image, characterized in that: include: Acquire an image to be segmented and first indication information, where the first indication information is used to indicate a first object in the image to be segmented; Performing feature extraction on the image to be segmented to obtain a first feature map; Determine a first clustering category according to the first feature graph, where the first clustering category is a clustering category in at least one clustering category, and the at least one clustering category corresponds one-to-one to at least one network branch; The image to be segmented is segmented by a first network branch corresponding to the first clustering category to obtain the first object, where the first network branch is a network branch in the at least one network branch.

2. The method according to claim 1, characterized in that The determining a first clustering category according to the first feature graph includes: Obtaining the distance between the first feature graph and the cluster center of at least one cluster category; The cluster category corresponding to the case where the distance is the smallest is used as the first cluster category.

3. The method according to claim 2, characterized in that The first feature map includes multiple groups of data, and the distance between the first feature map and the cluster center includes: The average value of the distances between the multiple groups of data of the first feature graph and the cluster center.

4. The method according to any one of claims 1 to 3, characterized in that The at least one clustering category is obtained based on K-means clustering algorithm training.

5. The method according to any one of claims 1 to 4, characterized in that The first indication information includes pixel information of the first object.

6. The method according to any one of claims 1 to 5, characterized in that The step of extracting features from the image to be segmented to obtain a first feature map includes: At least one feature among the color feature, texture feature, shape feature and spatial relationship feature of the image to be segmented is extracted through a backbone network to obtain the first feature map.

7. The method according to claim 6, characterized in that The backbone network includes at least one of a residual network ResNet, a visual self-attention model VIT, and a high-resolution network HRNet.

8. The method according to any one of claims 1 to 7, characterized in that The first object includes one or more objects.

9. The method according to claim 2 or 3, characterized in that: The distance includes similarity.

10. A device for segmenting an image, characterized in that: include: An acquisition unit, used for acquiring an image to be segmented and first indication information, wherein the first indication information is used for indicating a first object in the image to be segmented; A processing unit, configured to extract features from the image to be segmented to obtain a first feature map; The processing unit is further configured to determine a first clustering category according to the first feature graph, where the first clustering category is a clustering category in at least one clustering category, and the at least one clustering category corresponds one-to-one to at least one network branch; The processing unit is further configured to segment the image to be segmented by using a first network branch corresponding to the first clustering category to obtain the first object, wherein the first network branch is a network branch in the at least one network branch.

11. The device according to claim 10, characterized in that The processing unit is also used for: Obtaining the distance between the first feature graph and the cluster center of at least one cluster category; The cluster category corresponding to the case where the distance is the smallest is used as the first cluster category.

12. The device according to claim 11, characterized in that The first feature map includes multiple groups of data, and the distance between the first feature map and the cluster center includes: The average value of the distances between the multiple groups of data of the first feature graph and the cluster center.

13. The device according to any one of claims 10 to 12, characterized in that The at least one clustering category is obtained based on K-means clustering algorithm training.

14. The device according to any one of claims 10 to 13, characterized in that The first indication information includes pixel information of the first object.

15. The device according to any one of claims 10 to 14, characterized in that The processing unit is also used for: At least one feature among the color feature, texture feature, shape feature and spatial relationship feature of the image to be segmented is extracted through a backbone network to obtain the first feature map.

16. The device according to claim 15, characterized in that The backbone network includes at least one of a residual network ResNet, a visual self-attention model VIT, and a high-resolution network HRNet.

17. The device according to any one of claims 10 to 16, characterized in that The first object includes one or more objects.

18. The device according to claim 11 or 12, characterized in that The distance includes similarity.

19. A device for segmenting an image, characterized in that: The invention comprises a processor and a memory, wherein the memory is used to store a program, and the processor is used to call and run the program from the memory to execute the method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: The method comprises a computer program, which, when being run on a computer, causes the computer to execute the method according to any one of claims 1 to 9.

Images