Image segmentation apparatus and method
The electronic apparatus addresses the challenges of dynamic image segmentation by generating flexible segmentation data through adaptive class and cluster mapping, enhancing accuracy and efficiency in resource utilization and learning, suitable for vehicle navigation and augmented reality.
Patent Information
- Application Number
- JP2022089335
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-08-20
- Filing Date
- 2022-06-01
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2042-06-01
AI Technical Summary
Existing image segmentation technologies struggle to perform accurately in dynamic environments with unlabeled pixels and require significant resources and time for continuous learning, especially when new classes are added, leading to conflicts with previously learned knowledge.
An electronic apparatus that generates flexible and accurate segmentation data by extracting feature vectors from unlabeled pixels, using a processor to calculate class and cluster maps, and dynamically adjusts classifier and clustering layers based on environmental complexity, allowing for continuous learning and efficient addition of new classes.
Enables efficient image segmentation in dynamic environments with reduced resource and time requirements, while preventing conflicts between new and existing classes, and supports applications like vehicle navigation and augmented reality.
Smart Images

Figure 0007948512000001 
Figure 0007948512000002 
Figure 0007948512000003
Abstract
Description
TECHNICAL FIELD
[0001] The following disclosure relates to image segmentation technology.
BACKGROUND ART
[0002] Scene classification refers to the process of recognizing a natural scene and understanding the content of the scene. Humans recognize scenes with their eyes and identify prominent aspects of the scene (e.g., landmarks or people in front of tourist attractions). In the context of computer vision, scene classification involves capturing one or more images within a scene using a camera and identifying the elements of the scene. Semantic segmentation means the process of identifying parts or regions of an image that correspond to objects of a specific class (e.g., people, buildings, and trees).
SUMMARY OF THE INVENTION
[0003] An electronic apparatus according to an embodiment can generate segmentation data that is flexible for dynamic environments while being accurate, by extracting meaningful and rich feature vectors in association with unlabeled pixels (e.g., background) together with labeled pixels.
[0004] The electronic apparatus according to an embodiment can shorten the learning time and reduce required resources by performing continuous learning while retaining the learned knowledge of previous classes using labels for newly added classes, and can prevent degradation in learning caused by conflicts between newly added classes and the already learned background.
[0005] An electronic device according to one embodiment includes an image acquisition unit that acquires an input image, and a processor that extracts feature data from the input image based on a feature extraction layer, calculates one or more class maps from the extracted feature data based on a classifier layer, calculates one or more cluster maps from the extracted feature data based on a clustering layer, and generates image segmentation data based on the one or more calculated class maps and the one or more calculated cluster maps.
[0006] The processor can generate one or more class maps, one or more cluster maps, and image segmentation data based on the resolution of the input image.
[0007] The processor can determine that, in the one or more class maps, pixels classified as background belong to a cluster corresponding to any one of the one or more cluster maps.
[0008] The processor can exclude clustering results based on the one or more cluster maps for pixels in the image segmentation data that have been labeled based on the one or more class maps.
[0009] The processor can select one of several classifier layers based on the location of the electronic device, and calculate one or more class maps from the extracted feature data using the selected classifier layer.
[0010] While the processor calculates a class map using a first classifier layer having a first number of available classes, if the complexity of the surrounding environment increases in response to a change in the position of the electronic device, it may select a second classifier layer having a second number of available classes that is greater than the first number of available classes.
[0011] While the processor calculates a class map using a second classifier layer having a second number of available classes, if the complexity of the surrounding environment decreases in response to a change in the position of the electronic device, it may select a first classifier layer having a first number of available classes that is smaller than the second number of available classes.
[0012] The processor can select one of several clustering layers based on the location of the electronic device, and calculate one or more cluster maps using the selected clustering layer from the extracted feature data.
[0013] While the processor calculates one or more cluster maps using a first clustering layer having a first number of available clusters, if the complexity of the surrounding environment increases in response to a change in the location of the electronic device, it may select a second clustering layer having a second number of available clusters that is greater than the first number of available clusters.
[0014] While the processor calculates the one or more cluster maps using a second clustering layer having a second number of available clusters, if the complexity of the surrounding environment decreases in response to a change in the location of the electronic device, it may select a first clustering layer having a first number of available clusters that is smaller than the second number of available clusters.
[0015] The processor can update one or more combinations of the feature extraction layer, the classifier layer, and the clustering layer based on training data collected while the electronic device is in motion.
[0016] The processor can update the parameters of the classifier layer based on the ground truth labels of the collected training data and the loss between the one or more class maps calculated from the classifier layer.
[0017] The processor can update the parameters of the clustering layer using the training image of the collected training data and the loss calculated based on the enhanced video from the training image.
[0018] The processor can train a new classifier layer, which includes a previously constructed classifier layer and layers corresponding to additional classes, based on the collected training data.
[0019] The processor can obtain clustering results corresponding to the additional class from one or more cluster maps calculated by previously constructed clustering layers, and update the parameters of the new classifier layer based on the loss between the obtained clustering results and the class map calculated using the new classifier layer.
[0020] The electronic device is a vehicle, and includes the image acquisition unit, which acquires the input image relating to the surrounding scene of the vehicle when the vehicle is in motion, and the processor generates information relating to the vehicle's motion using the image segmentation data, and visually displays the information relating to the motion via a display.
[0021] The processor can use the image segmentation data to perform one or more combinations of speed control, acceleration control, and steering control of the vehicle while the vehicle is in motion.
[0022] The electronic device is an augmented reality device, and includes the electronic device, wherein the processor generates augmented reality content using the image segmentation data, and visually displays the generated augmented reality content to the user via an augmented reality display.
[0023] A method implemented by a processor according to an embodiment includes the steps of: extracting feature data from an input image based on a feature extraction layer; calculating one or more class maps from the extracted feature data based on a classifier layer; calculating one or more cluster maps from the extracted feature data based on a clustering layer; and generating image segmentation data based on the calculated one or more class maps and the calculated one or more cluster maps.
[0024] An electronic device according to an embodiment includes one or more processors, wherein the one or more processors are configured to acquire input data, extract feature data from the input data based on a feature extraction layer, select a clustering layer and a classifier layer based on a physical position of the electronic device, generate one or more class maps from the feature data based on the classifier layer, generate one or more cluster maps from the feature data based on the clustering layer, and generate segmentation data related to the input data based on the one or more cluster maps and the one or more class maps.
[0025] The input data may be image data relating to the surroundings of the electronic device.
[0026] The one or more processors may generate the one or more cluster maps and the one or more class maps using a machine learning model.
[0027] The machine learning model may include a neural network. Effects of the Invention
[0028] According to the electronic device of the embodiment, image segmentation can be performed on objects presented in a scene even in an environment where labels are not constructed for all objects appearing in the scene. Brief Description of the Drawings
[0029] [Figure 1] This figure illustrates image segmentation of an electronic device according to one embodiment. [Figure 2] This is a flowchart illustrating an image segmentation method according to one embodiment. [Figure 3] This is a diagram illustrating an image segmentation network according to one embodiment. [Figure 4] This figure illustrates the dynamic use of a model in response to the complexity of the surrounding environment according to one embodiment. [Figure 5] This figure illustrates the training of an image segmentation network according to one embodiment. [Figure 6] This figure illustrates the training of an image segmentation network according to one embodiment. [Figure 7] This block diagram shows the configuration of an electronic device according to one embodiment. [Modes for carrying out the invention]
[0030] The specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and can be modified in various ways. Therefore, the embodiments are not limited to any particular disclosure, and the scope of this specification includes modifications, equivalents, or substitutions that are part of the technical idea.
[0031] Terms such as "first" or "second" may be used to describe multiple components, but such terms should be interpreted solely for the purpose of distinguishing one component from others. For example, the first component may be named the second component, and similarly, the second component may also be named the first component.
[0032] When it is mentioned that one component is “linked” or “connected” to another component, it should be understood that it is directly linked to or connected to the other component, but that other components may be present in between.
[0033] A singular expression includes plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “has” should be understood to indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, without prejudice to the possibility of the presence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.
[0034] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this embodiment belongs. Commonly used, predefined terms should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless expressly defined herein.
[0035] The embodiments will be described in detail below with reference to the attached drawings. When describing with reference to the drawings, the same components will be given the same reference numerals regardless of the drawing number, and redundant explanations will be omitted.
[0036] Figure 1 is a diagram illustrating image segmentation of an electronic device according to one embodiment.
[0037] An electronic device 100 according to one embodiment generates image segmentation data 120 by dividing an input image into classes and / or clusters. Figure 1 shows an example in which the electronic device 100 is mounted on a vehicle. The electronic device 100 mounted on the vehicle includes an imaging module (e.g., a camera sensor) as an image acquisition unit that captures a scene 110 around the vehicle (e.g., in front). However, the image acquisition unit is not limited to an imaging module and may be a communication unit that receives images from an external imaging device.
[0038] The electronic device 100 generates an input image corresponding to the surrounding scene 110 by capturing the surrounding scene 110 via a camera sensor. For example, the image acquisition unit of the electronic device 100 can acquire an image of the surrounding scene 110 of the vehicle while the vehicle is in motion.
[0039] For example, the processor of an electronic device 100 mounted on a vehicle can generate information related to the vehicle's movement using image segmentation data 120. This information may include route guidance information, hazard warning information (e.g., accident information such as collisions), road condition information (e.g., road congestion), and surrounding environment information (e.g., distance maps to surrounding objects). For example, the electronic device 100 can identify objects from the image segmentation data 120, calculate the distance to the identified objects, and generate a distance map showing the distance to the identified objects. However, the examples of information related to the vehicle's movement are not limited to these. The electronic device 100 can visually display the information related to the vehicle's movement via a display. Figure 1 shows an example where the electronic device 100 outputs image segmentation data 120 to the vehicle's display panel.
[0040] Furthermore, the electronic device 100 mounted on the vehicle may perform autonomous driving using the aforementioned image segmentation data 120. The processor of the electronic device 100 may use the image segmentation data 120 to perform one or more combinations of speed control, acceleration control, and steering control when the vehicle is in motion. For example, the processor may calculate the distance to surrounding objects based on the image segmentation data 120, and based on the distance to the objects, it may change the vehicle's speed (e.g., increase or decrease), change the acceleration (e.g., increase or decrease), and change the steering, or one or more combinations of these.
[0041] In one embodiment, the electronic device 100 calculates a class map and a cluster map from an input image based on an image segmentation model in order to generate the aforementioned image segmentation data 120. The class map may be a set of elements that indicate the class labeled to each pixel of the input image, as it is a map that shows the classes labeled to the input image. For example, the element value of the class map corresponding to any class may indicate the probability that the pixel corresponding to that element belongs to that class. The electronic device 100 may calculate a class map individually for each of multiple classes. The cluster map is a set of elements that indicate the cluster mapped to each pixel of the input image, as it is a map that shows the result of clustering the input image. For example, the element value of the cluster map corresponding to any cluster indicates the probability that the pixel corresponding to that element belongs to that cluster. The class map and cluster map will be described in detail with reference to Figure 3 below.
[0042] The electronic device 100 can label classes for at least some areas of an input image. For example, the electronic device 100 labels a class for each pixel in the input image. The electronic device 100 labels pixels that do not correspond to the foreground or an object as background classes. The electronic device 100 can apply clustering results to pixels that are not labeled as foreground or object classes, for example, pixels that are labeled as background classes. As will be described later, the electronic device 100 may train both a classifier layer that outputs a class map and a clustering layer that outputs a cluster map. The electronic device 100 can train an image segmentation model to learn both class labeling and clustering. Therefore, the electronic device 100 can classify the input image into a limited number of classes while clustering the parts of the input image that show objects that deviate from the available classes. The electronic device 100 can dynamically generate image segmentation data 120 even if the available classes are limited. Even if the number of object types (e.g., people, objects, vehicles, animals, signs, and roads) that need to be recognized in a single scene 110 while the vehicle is in motion exceeds the number of available classes, the electronic device 100 can provide flexible image segmentation results via the aforementioned clustering layer.
[0043] In one embodiment, the electronic device 100 can upgrade its existing image segmentation model if it wishes to add other classes in addition to the currently available classes. For example, the electronic device 100 can be upgraded in its image segmentation model by switching an existing classifier layer to a new classifier layer that has an increased number of available classes. The upgrade of the image segmentation model will be explained with reference to Figure 6 below.
[0044] For reference, this specification primarily describes an example in which the electronic device 100 using an image segmentation model is mounted in a vehicle, but is not limited to this. The electronic device 100 may also be mounted in a device that provides augmented reality. The processor of the electronic device 100 can generate augmented reality content using image segmentation data 120. The augmented reality content may, as an example of content provided to the user in augmented reality, include content that is mapped to surrounding objects and / or backgrounds. The electronic device 100 can calculate the distance and / or position relative to the user to the identified objects and / or backgrounds using the image segmentation data 120, and generate the aforementioned augmented reality content using the calculated distance and / or position. For example, the electronic device 100 may generate augmented reality content (e.g., route guidance content such as a directional arrow) that has a position corresponding to an identified object (e.g., a road). The electronic device 100 visually displays the generated augmented reality content to the user via an augmented reality display. The display may include a HUD (Heads-Up Display). For example, the electronic device 100 can visually output augmented reality content to a determined position, as described above.
[0045] As described above, the electronic device 100 according to one embodiment can generate image segmentation data 120 for objects shown in scene 110 of the input image, even if labels are not constructed for all objects appearing in scene 110 of the input image. Furthermore, even after the image segmentation model is installed, the electronic device 100 can improve the efficiency of the cost and time required for learning by performing continuous learning based on previously learned knowledge for the image segmentation model. In addition, in continuous learning, the electronic device 100 can prevent conflicts between already learned knowledge (e.g., existing usable classes) and newly learned knowledge (e.g., newly added classes) regarding parts without object classes (e.g., background classes). Continuous learning will be explained with reference to Figure 6 below.
[0046] Figure 2 is a flowchart illustrating an image segmentation method according to one embodiment. Figure 3 is a diagram illustrating an image segmentation network according to one embodiment.
[0047] The electronic device generates a class map 325 and a cluster map 335 from an input image 301 using an image segmentation model 300, as described later. The image segmentation model 300 may include a neural network as a machine learning model designed and trained to output a class map 325 and a cluster map 335 from an image.
[0048] A neural network is an example of a deep neural network (DNN). DNNs include fully connected networks, deep convolutional networks, and recurrent neural networks. Neural networks can perform object classification, object recognition, and image recognition by mapping non-linearly related input and output data to each other based on deep learning, and in this specification, they primarily perform image segmentation. Image segmentation is the operation of identifying parts and / or regions of an image that show the same element (e.g., an object or background), and exemplifies the operation of labeling pixels that show the same element to the same class (e.g., class labeling) and clustering pixels that show the same element to the same cluster. For reference, clustering may be interpreted as the operation of pseudo-labeling pixels corresponding to parts and / or regions identified as showing the same element as an undefined class.
[0049] Deep learning is a machine learning method for solving problems such as image segmentation from big datasets, and it can map input and output data to each other through supervised or unsupervised learning. While this specification primarily describes examples where neural networks are trained using supervised learning, it is not limited to this. Training is described with reference to Figures 5 and 6 below.
[0050] A neural network of an image segmentation model 300 according to one embodiment can represent a model of a machine learning structure designed to extract feature data from input data and provide inference operations using the feature data. Feature data represents data relating to features of the input data (e.g., input image 301) that are abstracted. If the input data is an image, the feature data may be represented as data abstracting the image, for example, in the form of vectors. In this specification, inference operations using feature data include class labeling and clustering. Class labeling represents the operation of labeling the class to which each pixel belongs for each pixel of the input image 301. Clustering represents the operation of clustering pixels of the input image 301 that represent the same or similar objects to each other into the same cluster.
[0051] For example, the image segmentation model 300 shown in Figure 3 includes a neural network, which includes a feature extraction layer 310, a classifier layer 320, and a clustering layer 330. The feature extraction layer 310, as described above, is a layer that extracts feature data from the input data and may, for example, include a convolutional layer for convolution operations. The convolutional layer is a layer to which a convolution operation based on a kernel filter is applied to the data. The classifier layer 320 may be a layer that outputs a class map 325 from the aforementioned feature data. The clustering layer 330 may be a layer that outputs a cluster map 335 from the aforementioned feature data. The layer configuration of the feature extraction layer 310, the classifier layer 320, and the clustering layer 330 is not limited and may include various combinations of various layers such as convolutional layers, pooling layers, dense layers, and fully connected layers.
[0052] As shown in Figure 3, the aforementioned classifier layer 320 and clustering layer 330 may be connected as subsequent layers to the feature extraction layer 310. In other words, the classifier layer 320 and clustering layer 330 can receive common feature data extracted from the feature extraction layer 310. The classifier layer 320 and clustering layer 330 may each contain one or more layers, and each layer may contain one or more nodes.
[0053] For reference, the nodes of a layer may be connected to the nodes of the next layer via links. The number of links corresponds to the number of nodes in the next layer, and each link is assigned a weight value. Links may also be referred to as connecting lines. Each node in a layer may be input with the output of an activation function relating to the weighted input of the node in the previous layer. The weighted input is obtained by multiplying the input of the node in the previous layer by the weight value of the link mentioned above. The weight value may also be called the parameter of the neural network and can be updated by training, as described later, with reference to Figures 5 and 6. The transmission of the aforementioned weighted input to the subsequent layer may be referred to as propagation. The activation function may include sigmoid, hyperbolic tangent (tanh), and ReLU (rectified linear unit), and the activation function may introduce nonlinearity into the neural network.
[0054] Below, we will explain the image segmentation operation using the image segmentation model 300 mentioned above, in relation to Figure 2.
[0055] First, in step S210, the processor of the electronic device extracts feature data from the input image 301 based on the feature extraction layer 310. The electronic device can extract feature data from the input image 301 using the feature extraction layer 310 through the data propagation described above.
[0056] Then, in step S220, the processor of the electronic device calculates one or more class maps 325 from the extracted feature data based on the classifier layer 320. The electronic device calculates class maps 325 from the feature data using the classifier layer 320 through the data propagation described above. The electronic device calculates as many class maps 325 as there are available classes in the classifier layer 320. For example, the electronic device may calculate a class map 325 for each available class. Available classes indicate classes that can be classified by the classifier layer 320, and the number of available classes indicates the number of classes that can be classified in the classifier layer 320. For example, if there are N available classes in the classifier layer 320, the electronic device can calculate a class map 325 for each of the N available classes. Here, N may be an integer of 2 or more. In other words, the electronic device calculates N class maps 325. Exemplaryly, the number of elements in each class map 325 may be the same as the number of pixels in the input image 301. A class map 325 corresponding to any of the classes may be a set of element values that indicate the probability that a pixel in the input image 301 belongs to that class.
[0057] Next, in step S230, the processor of the electronic device calculates one or more cluster maps 335 from the extracted feature data based on the clustering layer 330. The electronic device calculates cluster maps 335 from the feature data using the clustering layer 330. The electronic device can calculate as many cluster maps 335 as there are available clusters in the clustering layer 330. For example, the number of available clusters indicates the number of clusters that can be clustered in the clustering layer 330. For example, if there are M available clusters in the clustering layer 330, the electronic device may calculate M cluster maps 335. Here, M may be an integer greater than or equal to 2. Exemplarily, the number of elements in each cluster map 335 may be the same as the number of pixels in the input image 301. A cluster map 335 corresponding to any cluster may be a set of element values indicating the probability that a pixel in the input image 301 belongs to that cluster. The object type indicated by each cluster is not fixed, and the object type indicated by the cluster map 335 may differ depending on the situation.
[0058] According to one embodiment, as shown in Figure 3, the electronic device can individually calculate a class map 325 and a cluster map 335 using feature data common to the clustering layer 330 and the classifier layer 320.
[0059] Then, in step S240, the electronic device generates image segmentation data 309 based on one or more calculated class maps 325 and one or more calculated cluster maps 335. The electronic device can generate image segmentation data 309 by labeling the pixels of the input image 301 with classes based on the class map 325 and clusters based on the clustering maps. The image segmentation data 309 shows data divided into classes and / or clusters to which each individual pixel of the input image 301 belongs.
[0060] For example, an electronic device may determine the class to label each pixel of an input image 301 using one or more class maps 325. The electronic device extracts element values corresponding to the same pixel in the input image 301 from multiple class maps 325 and uses the extracted element values to determine the class to label the pixel. For example, the electronic device can extract probability values for each class for each pixel in the input image 301 and label the pixel with the class of the class map 325 that has the highest probability value among the extracted probability values. However, this is merely an example, and labeling using class maps 325 is not limited to those described above. For reference, the electronic device may label at least some pixels in the input image 301 as object class or foreground class and label the remaining pixels as background class.
[0061] As a different example, the processor of an electronic device may exclude clustering results based on cluster map 335 for pixels in image segmentation data 309 that have been labeled based on class map 325. The electronic device may determine that pixels classified as background class in one or more class maps 325 belong to a cluster corresponding to one or more cluster maps 335. In other words, the electronic device can cluster the remaining pixels, excluding those labeled using class map 325, using the aforementioned cluster map 335. For regions classified as background class, the electronic device can extract element values corresponding to pixels like those in the input image 301 from multiple cluster maps 335 and use the extracted element values to determine which cluster to label the pixel. For example, the electronic device can extract a probability value for each cluster for each pixel classified as background class from the input image 301 and label the pixel with the cluster of the cluster map 335 that has the highest probability value among the extracted probability values. However, this is merely an example, and labeling using cluster map 335 is not limited to those described above. For example, if two or more cluster maps 335 indicate clustering results for the same or similar elements, the electronic device may temporarily merge the clusters indicated by the two or more cluster maps 335 into a single cluster. The electronic device may label the temporarily merged single cluster with pixels classified as background class. A single cluster that is temporarily merged in one scene may be separated in other scenes.
[0062] Instead of labeling the background, which is composed of various objects, into a single class, the electronic device may distinguish it into various clusters through the clustering method described above. By incorporating information about the similarities and differences between various objects in the unlabeled parts of the background into the learning process, the electronic device can extract richer features. Furthermore, the background collision problem can be incorporated into the continuous learning of image segmentation.
[0063] According to one embodiment, the class map 325, the cluster map 335, and the image segmentation data 309 correspond to the resolution of the input image 301. For example, the resolutions of the class map 325, the cluster map 335, and the image segmentation data 309 may be the same as the resolution of the input image 301. The number of elements in the class map 325, the cluster map 335, and the image segmentation data 309 may be the same as the number of pixels in the input image 301.
[0064] Figure 4 illustrates the dynamic use of a model in response to the complexity of the surrounding environment according to one embodiment.
[0065] According to one embodiment, the processor of the electronic device selects one of several classifier layers based on the location of the electronic device. For example, the electronic device may store several classifier layers in memory. At least one of the several classifier layers may have different available classes than the other classifier layers. The number of available classes for each of the several classifier layers may be different. The electronic device can selectively use one of the several classifier layers stored in memory depending on the required number of available classes. The electronic device may estimate its location using GNSS signals received using a GNSS (Global Navigation Satellite System) module. However, location estimation is not limited to this; the electronic device may also estimate its location using its positional relationship with a base station connected via a communication module and the geographical location of the access point (AP) to which it is connected. For reference, the required number of available classes, as described later, may vary depending on the complexity of the surrounding environment according to the location of the electronic device.
[0066] The electronic device can calculate one or more class maps using a selected classifier layer from the extracted feature data. The electronic device connects the selected classifier layer to the feature extraction layer. By propagating the feature data extracted from the input image using the feature extraction layer to the selected classifier layer, the electronic device can calculate class maps for each class available to the selected classifier layer.
[0067] In step S450, the electronic device according to one embodiment estimates the complexity of the surrounding environment based on the location of the electronic device. The complexity of the surrounding environment can vary depending on the number of object types that may potentially exist in the surrounding environment, as a measure of the complexity of the environment surrounding the electronic device. In other words, the higher the complexity of the surrounding environment, the greater the number of usable classes required, and the lower the complexity of the surrounding environment, the greater the number of usable classes required. For example, if the electronic device is mounted on a vehicle and the vehicle enters a city center, the complexity of the environment surrounding the electronic device increases. If the vehicle moves away from the suburbs, the complexity of the environment surrounding the electronic device decreases. As a different example, if the electronic device is mounted on an augmented reality device and the augmented reality device enters a room, the complexity of the environment surrounding the electronic device increases. If the augmented reality device moves away from the room, the complexity of the environment surrounding the electronic device decreases. However, this is merely an example, and the increase or decrease in the complexity of the surrounding environment is not limited as described above.
[0068] In one embodiment, the processor of an electronic device calculates a class map using a first classifier layer having a first number of available classes (e.g., 100). If the complexity of the surrounding environment increases in response to a change in the location of the electronic device, the processor may select a second classifier layer 422 having a second number of available classes (e.g., 1000), which is greater than the first number of available classes. Similarly, while the processor of the electronic device calculates a class map using a second classifier layer having a second number of available classes, if the complexity of the surrounding environment decreases in response to a change in the location of the electronic device, the processor may select a first classifier layer 421 having a first number of available classes (smaller than the second number of available classes). Thus, the electronic device can selectively provide labeling using classifier layers with various numbers of available classes using the same feature extraction layer. Although the first classifier layer 421 and the second classifier layer 422 have been described exemplarily, the electronic device may store two or more classifier layers. The electronic device may select a classifier layer from among the two or more classifier layers that corresponds to the estimated complexity of the surrounding environment.
[0069] Furthermore, while we have discussed increasing and decreasing the number of available classes for the classifier layer, the number of available clusters for the clustering layer can also increase and / or decrease, without being limited to this. For example, the processor of an electronic device may select one of several clustering layers based on the location of the electronic device. The electronic device may connect the selected clustering layer to a feature extraction layer. The electronic device may compute one or more cluster maps using the selected clustering layer from the extracted feature data. Thus, the electronic device can selectively provide clustering using clustering layers with varying numbers of available clusters, using the same feature extraction layer.
[0070] While the processor of the electronic device calculates a cluster map using a first clustering layer having a first number of available clusters, if the complexity of the surrounding environment increases in response to a change in the location of the electronic device, it may select a second clustering layer having a second number of available clusters greater than the first number of available clusters. While the processor of the electronic device calculates a cluster map using a second clustering layer having a second number of available clusters, if the complexity of the surrounding environment decreases in response to a change in the location of the electronic device, it may select a first clustering layer having a first number of available clusters smaller than the second number of available clusters. The first and second clustering layers have been described exemplarily, but the electronic device may store two or more clustering layers. The electronic device may select a clustering layer from among the two or more clustering layers that corresponds to the estimated complexity of the surrounding environment.
[0071] This specification primarily describes examples where the complexity of the surrounding environment changes depending on the location of the electronic device, but it is not limited to these examples. The electronic device may estimate the complexity of the surrounding environment by analyzing the scene of the input image.
[0072] Figures 5 and 6 illustrate the training of an image segmentation network according to one embodiment.
[0073] In one embodiment, the processor of an electronic device can update one or more combinations of the feature extraction layer, classifier layer 520, and clustering layer 530 based on training data. For example, the training data is collected during the movement of the electronic device, but is not limited to this. The training data includes a pair of training inputs (e.g., training image 501) and a ground truth label map 580. The ground truth label map 580 shows a map labeled with the class to which individual pixels of the corresponding training image 501 belong. In Figure 5, the portion shown as unlabeled is the portion to which the background class is labeled. In other words, portions where the object class is not labeled may be labeled with the background class.
[0074] The electronic device's processor can update the parameters of the classifier layer 520 based on a first loss 591 between the ground truth labels of the collected training data and one or more class maps calculated from the classifier layer 520. For example, the electronic device may propagate the training input to a feature extraction layer to calculate first feature data 515a. The electronic device may propagate the first feature data 515a to the classifier layer 520 to calculate extra-special class maps. Extra-special class maps may be generated for each available class of the classifier layer 520. The electronic device calculates the cross-entropy loss L between the extra-special class map and the ground truth label map 580. CE This can be calculated as the first loss of 591.
[0075] The electronic device's processor updates the parameters of the clustering layer 530 using the training image 501 of the collected training data and a second loss 592 calculated based on the augmented image from the training image 501. The electronic device generates image sets In, In', and In'' by applying various augmentations to the training image.
[0076] For example, the electronic device generates a first enhanced image 501b and a second enhanced image 501c by enhancing the training image 501. The electronic device can generate enhanced images by emphasizing the color components of the training image. Exemplarily, the first enhanced image 501b may be an image in which the green component is emphasized, and the second enhanced image 501b may be an image in which the blue component is emphasized. However, the enhancement method is not limited to changing color, and various image enhancement methods that store morphological components such as image edges, such as changing hue and / or brightness, can be used. The electronic device can calculate feature data corresponding to each image by individually propagating the training image 501, the first enhanced image 501b, and the second enhanced image 501c to the feature extraction layer 510. For example, in Figure 5, the first feature data 515a is data extracted from the training image 501, the second feature data 515b is data extracted from the first augmented image 501b, and the third feature data 515c is data extracted from the second augmented image 501c. Furthermore, although Figure 5 shows that two additional images are augmented, the number of augmented images is not limited to this.
[0077] The electronic device can calculate a cluster map corresponding to each feature data by individually propagating the first feature data 515a, the second feature data 515b, and the third feature data 515c to the clustering layer 530. For example, the electronic device can generate a first temporary cluster map calculated from the first feature data 515a, a second temporary cluster map calculated from the second feature data 515b, and a third temporary cluster map calculated from the third feature data 515c. The electronic device can calculate the second loss 592 using the temporary cluster maps generated for the original image (e.g., training image) and the augmented image. For example, the electronic device can calculate the loss L corresponding to the difference between the first temporary cluster map, the second temporary cluster map, and the third temporary cluster map. cluster This is calculated as the second loss of 592. The difference between temporary cluster maps includes, for example, the difference between vectors showing cluster probabilities for each pixel, and the difference between data obtained by transforming said vectors (for example, data obtained by normalizing or transforming the vector values). The aforementioned vector showing cluster probabilities for each pixel is a 1×1×M-dimensional vector in which, for each of the W×H pixels in a W×H×M-dimensional cluster map, each pixel has as an element the probability that each pixel belongs to each cluster.
[0078] The electronic device updates the parameters of the layers in the image segmentation model using the overall loss based on the first loss 591 and the second loss 592 described above. For example, the electronic device can backpropagate the overall loss and update the parameters until the overall loss falls below a threshold. The electronic device may, exemplarily, update the parameters based on a slope descent method. However, it may also repeat the parameter updates until the overall loss converges. The overall loss may, but is not limited to, a weighted sum of the first loss 591 and the second loss 592.
[0079] The electronic device can train the image segmentation model to reduce the aforementioned first loss 591, thereby allowing the classifier layer 520 to learn the label information of the ground truth label map 580. The electronic device can also train the image segmentation model to reduce the aforementioned second loss 592, thereby allowing the pixel-wise feature vectors v of each image extracted from the feature extraction layer 510 to learn the label information of the ground truth label map 580. ij 、v' ij 、v´´ ij The parameters of the clustering layer 530 can be updated to form either identical or similar clusters. In other words, through training with the second loss 592, the electronic device can group pixels that represent identical or similar objects and / or backgrounds in the input image.
[0080] Therefore, the electronic device can label clusters on pixels that are labeled with classes by training the parameters of the classifier layer 520 and the clustering layer 530 in a time series or in parallel. Thus, the electronic device can efficiently train even if only some classes are labeled (annotated) in the training image. For reference, the second loss 592 shown in Figure 5 is an illustrative clustering loss, and the loss used to train the clustering layer 530 is not limited to this.
[0081] Figure 6 illustrates an example of training when the number of target classes increases while upgrading the classifier layer.
[0082] Exemplary, with reference to Figure 5, the electronic device updates the parameters of the feature extraction layer 610, the classifier layer 620a, and the clustering layer 630 through training similar to that described above. The classifier layer 620a has N available classes, and exemplary examples include N=4 classes: Road, Sidewalk, Car, and Background. After the training described above is complete for the currently available classes, additional available classes may be added.
[0083] According to one embodiment, the processor of the electronic device can train a new classifier layer 620b, which includes a previously constructed classifier layer 620a and a layer corresponding to an additional class, based on the collected training data. The electronic device can use a previously trained image segmentation model to generate new ground truth class data 680 by combining a cluster map 672 corresponding to a new label (e.g., traffic light) from the calculated clustering map with the output of the previously constructed classifier layer, and collect it as training data.
[0084] For example, the processor of an electronic device may obtain clustering results corresponding to classes added from one or more cluster maps calculated by a previously constructed clustering layer 630.
[0085] For example, in Figure 6, the electronic device can generate cluster maps U for the number of available clusters using the clustering layer 630. The input image 601 has a resolution of H × W, where H is the number of pixels corresponding to the height and W is the number of pixels corresponding to the width. The electronic device generates M cluster maps of H × W resolution, which is the number of available clusters. The set of cluster maps is denoted as cluster data U, and U[H × W × M] represents cluster data 635 containing M cluster maps of H × W resolution. Similarly, the electronic device can generate class maps for the number of available classes using the classifier layer. The electronic device generates N class maps of H × W resolution, which is the number of available classes. Here, N and M may each be integers greater than or equal to 1. The set of class maps is denoted as class data O, and O[H × W × N] represents class data 625a containing N class maps of H × W resolution, which is the number of available classes.
[0086] The electronic device selects a cluster map 672 from one or more cluster maps that corresponds to the class to be added. For example, the electronic device can extract the probability (e.g., cluster probability) of each cluster belonging to a pixel 671 corresponding to the class to be added from among pixels labeled with the background class. The electronic device may determine the cluster in cluster map 672 that shows the highest cluster probability among the cluster probabilities of pixels 671 corresponding to the class to be added as the cluster corresponding to the class to be added. However, the electronic device may also determine the cluster corresponding to the class to be added as a cluster that has a probability value belonging to a predetermined rank in descending order for the pixel 671 corresponding to the class to be added. For example, the electronic device may select a cluster map 672 from M cluster maps that has a high probability for pixels 671 corresponding to the traffic light class to be added as the cluster map corresponding to the class to be added. Exemplaryly, in an image, pixels 671 corresponding to the class to be added may be specified by the user and / or experts. The cost of constructing the dataset can be reduced because only the class to be added needs to be annotated.
[0087] The electronic device can update the parameters of the new classifier layer 620b based on the third loss between the calculated class maps, using the acquired clustering results and the new classifier layer 620b. For example, the electronic device can calculate new class data 625b by propagating the feature data extracted via the feature extraction layer 610 to the new classifier layer 620b.
[0088] The new class data 625b includes N+1 class maps corresponding to the existing N available classes and the newly added available classes. In Figure 6, only N+1 H×W class maps may be generated. However, this is an example where one available class is added; if k available classes are added, N+k new class maps may be generated, where k is an integer greater than or equal to 1. The electronic device can generate new ground truth class data 680 O'[H×W×(N+1)] by combining the class data 625a O[H×W×N], which is the output of the previously described classifier layer 620a, with the cluster map 672 corresponding to the added classes.
[0089] The electronic device calculates the loss (e.g., distillation loss) L between the output of the new classifier layer 620b and the new ground truth class data 680 O'[H×W×(N+1)]. distillThis is calculated as the third loss 693. Here, the electronic device calculates and sums the third loss 693 only for the portion corresponding to the newly added class (e.g., pixel 671 corresponding to a traffic light) in the new ground truth class data 680 O'[H×W×(N+1)] and the class data 625b which is the output of the new classifier layer 620b, excluding the remaining portion from the calculation of the third loss 693. The electronic device iteratively updates the parameters of the new classifier layer 620b so that the third loss 693 decreases. Once the update based on the third loss 693 is complete, the initial parameters of the new classifier layer 620b may be set. The electronic device can also calculate the overall loss by summing the first loss (e.g., the first loss 591 shown in Figure 5), the second loss (e.g., the second loss 592 shown in Figure 5), and the third loss 693, and iteratively train at least one layer of the image segmentation model until the overall loss decreases or converges to a threshold. Therefore, the electronic device can reduce learning time and resource requirements via the third loss 693, and prevent learning degradation due to conflicts between newly added classes and already learned backgrounds. Since the electronic device uses only labels for the added classes, it retains learning knowledge from previous classes and enables continuous learning.
[0090] Here, we have described an example of utilizing the output of a previously constructed classifier layer 620a for previously constructed usable classes among the usable classes of the new classifier layer 620b, but we are not limited to this. The electronic device can utilize the output of a previously constructed clustering layer 630 only for newly added usable classes, and calculate the third loss 693 using a pre-prepared ground truth label map for existing usable classes.
[0091] Furthermore, the electronic device may update the feature extraction layer 610 and clustering layer 630 by retaining their parameters and replacing them with a new classifier layer 620b. However, it is not limited to this, and the electronic device may update all the parameters of the feature extraction layer 610, clustering layer 630, and the new classifier layer 620b. Here, the number of usable clusters in the clustering layer 630 may decrease by the number of added classes. For example, if the number of usable classes in the new classifier layer 620b is N+k, the electronic device may modify the image segmentation model to have a new clustering layer with Mk usable clusters.
[0092] Figure 7 is a block diagram showing the configuration of an electronic device according to one embodiment.
[0093] An electronic device 700 according to one embodiment includes an image acquisition unit 710, a processor 720, a memory 730, and a display 740.
[0094] The image acquisition unit 710 acquires images of the surrounding scene of the electronic device 700. According to one embodiment, the image acquisition unit 710 acquires the surrounding scene (e.g., in front) of the device on which the electronic device 700 is mounted (e.g., a vehicle and / or an augmented reality device). For example, the image acquisition unit 710 may include an imaging module that generates an image by photographing the surrounding scene. The image acquisition unit 710 may include a camera sensor, a LiDAR sensor, a radar sensor, and an ultrasonic sensor. The electronic device 700 generates a color image via detection data from the camera sensor. The electronic device 700 generates a LiDAR image via detection data from the LiDAR sensor. The electronic device 700 generates a radar scan image via detection data from the radar sensor. The electronic device 700 generates an ultrasonic image via detection data from the ultrasonic sensor. However, the image acquisition unit 710 may include a communication unit, but is not limited thereto. For example, the communication unit of the electronic device 700 may receive images via wired and / or wireless means from different imaging devices located outside the electronic device 700.
[0095] The processor 720 can extract feature data from an input image based on a feature extraction layer. The processor 720 can calculate one or more class maps from the extracted feature data based on a classifier layer. The processor 720 can calculate one or more cluster maps from the extracted feature data based on a clustering layer. The processor 720 can generate image segmentation data based on the calculated one or more class maps and one or more calculated cluster maps. However, the operation of the processor 720 is not limited to this, and it may perform the operations described above with reference to Figures 1 to 6.
[0096] Memory 730 temporarily or permanently stores the data required for image segmentation. For example, memory 730 may store input images, training data, an image segmentation model, a feature extraction layer, a classifier layer, a clustering layer, parameters for each layer, and inference results (e.g., image segmentation data).
[0097] The display 740 visually outputs the aforementioned image segmentation data. For example, in an electronic device 700 mounted on a vehicle, the display 740 can visually display existing driving information generated using the image segmentation data. The vehicle may be an autonomous vehicle and / or a vehicle equipped with ADAS (Advanced Driver Assistance Systems).
[0098] As a different example, in an electronic device 700 installed in a device that provides augmented reality, the display 740 can visually display augmented reality content generated using image segmentation data to the user.
[0099] In one embodiment, the electronic device 700 can retain knowledge of previously learned classes when adding new classes by performing training and inference using both a classifier layer and a clustering layer. Furthermore, the electronic device 700 can train its feature extraction layer to output meaningful and rich feature vectors not only for labeled pixels but also for unlabeled pixels (e.g., the background).
[0100] The electronic device 700 according to one embodiment can be applied not only to vehicles and augmented reality devices, but also to content-based image retrieval, machine vision, medical imaging, and video surveillance equipment that utilize image segmentation.
[0101] The electronic device 700 can perform continuous learning, which allows it to adapt sequentially to variously changing training data without needing to retrain the network on all training data. Because the electronic device 700 only annotates and trains newly added classes, it can save costs and time. Furthermore, by performing clustering on the background, the electronic device 700 can provide pseudo-labeling results even for labeled portions. Through learning using distillation, knowledge collisions between already learned background portions and portions corresponding to newly added classes can be prevented during the learning process.
[0102] The embodiments described above are embodied in hardware components, software components, or combinations of hardware and software components. For example, the devices and components described in these embodiments are embodied using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or different devices that execute and respond to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the OS. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For convenience of understanding, the processing device may sometimes be described as being used as a single unit, but a person with ordinary skill in the art will understand that the processing device includes multiple processing elements and / or multiple types of processing elements. For example, the processing device includes multiple processors or one processor and one controller. Furthermore, other processing configurations, such as parallel processors, are also possible.
[0103] Software includes computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or can instruct the processing unit independently or in combination. Software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, in order to be interpreted by a processing unit or to provide instructions or data to a processing unit. Software can be distributed across a network of computer systems and stored or executed in a distributed manner. Software and data can be stored on a recording medium readable by one or more computers.
[0104] The method according to this embodiment is embodied in the form of program instructions that are implemented via various computer means and recorded on a computer-readable recording medium. The recording medium includes program instructions, data files, data structures, etc., individually or in combination. The recording medium and program instructions may be specifically designed and configured for the purposes of the present invention, or they may be known and usable by those skilled in the art who have technology in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code executed by a computer using an interpreter or the like.
[0105] The hardware device described above may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.
[0106] As described above, embodiments have been illustrated with limited drawings as examples, but a person with ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or assembled in a different manner than described, and may be replaced or substituted by other components or equivalents, while still achieving satisfactory results.
[0107] Therefore, other implementations, other embodiments, and equivalents to the claims described below also fall within the scope of the claims.
Claims
1. An electronic device, An image acquisition unit that acquires an input image, A processor that extracts feature data from the input image based on a feature extraction layer, calculates one or more class maps from the extracted feature data based on a classifier layer, calculates one or more cluster maps from the extracted feature data based on a clustering layer, and generates image segmentation data based on the one or more calculated class maps and the one or more calculated cluster maps. Includes, The processor is an electronic device that excludes clustering results based on one or more cluster maps for pixels of the image segmentation data that have been labeled based on one or more class maps.
2. The electronic device according to claim 1, wherein the processor generates one or more class maps, one or more cluster maps, and image segmentation data based on the resolution of the input image.
3. The electronic device according to claim 1, wherein the processor determines that, in the one or more class maps, pixels classified as background belong to a cluster corresponding to any one of the one or more cluster maps.
4. The aforementioned processor, Based on the location of the electronic device, select one of several classifier layers. The electronic device according to claim 1, wherein one or more class maps are calculated from the extracted feature data using the selected classifier layer.
5. The electronic device according to claim 1, wherein, while the processor calculates a class map using a first classifier layer having a first number of available classes, if the complexity of the surrounding environment increases in accordance with a change in the position of the electronic device, it selects a second classifier layer having a second number of available classes that is greater than the first number of available classes.
6. The electronic device according to claim 1, wherein, while the processor calculates a class map using a second classifier layer having a second number of available classes, if the complexity of the surrounding environment decreases in accordance with a change in the position of the electronic device, it selects a first classifier layer having a first number of available classes that is smaller than the second number of available classes.
7. The aforementioned processor, Based on the location of the electronic device, select one of the clustering layers from among multiple clustering layers. The electronic device according to claim 1, wherein one or more cluster maps are calculated from the extracted feature data using the selected clustering layer.
8. The electronic device according to claim 1, wherein, while the processor calculates one or more cluster maps using a first clustering layer having a first number of available clusters, if the complexity of the surrounding environment increases in accordance with a change in the location of the electronic device, the processor selects a second clustering layer having a second number of available clusters that is greater than the first number of available clusters.
9. The electronic device according to claim 1, wherein, while the processor calculates one or more cluster maps using a second clustering layer having a second number of available clusters, if the complexity of the surrounding environment decreases in response to a change in the location of the electronic device, the processor selects a first clustering layer having a first number of available clusters that is smaller than the second number of available clusters.
10. The electronic device according to claim 1, wherein the processor updates one or more combinations of the feature extraction layer, the classifier layer, and the clustering layer based on training data collected while the electronic device is in motion.
11. The electronic device according to claim 10, wherein the processor updates the parameters of the classifier layer based on the ground truth labels of the collected training data and the loss between the one or more class maps calculated from the classifier layer.
12. The electronic device according to claim 10, wherein the processor updates the parameters of the clustering layer using the training image of the collected training data and the loss calculated based on the enhanced image from the training image.
13. The electronic device according to claim 10, wherein the processor causes a new classifier layer, which includes a previously constructed classifier layer and a layer corresponding to an additional class, to be trained based on the collected training data.
14. The aforementioned processor, Obtain the clustering result corresponding to the additional class from one or more cluster maps calculated by the previously constructed clustering layer, The electronic device according to claim 13, which updates the parameters of the new classifier layer based on the loss between the acquired clustering results and the class map calculated using the new classifier layer.
15. It is a vehicle, The electronic device described in claim 1 includes, The image acquisition unit acquires the input image relating to the surrounding scene of the vehicle while the vehicle is in motion. A vehicle in which the processor generates information about the vehicle's movement using the image segmentation data and visually displays the information about the vehicle's movement via a display.
16. The vehicle according to claim 15, wherein the processor uses the image segmentation data to perform one or more combinations of speed control, acceleration control, and steering control of the vehicle when the vehicle is in motion.
17. It is an augmented reality device, The electronic device described in claim 1 includes, An augmented reality device comprising: a processor that generates augmented reality content using the image segmentation data, and an augmented reality display that visually displays the generated augmented reality content to a user via an augmented reality display.
18. A method implemented by the processor, The steps include: extracting feature data from the input image based on a feature extraction layer, The steps include calculating one or more class maps based on a classifier layer from the extracted feature data, The steps include calculating one or more cluster maps based on a clustering layer from the extracted feature data, The steps include generating image segmentation data based on the one or more class maps and one or more cluster maps calculated, Includes, This method is For pixels in the image segmentation data that have been labeled based on one or more class maps, the step of excluding the clustering results based on one or more cluster maps is performed. A method that further includes this.
19. A computer-readable recording medium storing one or more computer programs including instruction words for performing the method described in claim 18.
20. An electronic device, Includes one or more processors, The one or more processors described above are: Get the input data, Based on the feature extraction layer, feature data is extracted from the input data. Based on the physical location of the electronic device, the clustering layer and classifier layer are selected. Based on the classifier layer, one or more class maps are generated from the feature data. Based on the clustering layer, one or more cluster maps are generated from the feature data. Based on the one or more cluster maps and the one or more class maps, image segmentation data is generated. The one or more processors exclude clustering results based on the one or more cluster maps for pixels in the image segmentation data that have been labeled based on the one or more class maps. electronic equipment.
21. The electronic device according to claim 20, wherein the input data is image data relating to the periphery of the electronic device.
22. The electronic device according to claim 20, wherein the one or more processors generate the one or more cluster maps and the one or more class maps using a machine learning model.
23. The electronic device according to claim 22, wherein the machine learning model includes a neural network.
Citation Information
Patent Citations
Image processing device, image processing system, and image processing method
JP2018097738A
Method for image region classification using unsupervised and supervised learning
US20030147558A1
Intersection region detection and classification for autonomous machine applications
US20200410254A1
