Graph-based color description generation

Abstract

A method, system, and computer program product for generating natural language descriptions of colors are presented, including obtaining a list of tuples, generating a graph by using each of the tuples as a node and adding an edge between the nodes if the color difference between the nodes in terms of human perception is outside a predetermined range, filtering the edges based on an external color comparison description, incorporating a new node into the graph by finding a nearest neighboring node based on the color difference in terms of human perception and adding a new edge between the new node and the nearest neighboring node, learning a feature vector for each of the nodes using message passing and using the node's color as an initial seed, and generating a description of the new node by using each of the learned feature vectors as an initial state of a neural network-based decoder.

Description

Technical Field

[0001] The present invention generally relates to color naming, and more specifically, to the generation of graph-based color descriptions.

Background Art

[0002] Color is one of the major visual cues and has been extensively studied at various levels, starting from the physics and psychophysics of color to the use of color principles in practical problems. It includes numerous applications in accurate rendering, display and reproduction, image filtering, encoding, retrieval, other scientific visualizations, computer graphics, and image and video processing. Interestingly, despite being one of the most common visual tasks, color naming has received relatively little attention in the engineering community. However, with the rapid development of visual technologies and multimedia, and the development of advanced user interfaces and human-machine interactions, the ability to name individual color names, point to specific color objects, and convey the impression of color compositions has become an increasingly important task. Color cues can be utilized in interactive visualizations and computer graphics. Color naming facilitates the design of natural user interfaces. Since higher-order color descriptors often provide links to image content, the extraction of such higher-order color descriptors is a challenging task in image analysis and computer vision. It is advantageous to be able to utilize color naming for object selection by color, description of the appearance of an image, and even generation of semantic annotations when combined with image segmentation.

Summary of the Invention

[0003] According to a first aspect, the present invention provides a method for generating a description of a color in natural language. The method includes obtaining a list of tuples, each tuple including a color and a description; generating a graph by using each tuple as a node and adding an edge between nodes when the color difference between the nodes is outside a predetermined range from the perspective of human perception; filtering the edges based on external color comparison descriptions stored in an external color comparison database; finding the nearest neighbor node based on the color difference from the perspective of human perception, and incorporating a new node into the graph by adding a new edge between the new node and the nearest neighbor node, the new node including a new color not included in the list of tuples and having no description; learning a feature vector for each node in the graph by using message passing and using the color of the node as an initial seed; and generating a description of the new node by using each of the learned feature vectors as an initial state of a recurrent neural network (RNN)-based decoder in a semi-supervised manner.

[0004] According to another aspect, the present invention provides a method for generating a description of a color in natural language. The method includes obtaining a list of tuples, each tuple including a color and a description; generating a graph by using each tuple as a node and adding an edge between nodes based on a similarity threshold; filtering the edges based on external color comparison descriptions stored in an external color comparison database; finding the nearest neighbor node based on the similarity threshold, and incorporating a new node into the graph by adding a new edge between the new node and the nearest neighbor node.

[0005] A computer program product for generating a natural language description of a color is presented. This computer program product includes a computer-readable storage medium embodying program instructions therein. The program instructions are executable by a computer and cause the computer to obtain a list of tuples, each tuple including a color and a description; generate a graph by using each tuple as a node and adding an edge between nodes when the color difference between the nodes is outside a predetermined range from the perspective of human perception; filter the edges based on external color comparison descriptions stored in an external color comparison database; find the nearest neighbor node based on the color difference from the perspective of human perception and incorporate a new node into the graph by adding a new edge between the new node and the nearest neighbor node, where the new node includes a new color not included in the list of tuples and has no description; learn a feature vector for each node in the graph by using message passing and using the color of the node as an initial seed; and generate a description of the new node by using each of the learned feature vectors as an initial state of a recurrent neural network (RNN)-based decoder in a semi-supervised manner.

[0006] A computer program product for generating a natural language description of a color is presented. This computer program product includes a computer-readable storage medium embodying program instructions therein. The program instructions are executable by a computer and cause the computer to obtain a list of tuples, each tuple including a color and a description; generate a graph by using each tuple as a node and adding an edge between nodes based on a similarity threshold; filter the edges based on external color comparison descriptions stored in an external color comparison database; find the nearest neighbor node based on the similarity threshold and incorporate a new node into the graph by adding a new edge between the new node and the nearest neighbor node.

[0007] According to yet another embodiment, a method for generating a description of a color in natural language is provided. The method includes generating a graph that structures the color relationships, including a plurality of nodes and edges; removing redundant edges based on color comparison descriptions; incorporating new nodes into the graph along with new edges based on a similarity threshold; learning a feature vector for each node of the graph; and generating a description of the new nodes using the feature vectors, wherein the feature vectors are provided to a neural network in a semi-supervised manner.

[0008] It should be noted that the exemplary embodiments are described with reference to different subjects. In particular, some embodiments are described with reference to method-type claims, while other embodiments are described with reference to apparatus-type claims. However, those skilled in the art will appreciate from the above and following descriptions that, unless otherwise specified, in addition to any combination of features belonging to one type of subject, any combination between features related to different subjects, particularly any combination between the features of method-type claims and the features of apparatus-type claims, is also considered to be described in this document.

[0009] These and other features and advantages will become apparent from the following detailed description of its exemplary embodiments, read in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0010] The present invention will be described in detail in the following description of the preferred embodiments with reference to the following drawings.

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

[0012] Throughout the drawings, the same or similar reference numerals represent the same or similar elements.

DETAILED DESCRIPTION OF THE INVENTION

[0013] Exemplary embodiments according to the present invention provide for generating a description of a color in natural language. A color grounding task enables characterizing the relationship between language and color perception. Learning the mapping between language and color is important for understanding some aspects of human expression creativity. Applications of color grounding are related to advertising and / or marketing or both, for example, to predict public opinion (unstructured, abstract) and utilize that public opinion for support of branding or speculation of user reactions, etc. The task in color grounding is to learn to generate a description of a color in natural language. Exemplary embodiments of the present invention configure a training set as a graph and perform description generation in a semi-supervised manner.

[0014] Although color spaces allow for unambiguous color specification, in everyday life colors are primarily identified by their names. Identifying colors by names is a method of communication that requires a fairly general color vocabulary and lacks precision, but is understandable to everyone. Thus, several attempts have been made to design a vocabulary, syntax, and standard method for selecting color names. The Munsell color system, known to those skilled in the art, is widely used in applications that require precise specification of colors. Examples include the manufacture of paints, textiles, etc. The Munsell color system is often complemented by the Munsell Color Book, which contains 1,200 precisely controlled color samples (chips), and is used as an industry standard. The chips are arranged so that the unit steps between the chips are perceptually equal. Each chip is identified by three codes. The lightness scale is represented by the Munsell value, with black being 0 / and white being 10 / . The Munsell chroma is two steps higher ( / 2, / 4, ..., / 10). The hue scale is divided into 10 hues: red (R), yellow-red (YR), yellow (Y), yellow-green (GY), green (G), blue-green (BG), blue (B), purple-blue (PB), purple (P), and red-purple (RP), and each hue can be further divided into 10 sections. However, one notable drawback of color-based processing with the Munsell system is the inability to convert accurately from any color space to the Munsell system. For example, conversions proposed by others are quite complex and may be inaccurate in certain areas of CIE XYZ.

[0015] The first list of over 3000 English words and phrases used to name colors was devised by Maerz and Paul and published as the Dictionary of colors. A more detailed dictionary was published by the National Bureau of Standards in the United States, which lists about 7500 different names that have come into common use in specific fields such as biology, geology, stamp collecting, the textile, dye, and paint industries. Both dictionaries contain examples of rare or difficult words and are not suitable for general use because the terms are listed in a non-systematic way.

[0016] The National Bureau of Standards of the United States developed the ISCC-NBS Color Name Dictionary, which lists the color names of 267 regions in the color space. This dictionary uses English terms to describe colors along the three dimensions of hue, lightness, and saturation in the color space. One problem with the ISCC-NBS model is that it has no systematic syntax. To address this, a new Color-Naming System (CNS) was designed. CNS is partially based on the ISCC-NBS model. CNS uses the same three dimensions, but the rules used to combine the words of these dimensions are defined by a formal syntax. A model called Color-Naming Method (CNM), which extends the CNS model, uses a syntax similar to the syntax described in the CNS model and maps the color names of CNM to the color range of the Munsell system. All of the above methods are closely related to the Munsell model and thus provide an explanation of how to position each name within the Munsell color space.

[0017] However, it is not clear how to use these methods to label color samples with color names, show examples of named colors, describe the color regions and objects in a scene, and ultimately convey the color composition of an image.

[0018] Exemplary embodiments of the present invention disclose a method and system for alleviating such problems by generating a graph-based color description mechanism or structure. The training set is structured as a graph and performs description generation in a semi-supervised manner. Exemplary embodiments of the present invention use context information that incorporates the global perception of color representation. The use of human perception enables more realistic color relationships, and the graph-based representation enables neighboring nodes to share information / context. Context information provides useful diversity that potentially leads to the discovery of more associations between colors and languages when exact matches are not met. Furthermore, in view of the exemplary embodiments of the present invention, there are fewer memory-related problems compared to the standard model.

[0019] The present invention is described from the perspective of a given exemplary architecture, but it will be understood that other architectures, structures, substrate materials, and process features and steps / blocks can be varied within the scope of the present invention. Note that for clarity, not all specific features can be shown in all figures. This is not intended to be construed as a limitation of any particular embodiment, illustration, or claim.

[0020] Various exemplary embodiments of the present invention will be described below. For clarity, not all features of an actual implementation are described herein. Of course, in the development of such an actual implementation, many implementation-specific decisions must be made to achieve the specific goals of the developer, such as compliance with system-related and business-related constraints, which will vary from implementation to implementation. Further, although such development efforts may be complex and time-consuming, it will be understood that they would be routine work for those skilled in the art having the benefit of the present invention.

[0021] FIG. 1 is an exemplary diagram illustrating context color graph generation according to an embodiment of the present invention.

[0022] Regarding context color graph generation 10, data 12 is structured as a graph based on a similarity threshold t. The threshold t is defined based on Delta E, a metric that explains how humans perceive color differences (useful metric as distances in RGB space are not linearly perceived). Specifically, for example, when the Delta E value between two colors is between 11 and 49, an exemplary method generates a link. When the Delta E value is outside that range (too similar, almost the same, or too far), the exemplary method does not add an edge.

[0023] The context color graph can, in one non-limiting example, include a first color 20, a second color 22, a third color 24, a fourth color 26, a fifth color 28, a sixth color 30, and a seventh color 32. The first color 20 is connected to the second color 22 via edge 11. The second color 22 is connected to the third color 24 via edge 13. The third color 24 is connected to the fourth color 26 via edge 17. The first color 20 is connected to the fourth color 26 via edge 15. The third color 24 is connected to the fifth color 28 via edge 19. The fourth color 26 is connected to the fifth color 28 via edge 21. The fourth color 26 is connected to the sixth color 30 via edge 23. The fifth color 28 is connected to the sixth color 30 via edge 25. The fifth color 28 is connected to the seventh color 32 via edge 27.

[0024] Data 12 can be specified as a list T of (color, description) tuples (c i , d i ), where c i is a color represented as an RGB tuple and d i is a sequence of tokens.

[0025] Graph G can be given as follows. [Number]

[0026] Delta E (or dE) is a single number representing the "distance" between two colors. The idea is that a dE of 1.0 is the smallest color difference perceptible to the human eye. That is, any dE smaller than 1.0 cannot be perceived, and of course, any dE larger than 1.0 can be noticed. However, depending on the color difference, there are cases where it can be completely acceptable even if it is larger than 1, or even unnoticed. Also, even for the same dE color difference, the color difference between two types of yellow and the color difference between two types of blue may not look the same to the eye, and in another place, it may look different to the observer's eye.

[0027] The Delta E level is the difference between the displayed color and the original color standard of the input content. A lower Delta E value indicates higher accuracy, while a higher Delta E value indicates a larger mismatch. The "E" in Delta E represents "Empfindung" in German, which means "sensation". Overall, the term Delta E means a difference in sensation. Exemplary embodiments create or generate a graph-based method to assist in the definition of new colors.

[0028] Figure 2 is an exemplary diagram illustrating edge filtering according to an embodiment of the present invention.

[0029] Regarding edge filtering, external data from the external data color comparison database 35 is used to remove redundant or infeasible edges. Color comparison data 40 is used. The color comparison data can include several different colors 41-48 in one non-limiting example. For each edge, if the (source, target) pair is close to any pair within the dataset, that edge is retained or maintained. Otherwise, the edge is deleted. This allows for a smoother transition (e.g., avoiding sudden color changes), which ultimately facilitates learning. As a result of edge removal, a context color graph 10' is obtained. In this graph 10', three edges have been removed. The first removed edge is shown as 50 and was located between the first color 20 and the fourth color 26. The second removed edge is shown as 52 and was located between the third color 24 and the fifth color 28. The third removed edge is shown as 54 and was located between the fourth color 26 and the fifth color 28. Thus, based on the external data, these three edges were determined to be redundant or infeasible edges. Note that the external data is stored in the external data color comparison database 35, which is a custom database for storing only comparison color data.

[0030] Figure 3 is an exemplary diagram illustrating the addition of label-free nodes according to an embodiment of the present invention.

[0031] Regarding the addition of unlabeled nodes, when an unseen color is given, it is represented as an unlabeled node (having no description, only color). The unseen color 62 is incorporated into the main graph 60 by finding the nearest neighbor in terms of color difference. This process also uses the delta E, which is a metric of human perception, as a threshold (the same configuration as described above for the generation of the context color graph in FIG. 1). After the node 62 is added to the graph 60, the goal is to use the context information provided by the edges from the set of labeled nodes to the unlabeled node, thereby learning the description d for c u for u The new node 62 has edges 61 and 63. Edge 61 connects or links the new node 62 to the fifth color 28, and edge 63 connects or links the new node 62 to the seventh node 32. Therefore, the color of node 62 is similar to or close to colors 28 and 32.

[0032] FIG. 4 is an exemplary diagram 70 illustrating color-based node feature learning according to an embodiment of the present invention.

[0033] Regarding color-based node feature learning, colors are used as the initial seeds, and a message-passing algorithm is used to learn the feature vectors for all nodes. As a result, each obtained vector encodes the relationship between each color and its surrounding neighbors.

[0034] Specifically, five colors are shown. Therefore, five vectors are generated. For example, the fourth color 26 has a feature vector 72, the fifth color 28 has a feature vector 76, the sixth color 30 has a feature vector 74, the seventh color 32 has a feature vector 78, and the new node 62 has a feature vector 80. The arrows between colors 28 and 32, colors 30 and 32, and colors 26 and 30 indicate the relationships developed between different colors.

[0035] Note that the Message Passing Algorithm (MPA) is a type of probability propagation algorithm that operates in the graphical model of codes. Since messages repeatedly travel back and forth between variable nodes and check nodes until a result is reached or the process stops, MPA is also known as an iterative algorithm.

[0036] FIG. 5 is an exemplary diagram 90 illustrating description generation according to an embodiment of the present invention.

[0037] Regarding description generation, the learned vectors are used in a semi-supervised setting. Each vector is passed to recurrent neural network (RNN)-based decoders 92, 94, 96 and functions as its initial seed for generation. The loss in the RNN is calculated with respect to the labeled part of the graph. Thus, learning affects all related representations (both labeled and unlabeled). Further, the feature vector 80 of the new node 62 passes through the decoders 92, 94, 96 to generate weights w1, w2, w3.

[0038] FIG. 6 is a block / flow diagram of an exemplary method for context color graph generation according to an embodiment of the present invention.

[0039] In block 110, the data is structured as a graph based on a similarity threshold, which is defined based on delta E (a metric that explains how humans perceive color differences).

[0040] In block 112, it is determined whether the delta E values for two colors are between 11 and 49. If YES, the process proceeds to block 114. If NO, the process proceeds to block 116.

[0041] In block 114, a link is generated between the two colors.

[0042] In block 118, after the context color graph is generated, edge filtering is started.

[0043] In block 116, if the Delta E value is not between 11 and 49, no connection is defined between the two colors.

[0044] FIG. 7 is a block / flow diagram of an exemplary method for edge filtering according to an embodiment of the present invention.

[0045] In block 120, as described with reference to FIG. 6, a context color graph is generated.

[0046] In block 122, an edge filtering process is started using external data including at least color comparison data.

[0047] In block 124, it is determined whether a (source, target) pair is close to any pair in the color comparison data. If YES, the process proceeds to block 126. If NO, the process proceeds to block 128.

[0048] In block 126, the edge is maintained or held.

[0049] In block 129, after edge filtering is performed, unlabeled node addition is performed.

[0050] In block 128, if the pair is not close to any pair in the color comparison data, the edge is removed.

[0051] FIG. 8 is a block / flow diagram of an exemplary method for unlabeled node addition according to an embodiment of the present invention.

[0052] In block 120, as described with reference to FIG. 6, a context color graph is generated.

[0053] In block 118, as described with reference to FIG. 7, edge filtering is performed.

[0054] In block 130, the unseen color is represented as a label - less node (no description).

[0055] In block 132, the unseen color is incorporated into the main graph (as a node) by finding the nearest neighbor from the perspective of color difference.

[0056] In block 134, after adding label - less nodes, color - based node feature learning is performed.

[0057] FIG. 9 is a block / flow diagram of an exemplary method for color - based node feature learning according to an embodiment of the present invention.

[0058] In block 120, as described with reference to FIG. 6, a context color graph is generated.

[0059] In block 118, as described with reference to FIG. 7, edge filtering is performed.

[0060] In block 129, as described with reference to FIG. 8, adding label - less nodes to the context color graph is performed.

[0061] In block 140, using a message - passing algorithm, a feature vector is learned for all nodes to encode the relationship between each color and its surrounding neighbors.

[0062] In block 142, after performing color - based node feature learning, description generation is performed.

[0063] FIG. 10 is a block / flow diagram of an exemplary method for description generation according to an embodiment of the present invention.

[0064] In block 120, as described with reference to FIG. 6, a context color graph is generated.

[0065] In block 118, edge filtering is performed as described with reference to FIG. 7.

[0066] In block 129, adding unlabeled nodes to the context color graph is performed as described with reference to FIG. 8.

[0067] In block 140, feature vectors are learned for all nodes of the context color graph as described with reference to FIG. 9.

[0068] In block 150, the learned vectors are used in a semi-supervised setting for description generation (each vector is passed to an RNN-based decoder and functions as an initial seed for generation).

[0069] In block 152, the color-language mapping is incorporated in tabular form and displayed on a computing device.

[0070] FIG. 11 is a block / flow diagram of an exemplary method of structuring a training set as a graph and performing description generation in a semi-supervised manner, according to an embodiment of the present invention.

[0071] In block 210, a list of tuples is obtained, where each tuple includes a color and a description.

[0072] In block 220, a graph is generated by using each tuple as a node and adding an edge between nodes when the color difference between the nodes is outside a predetermined range in terms of human perception.

[0073] In block 230, the edges are filtered based on external color comparison descriptions stored in an external database.

[0074] In block 240, find the nearest neighbor node based on the color difference from the perspective of human perception, and incorporate the new node into the graph by adding a new edge between the new node and the neighbor node. The new node contains a new color not included in the list and has no description.

[0075] In block 250, use a message-passing algorithm and use the color of the node as the initial seed to learn the feature vector for each of the nodes in the graph.

[0076] In block 260, generate a description of the new node by using each of the learned feature vectors as the initial state of an RNN-based decoder in a semi-supervised manner.

[0077] In summary, the exemplary embodiments of the present invention use context information incorporating the global perception of color representation. The use of human perception enables more realistic color relationships, and the graph-based representation enables neighboring nodes to share information / context. The context information provides useful diversity that potentially leads to the discovery of more associations between colors and languages when exact matches are not met. Further, compared to the standard model, there are fewer problems related to memory. Also, the graph generated for color-language mapping is only one for each dataset. Thus, the model is learned, the color relationships are structured using the graph, and a syntactically valid description is generated that can be used for several generation and retrieval tasks.

[0078] FIG. 12 is a block / flow diagram of an exemplary cloud computing environment according to an embodiment of the present invention.

[0079] The present invention includes a detailed description of cloud computing, but it will be understood that the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in combination with any other type of computing environment, whether now known or later developed.

[0080] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0081] The features are as follows. On-demand self-service: Cloud consumers can automatically and unilaterally provision computing capabilities such as server time and network storage as needed, without the need for a human to interact with the service provider. Broad network access: The capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs). Resource pooling: The provider's computing resources are pooled using a multi-tenant model to serve multiple consumers, and different physical and virtual resources are dynamically assigned and reallocated in response to requests. Consumers are generally location-independent in that they have no control or knowledge of the exact location of the resources provided, although they may be able to specify a higher level of abstraction (e.g., country, state, or data center). Rapid elasticity: The capabilities can be provisioned quickly and elastically, and in some cases automatically, to scale out rapidly and release quickly to scale in. For consumers, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time. Measured service: The cloud system automatically controls and optimizes resource use by using some form of metering capabilities at an appropriate level of abstraction for the type of service (e.g., storage, processing, bandwidth, and active user accounts). It can monitor, control, and report resource use, providing transparency to both the provider and the consumer of the services used.

[0082] The service model is as follows. Software as a Service (SaaS): The function provided to the consumer is to use the provider's application that operates on the cloud infrastructure. These applications can be accessed from various client devices through a thin-client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, server, operating system, storage, or individual application functions, with the exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The function provided to the consumer is to deploy the consumer-created or acquired application, which is generated using the programming languages and tools supported by the provider, onto the cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure including the network, server, operating system, or storage, but has control over the deployed application and, in some cases, the application hosting environment configuration. Infrastructure as a Service (IaaS): The function provided to the consumer is to provision processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software that may include an operating system and applications. The consumer does not manage or control the underlying cloud infrastructure, but has limited control over the operating system, storage, control of the deployed applications, and, in some cases, selection of network components (e.g., host firewall).

[0083] The deployment model is as follows. Private cloud: The cloud infrastructure is operated only for a certain organization. This can be managed by that organization or a third party and can exist on-premises or off-premises. Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). This can be managed by those organizations or a third party and can exist on-premises or off-premises. Public cloud: The cloud infrastructure is available to the general public or large industry groups and is owned by an organization that sells cloud services. Hybrid cloud: The cloud infrastructure remains a distinct entity but is a combination of two or more clouds (private, community, or public) linked together by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0084] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is the infrastructure that includes a network of interconnected nodes.

[0085] Referring now to FIG. 12, an exemplary cloud computing environment 1250 enabling the use cases of the present invention is depicted. As shown, the cloud computing environment 1250 includes one or more cloud computing nodes 1210 with which local computing devices such as, for example, a personal digital assistant (PDA) or cellular telephone 1254A, desktop computer 1254B, laptop computer 1254C, or in-vehicle computer system 1254N, or combinations thereof, used by cloud consumers can communicate. The nodes 1210 can communicate with one another. These can be physically or virtually grouped (not shown) to form one or more networks such as a private cloud, community cloud, public cloud, or hybrid cloud as described above, or combinations thereof. Thereby, the cloud computing environment 1250 can provide infrastructure, platform, software, or combinations thereof, as services such that cloud consumers need not maintain resources on local computing devices. The types of computing devices 1254A-N shown in FIG. 12 are intended as examples only, and it is understood that the computing nodes 1210 and cloud computing environment 1250 can communicate with any type of computerized device via any type of network or network addressable connection, or both (e.g., using a web browser).

[0086] FIG. 13 is a schematic diagram of an exemplary abstraction model layer according to an embodiment of the present invention. It should be understood in advance that the components, layers, and functions shown in FIG. 13 are intended merely as illustrations and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided.

[0087] The hardware and software layer 1360 includes hardware and software components. Examples of hardware components include mainframe 1361, RISC (Reduced Instruction Set Computer) architecture-based server 1362, server 1363, blade server 1364, storage device 1365, and network and networking components 1366. In some embodiments, the software components include network application server software 1367 and database software 1368.

[0088] The virtualization layer 1370 provides an abstraction layer that can provide the following examples of virtual entities: virtual server 1371, virtual storage 1372, virtual network 1373 including a virtual private network, virtual applications and operating systems 1374, and virtual clients 1375.

[0089] In one example, the management layer 1380 can provide the functions described below.

[0090] Resource provisioning 1381 provides for the dynamic procurement of computing resources and other resources utilized to execute tasks within a cloud computing environment.

[0091] Metering and Pricing 1382 provides cost tracking when resources are utilized within a cloud computing environment and billing or invoicing for the consumption of these resources. As an example, these resources can include application software licenses. Security provides identification verification for cloud consumers and tasks and protection for data and other resources. User Portal 1383 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 1384 provides allocation and management of cloud computing resources so that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 1385 provides pre - placement and procurement of cloud computing resources whose future requirements are predicted according to the SLA.

[0092] Workload Layer 1390 provides examples of functions that can utilize the cloud computing environment for that purpose. Examples of workloads and functions that can be provided from this layer include Mapping and Navigation 1391, Software Development and Lifecycle Management 1392, Virtual Classroom Education Delivery 1393, Data Analytics Processing 1394, Transaction Processing 1395, and Graph - based Color Description in Cloud Servers 1396.

[0093] As used herein, the terms "data," "content," "information," and like terms can be used interchangeably to refer to data that can be captured, transmitted, received, displayed, or stored, or any combination thereof, according to various exemplary embodiments. Thus, the use of any such terms should not be taken as limiting the spirit and scope of the present disclosure. Further, when a computing device is described herein as receiving data from another computing device, the data can be received directly from the other computing device or indirectly via one or more intervening computing devices such as, for example, one or more servers, repeaters, routers, network access points, base stations, or combinations thereof.

[0094] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, as well as a keyboard and a pointing device by which the user can provide input to the computer, such as, for example, a mouse or a trackball. Other kinds of devices can also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input received from the user can be in any form, including, but not limited to, acoustic, speech, or tactile input.

[0095] The present invention can be a system, a method, or a computer program product, or any combination thereof. The computer program product can include one or more computer-readable storage media having computer-readable program instructions thereon for causing a processor to implement aspects of the present invention.

[0096] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction-executing device. The computer-readable storage medium can be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing, but is not limited thereto. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves having recorded instructions, and any suitable combination of the foregoing. A computer-readable storage medium, as used herein, should not be construed to be a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0097] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or can be downloaded from an external computer or an external storage device via, for example, the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage to a computer-readable storage medium within the respective computing / processing device.

[0098] The computer-readable program instructions for carrying out the operations of the present invention may be any combination of source code or object code written in one or more programming languages including, for example, assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or any combination of one or more programming languages such as object-oriented programming languages like Smalltalk or C++ and conventional procedural programming languages like the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, may be executed in part on the user's computer as a stand-alone software package, may be executed in part on the user's computer and in part on a remote computer, or may be executed entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to customize the electronic circuit to implement aspects of the present invention.

[0099] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0100] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for implementing the functions / operations specified in one or more blocks of a flowchart, a block diagram, or both. These computer-readable program instructions can be stored in a computer-readable storage medium that can direct a computer, a programmable data processing apparatus, or other devices to function in a particular manner, such that the computer-readable medium storing the instructions contains a manufacture including instructions for implementing the aspects of the functions / operations specified in one or more blocks of a flowchart, a block diagram, or both.

[0101] The computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks of a flowchart, a block diagram, or both.

[0102] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram can represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently depending on the functionality involved, or these blocks may sometimes be executed in the reverse order. It should also be noted that each block of the block diagrams or flowchart diagrams, or combinations of blocks in the block diagrams or flowchart diagrams or both, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions.

[0103] As used herein, references to "one embodiment" or "an embodiment" of the principles and other variations thereof mean that the particular features, structures, characteristics, etc. described in connection with that embodiment are included in at least one embodiment of the principles. Accordingly, the appearances of the phrase "in one embodiment" or "in an embodiment" and other variations in various places throughout this specification are not necessarily all referring to the same embodiment.

[0104] It should be understood that any use of the following: " / " , "and / or" , and "at least one of" , such as in "A / B" , "A and / or B" , and "at least one of A and B" , is intended to include the selection of only the first-listed option (A) , or only the second-listed option (B) , or the selection of both options (A and B) . As a further example , in the case of "A , B , and / or C" and "at least one of A , B , and C" , such expressions are intended to include the selection of only the first-listed option (A) , or only the second-listed option (B) , or only the third-listed option (C) , or the selection of only the first and second-listed options (A and B) , or the selection of only the first and third-listed options (A and C) , or the selection of only the second and third-listed options (B and C) , or the selection of all three options (A and B and C) . This can be extended for as many listed items as will be readily understood by those of skill in this art and related arts .

[0105] Preferred embodiments of systems and methods for generating descriptions of colors in natural language (which are exemplary and not intended to be limiting) have been described , but it should be noted that modifications and variations can be made by those of skill in the art in light of the above teachings . Thus , it will be understood that changes can be made in the particular embodiments described within the scope of the invention as outlined by the appended claims . In this way , while the aspects of the invention have been described with the detail and specificity required by patent law , those who wish to be claimed and protected by a patent are defined by the appended claims .

Claims

1. A method for generating a description of a color in natural language, executed by a computer, comprising: obtaining a list of tuples, each tuple including a color and a description; generating a graph by using each of the tuples as a node and adding an edge between the nodes when a color difference between the nodes is outside a predetermined range from the perspective of human perception; filtering the edges based on external color comparison descriptions stored in an external color comparison database; finding the nearest neighbor node based on the color difference from the perspective of human perception, and incorporating the new node into the graph by adding a new edge between the new node and the nearest neighbor node, wherein the new node includes a new color not included in the list of tuples and has no description; learning a feature vector for each of the nodes in the graph by using message passing and using the color of the nodes as an initial seed; generating a description of the new node by using each of the learned feature vectors as an initial state of a decoder based on a recurrent neural network (RNN) in a semi-supervised manner; A method comprising the above.

2. The method according to claim 1, wherein filtering the edges includes removing redundant edges from the graph.

3. The method according to claim 1, wherein the color difference from the perspective of human perception is provided by Delta E which is a metric.

4. The method according to claim 3, wherein an edge is generated between two colors when the Delta E between the two colors is between 11 and 49.

5. The method according to claim 1, wherein after the new node is added to the graph, a description for the new node is learned by using context information from the edges from a set of labeled nodes to a set of unlabeled nodes.

6. The method according to claim 1, wherein the loss in the RNN is calculated for the labeled part of the graph.

7. The method according to claim 1, wherein the new node is defined as an unlabeled node.

8. A method for generating a description of a color in natural language, executed by a computer, comprising: obtaining a list of tuples, each tuple including a color and a description; Generating a graph by using each of the tuples as a node and adding edges between the nodes based on a similarity threshold; Filtering the edges based on external color comparison descriptions stored in an external color comparison database; Finding the nearest neighbor node based on the similarity threshold and incorporating the new node into the graph by adding a new edge between the new node and the nearest neighbor node; A method comprising.

9. The method according to claim 8, wherein the new node includes a new color not included in the list of the tuples and has no description.

10. The method according to claim 9, further comprising learning a feature vector for each of the nodes in the graph.

11. The method according to claim 10, wherein the feature vector is learned by using message passing and using the color of the node as an initial seed.

12. The method according to claim 11, further comprising generating a description of the new node by designating each of the learned feature vectors as an initial state.

13. The method according to claim 12, wherein the initial state is provided to a recurrent neural network (RNN)-based decoder in a semi-supervised manner.

14. The method according to claim 13, wherein the similarity threshold is a color difference in terms of human perception between the nodes outside a predetermined range.

15. The method according to claim 14, wherein the color difference in terms of human perception is provided by Delta E which is a metric.

16. The method according to claim 15, wherein an edge is generated between two colors when Delta E between the two colors is between 11 and 49.

17. The method according to claim 16, wherein the loss in the RNN is calculated for the labeled part of the graph.

18. The method according to claim 8, wherein filtering the edges includes removing redundant edges from the graph.

19. The method according to claim 8, wherein after the new node is added to the graph, a description for the new node is learned by using context information from the edges from a set of labeled nodes to a set of unlabeled nodes.

20. A method for generating a description of a color in natural language, which is executed by a computer, comprising: Generating a graph that structures the relationship of colors, including a plurality of nodes and edges; Removing redundant edges based on color comparison descriptions; Incorporating new nodes together with new edges into the graph based on a similarity threshold; Learning a feature vector for each node of the graph; Generating a description of the new node using the feature vector, wherein the feature vector is provided to a neural network in a semi-supervised manner; A method comprising the above.

21. A computer program for generating a description of a color in natural language, which is executable by a computer and causes the computer to execute the method according to any one of Claims 1 to 20.

22. A computer-readable storage medium storing the computer program according to Claim 21.

Images