Apparatus and Method for Identifying Attribute and Object

Abstract

The present disclosure provides an attribute-object identification device and method capable of accurately identifying attribute-object combinations that have not been learned in an image by using an object embedding vector obtained by performing neural network operations on an image through a backbone network as a guide, emphasizing a plurality of intermediate feature maps extracted during the process of obtaining the object embedding vector, generating an attribute embedding vector considering the object context based on the emphasized plurality of intermediate feature maps, and obtaining an attribute-object feature vector by combining and embedding the object embedding vector and the attribute embedding vector.

Description

Technology Field

[0001] The present disclosure relates to an attribute-object identification device and method, and more specifically, to an attribute-object identification device and method capable of identifying an object along with an attribute of said object in an image. Background Technology

[0002] Currently, artificial neural networks are being actively utilized in a wide variety of fields, and are particularly useful in image processing areas such as object recognition, object identification, object detection, and object segmentation.

[0003] Furthermore, regarding object identification, conventional methods were limited to classifying objects by merely identifying their classes within an image; however, recently, attribute-object identification techniques that extract not only the object but also its attributes have been proposed and are being utilized. Nevertheless, to identify objects along with their attributes, artificial neural networks had to be trained using data that combined both attributes and objects. Consequently, there is a problem in that the required attribute-object identification performance is not achieved for attribute-object combinations other than those learned.

[0004] For example, even if an artificial neural network is trained on each attribute-object combination, such as Blue Car and Old Dog, there is a problem in that it cannot accurately identify Old Car or Blue Dog, which are variations of the attribute-object combination. In particular, since attribute-object combinations require context to be considered, identification is even more difficult. In the above example, Old Dog and Old Car are cases where the same attribute is combined with different objects, but the "Old" in Old Dog and the "Old" in Old Car have different meanings (old, worn out) in context. Therefore, attribute-object combinations can only be identified if the context is understood.

[0005] Furthermore, attributes and objects can be combined in a wide variety of ways, and it is practically impossible to train each of these attribute-object combinations individually; consequently, there is a limitation in that attribute-objects cannot be identified from images with the required performance. The problem to be solved

[0006] The object of the present disclosure is to an attribute-object identification device and method capable of accurately identifying attribute-object combinations that have not been learned in an image.

[0007] The purpose of the present disclosure is to provide an attribute-object identification device and method capable of using an object embedding vector as a guide to highlight a plurality of intermediate feature maps extracted during the process of acquiring an object embedding vector, and generating an attribute embedding vector that considers the object context based on the highlighted plurality of intermediate feature maps. means of solving the problem

[0008] According to one embodiment of the present disclosure, an attribute-object identification device comprises: a memory; and a processor that executes at least a portion of an operation according to a program stored in the memory, wherein the processor uses an object embedding vector obtained by performing neural network operations on an image through a backbone network as a guide, emphasizes a plurality of intermediate feature maps extracted during the process of obtaining the object embedding vector, generates an attribute embedding vector considering the object context based on the emphasized plurality of intermediate feature maps, and obtains an attribute-object feature vector by combining and embedding the object embedding vector and the attribute embedding vector.

[0009] The above processor can extract a feature map from an image using a backbone network comprising a plurality of computation layers, each performing neural network operations, and obtain an object embedding vector by encoding the final feature map output from the final computation layer of the backbone network.

[0010] The processor can generate an attribute embedding vector by using the object embedding vector as a guide to emphasize and encode multiple intermediate feature maps extracted from multiple intermediate computation layers of the backbone network.

[0011] The processor obtains a plurality of combined feature maps by scaling and combining each of the plurality of intermediate feature maps and the object embedding vector to the same size, and

[0012] Multiple attribute highlighting maps can be obtained by performing pooling and neural network operations on each of the above multiple combined feature maps.

[0013] The processor can obtain a plurality of attribute feature maps by weighting the plurality of attribute highlighting maps to the corresponding intermediate feature maps and adding them to the intermediate feature maps again, and obtain an attribute embedding vector by combining and encoding the plurality of attribute feature maps.

[0014] The processor can average pool each of the plurality of combined feature maps in the 2D spatial axis direction and the channel axis direction, respectively, perform neural network operations to obtain a channel emphasis map and a spatial emphasis map, and obtain an attribute emphasis map by multiplying the channel emphasis map and the spatial emphasis map.

[0015] The processor converts each of a plurality of candidate attribute words and a plurality of candidate object words into a vector, combines the converted vectors into a plurality of combinations, and then embeddings them to obtain a plurality of attribute-object word vectors, and can identify combinations of attribute words and object words based on the similarity between each of the plurality of attribute-object word vectors and the attribute-object feature vectors.

[0016] The processor can obtain multiple attribute word vectors and multiple object word vectors by performing neural network operations on multiple candidate attribute words and multiple candidate object words obtained in advance, and combine the multiple attribute word vectors and the multiple object word vectors according to all possible combinations, and encode the attribute word vectors and object word vectors combined according to each combination to obtain the multiple attribute-object word vectors.

[0017] The processor calculates the similarity between each of the attribute-object feature vector and the plurality of attribute-object word vectors, determines the attribute-object word vector with the highest calculated similarity, and can output a combination of an attribute word and an object word according to the determined attribute-object word vector among the plurality of candidate attribute words and the plurality of candidate object words.

[0018] The processor calculates and sums the similarity between each of the attribute-object feature vector and each of the plurality of attribute-object word vectors, along with the similarity between each of the attribute word vector and the object word vector according to the attribute-object word vector and each of the attribute embedding vector and the object embedding vector according to the attribute-object word vector, determines the attribute-object word vector with the highest summed similarity, and can output a combination of the attribute word and object word according to the determined attribute-object word vector among the plurality of candidate attribute words and the plurality of candidate object words.

[0019] According to another embodiment of the present disclosure, an attribute-object identification method is a method performed by a processor, comprising: a step of emphasizing a plurality of intermediate feature maps extracted during the process of obtaining an object embedding vector by using an object embedding vector obtained by performing neural network operations on an image through a backbone network as a guide; a step of generating an attribute embedding vector considering an object context based on the emphasized plurality of intermediate feature maps; and a step of obtaining an attribute-object feature vector by combining and embedding the object embedding vector and the attribute embedding vector. Effects of the invention

[0020] The attribute-object identification apparatus and method of the present disclosure can accurately identify attribute-object combinations that have not been learned in an image by acquiring an object embedding vector from an image, using the object embedding vector as a guide to emphasize a plurality of intermediate feature maps extracted during the process of acquiring the object embedding vector, and generating an attribute embedding vector that considers the object context based on the emphasized plurality of intermediate feature maps. Brief explanation of the drawing

[0021] FIG. 1 shows a schematic configuration of an attribute-object identification device according to one embodiment, classified according to operation. Figure 2 is a diagram illustrating the operation of the attribute-object feature combination vector acquisition module in the attribute-object identification device of Figure 1. Figure 3 shows an example of the detailed configuration of the object embedding vector acquisition module and the attribute embedding vector acquisition module of Figure 1. Figure 4 shows an example of the detailed configuration of the object guide highlighting module of Figure 3. Figure 5 is a diagram illustrating the operation of the object guide highlighting module of Figure 4. Figure 6 is a diagram illustrating the operation of the attribute-object word combination vector acquisition module of Figure 1. FIG. 7 illustrates an attribute-object identification method according to one embodiment. FIG. 8 is a drawing for explaining a computing environment including a computing device according to one embodiment. Specific details for implementing the invention

[0022] Hereinafter, specific embodiments according to embodiments of the present disclosure will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0023] In describing the embodiments of the present disclosure, detailed descriptions of known technology related to the present invention are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the embodiments. Furthermore, terms described below are defined with consideration of their functions in the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe specific embodiments and should not be limiting. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as “include” or “compose” are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described. Additionally, terms such as “...part,” “...unit,” “module,” and “block” described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0024] FIG. 1 shows a schematic configuration of an attribute-object identification device according to one embodiment, divided according to operation; FIG. 2 is a diagram for explaining the operation of an attribute-object feature combination vector acquisition module in the attribute-object identification device of FIG. 1; FIG. 3 shows an example of the detailed configuration of the object embedding vector acquisition module and the attribute embedding vector acquisition module of FIG. 1; FIG. 4 shows an example of the detailed configuration of the object guide highlighting module of FIG. 3; FIG. 5 is a diagram for explaining the operation of the object guide highlighting module of FIG. 4; and FIG. 6 is a diagram for explaining the operation of the attribute-object word combination vector acquisition module of FIG. 1.

[0025] Referring to FIG. 1, an attribute-object identification device according to one embodiment may primarily include an attribute-object feature vector acquisition module, an attribute-object word vector acquisition module (50), and a word combination selection module (60).

[0026] The attribute-object feature vector acquisition module receives an image as input and performs neural network operations to acquire object embedding vectors contained in the image. Simultaneously, using the acquired object embedding vectors as a guide, it emphasizes multiple intermediate feature maps extracted during the acquisition process and generates attribute embedding vectors that consider the object context based on these emphasized intermediate feature maps. Finally, it combines and embeds the object embedding vectors and attribute embedding vectors to obtain an attribute-object feature vector (p x Acquires ).

[0027] Meanwhile, the attribute-object word vector acquisition module (50) converts each of the multiple candidate attribute words and multiple candidate object words into a vector, combines them in various combinations, and then embeddings them to obtain multiple attribute-object word vectors (s y Acquires ).

[0028] The word combination selection module (60) determines an attribute-object word vector similar to the attribute-object feature vector obtained from the attribute-object feature vector acquisition module among a plurality of attribute-object word vectors obtained from the attribute-object word vector acquisition module (50), and combines and outputs an attribute word and an object word according to the determined attribute-object word vector. That is, it identifies a combination of an attribute and an object.

[0029] Specifically, the attribute-object feature vector acquisition module may include an input module (10), a backbone network (21), an object encoder (22), an attribute embedding vector acquisition module (30), and an attribute-object feature combination module (40).

[0030] The input module (10) may include an image acquisition module (11) and a location information generation module (12). The image acquisition module (11) acquires an image containing an object to be identified. The location information generation module (12) may generate location information for each pixel or area of ​​the image acquired by the image acquisition module (11).

[0031] The backbone network (21) is implemented as a trained artificial neural network and obtains a feature map by performing neural network operations on an image obtained from the input module (10). The backbone network (21) is implemented as an artificial neural network including a plurality of computation layers (70 to 7N) that each perform neural network operations, and each computation layer receives an image or the output of a previously placed layer and performs neural network operations to obtain a feature map. That is, the backbone network (21) has a plurality of computation layers connected in a cascade manner to repeatedly extract features and obtain a feature map. Here, the feature map obtained from the computation layer (71 to 7N-1) placed in the middle among the plurality of computation layers (70 to 7N) connected in a cascade manner is an intermediate feature map (z 1 ~ z N-1 ) and the feature map output from the computation layer (7N) placed last is called the final feature map (z N It is called ).

[0032] The backbone network (21) may utilize various neural networks used in the field of image processing, but here it is assumed that a Vision Transformer (ViT) is used as an example. As is known, since the Vision Transformer utilizes location information of each region in the image during neural network computation, the input module (10) includes a location information generation module (12). Therefore, the location information generation module (12) can be viewed as a part of the backbone network (21), and it may be omitted if the backbone network (21) is implemented as another neural network that does not require location information.

[0033] The object encoder (22) outputs the final feature map (z) from the last computational layer (7N) of the backbone network (21). N ) is input and encoded using neural network computation to produce an object embedding vector (p o ) is obtained. The object encoder (22) encodes the final feature map and projects it into a virtual embedding space, thereby obtaining an object embedding vector (p o Acquires ).

[0034] Meanwhile, the attribute embedding vector acquisition module (30) obtains a plurality of intermediate feature maps (z) from the backbone network (21). n , where n ∈ {0, … , N-1}) is input, and the object embedding vector (p) obtained from the object encoder o Guided by ), a plurality of authorized intermediate feature maps (z n By emphasizing each ), multiple attribute feature maps ( ) obtains. And multiple obtained attribute feature maps( attribute embedding vector(p) from ) a Extracts ).

[0035] Referring to FIGS. 2 and 3, the attribute embedding vector acquisition module (30) may include a plurality of object guide enhancement modules (8l, 8m, 8h) and a global pooling module (89). Each of the plurality of object guide enhancement modules (8l, 8m, 8h) obtains an intermediate feature map (z) from a corresponding computation layer (7l, 7m, 7h) among a plurality of computation layers of the backbone network (21). l , z m , z h ) is received, and the object embedding vector (p) obtained from the object encoder (22) o Using ) as a guide, the authorized intermediate feature map (z l , z m , z h By emphasizing each ), the attribute feature map( , , Acquires ).

[0036] Here, in order to reduce the amount of computation of the attribute embedding vector acquisition module (30), the intermediate feature map (z) output from the first, m, and h computation layers (7l, 7m, 7h) of the attribute embedding vector acquisition module (30) l , z m , z h It is illustrated assuming that only three object guide enhancement modules (8l, 8m, 8h) are provided. However, the attribute embedding vector acquisition module (30) is provided with object guide enhancement modules in a number (N-1) corresponding to the number of computation layers (70 to 7N) of the backbone network (21), and intermediate feature maps (z) output from the remaining computation layers (70 to 7N-1) excluding the final computation layer (7N) 0 , … , z N-1 It may also be configured to have all of ) authorized.

[0037] In FIGS. 4 and 5, as an example, the intermediate feature map (z) output from the nth operation layer (7n) n ) authorized attribute feature map( An object guide enhancement module (8n) for obtaining ) is illustrated. Referring to FIGS. 4 and 5, the object guide enhancement module (8n) may include a scaling module, a feature object combining module (93), a channel enhancement module, a spatial enhancement module, an attribute enhancement map acquisition module (98), and an attribute feature map acquisition module (99).

[0038] The scaling module is the object embedding vector (p o ) and intermediate feature map(z n It includes an object scaling module (91) and a feature scaling module (92) that each receive and expand into a 3D map of a specified size (H × W × D). In the backbone network (21), as a plurality of computation layers (70 to 7N) sequentially perform neural network computations, the 3D feature map (z) output from each computation layer (70 to 7N) 0 , … , z N The size of ) changes. In particular, the two-dimensional size of height (H) and width (W) changes. Accordingly, the feature scaling module (92) applies the applied intermediate feature map (z n By scaling ) so that it has a specified common size, the expanded intermediate feature map (Z n Acquires ).

[0039] And the object scaling module (91) is an object embedding vector (p) which is a one-dimensional vector. o ) also performs a transformation to have a specified common size, thereby extending the object feature map (P o ) is obtained. At this time, the object scaling module (91) obtains an object embedding vector (p) which is a one-dimensional vector. o Scaling can be performed by repeatedly arranging ) in a two-dimensional plane to have a height (H) and width (W) according to a common size.

[0040] The feature object combining module (93) is an expanded intermediate feature map (Z) converted to the same common size in the scaling module. n ) and extended object feature map (P oCombine (concatenation) the combined feature map (U n ) obtains. The feature object combining module (93) obtains the attribute embedding vector (p) from the attribute embedding vector acquisition module (30). a In the process of acquiring the object embedding vector (p o Extended intermediate feature map (Z) so that contextual information with the object specified by ) is reflected n ) and extended object feature map (P o Combines ).

[0041] The channel enhancement module includes a spatial pooling module (94) and a channel enhancement map acquisition module (95). The spatial pooling module (94) includes a combined feature map (U n ) is received and average value pooling is performed in the 2D spatial axis direction of height (H) and width (W) to convert it into a vector in the channel axis direction, and the channel emphasis map acquisition module (95) performs a neural network operation (here, as an example, a 1 × 1 convolution operation) to obtain a channel emphasis map (A) in which important information is emphasized on the channel axis. c n Acquires ).

[0042] Meanwhile, the spatial enhancement module includes a channel pooling module (96) and a spatial enhancement map acquisition module (96). The channel pooling module (96) includes a combined feature map (U n ) is received and average value pooling is performed in the channel direction to convert it into a 2D feature map, and the spatial enhancement map acquisition module (97) performs a neural network operation (for example, a 3 × 3 convolution operation) to obtain a spatial enhancement map (A) in which important information is emphasized in space. s n Acquires ).

[0043] The attribute highlighting map acquisition module (98) is a channel highlighting map (A c n ) and spatial emphasis map(A s n Multiplying ) to attribute highlight map(A n = A c n × A sn ) is obtained. And the attribute feature map acquisition module (99) obtains an attribute highlight map (A) as shown in FIG. 5. n ) and extended intermediate feature map(Z n Multiply the elements of ) and expand the intermediate feature map (Z n Adding ) to the attribute feature map( Acquires ).

[0044] Meanwhile, in FIG. 3, the global pooling module (89) obtains attribute feature maps (8l, 8m, 8h) respectively from a plurality of object guide emphasis modules (8l, 8m, 8h). , , Each of these is global average pooled (GAP) and combined and passed to the attribute encoder (32).

[0045] The attribute encoder (32) pools global average values ​​and combines the attribute feature map (GAP( ), GAP( ), GAP( )) is input and encoded using neural network computation to obtain an attribute embedding vector (p a ) is obtained. The attribute encoder (32) is similar to the object encoder (22) in that it pools and combines the attribute feature map (GAP( ), GAP( ), GAP( By encoding )) and projecting it into a virtual embedding space, the attribute embedding vector (p a Acquires ).

[0046] Here, the attribute embedding vector (p a ) is the object embedding vector (p o It can be viewed as attribute features of regions contextually corresponding to objects in the image being extracted, guided by ). For example, object embedding vector (p o If ) is a vector for k in the image, then the attribute embedding vector (p a ) can be seen as having extracted attribute features for cars such as Old or Blue.

[0047] The attribute-object feature combining module (40) is an object embedding vector (p o ) and attribute embedding vector(p a It receives ) and combines and encodes the attribute-object feature vector (p x ) is obtained. As illustrated in FIG. 1, the attribute-object feature combining module (40) may include an embedding vector combining module (41) and a combined feature encoder (42). The embedding vector combining module (41) obtains object embedding vectors (p) obtained from the object encoder (22) and the attribute encoder (32), respectively. o ) and attribute embedding vector(p a ) is concatenated, and the concatenated feature encoder (42) encodes it into a neural network operation and projects it into an embedding space, thereby obtaining an attribute-object feature vector (p x Acquires ).

[0048] Meanwhile, referring to FIGS. 1 and FIGS. 6, the attribute-object word vector acquisition module (50) may include a word storage module (51), a vector conversion module (52), and a word vector encoder (53).

[0049] The word storage module (51) stores multiple candidate words for each of the attributes and objects. The word storage module (51) can be implemented as a storage device such as a database or memory, and multiple attribute words and multiple object words can be stored as candidate words to be identified by the attribute-object identification device. In addition, multiple attribute words (a) and multiple object words (o) can be stored separately from each other.

[0050] The vector conversion module (52) vector-converts each of a plurality of attribute words (a) and a plurality of object words (o) to obtain a plurality of attribute word vectors (w(a)) and a plurality of object word vectors (w(o)). The vector conversion module (52) can be implemented with a pre-trained artificial neural network, and, for example, can be implemented with word2vec, which is well known as a neural network that converts words into vectors.

[0051] The word vector encoder (53) combines and encodes a plurality of attribute word vectors (w(a)) and a plurality of object word vectors (w(o)) vector-converted in the vector conversion module (52) to form a plurality of attribute-object word vectors (s y Acquires ).

[0052] Here, the vector transformation module (52) can be seen as projecting individual attribute words (a) and object words (o) into an embedding space, respectively, to obtain attribute word vectors (w(a)) and object word vectors (w(o)), and the word vector encoder (53) projects the result of combining the attribute word vectors (w(a)) and object word vectors (w(o)) into an embedding space to obtain attribute-object word vectors (s y It can be seen as acquiring )

[0053] The word combination selection module (60) is the attribute-object feature vector (p) obtained from the attribute-object feature vector acquisition module. x ) and a plurality of attribute-object word vectors (s) obtained from the attribute-object word vector acquisition module (50) y ) Calculate the similarity between each and the attribute-object word vector (s) with the highest similarity y It can determine ). Here, similarity can be calculated as cosine similarity. And the attribute word and object word according to the determined attribute-object word vector can be combined and output as a result.

[0054] However, attribute-object feature vector (p x ) and attribute-object word vector(s y The similarity between ) is a comparison of attribute-object combinations and may differ from the similarity between words and features of individual objects and individual attributes. To prevent such errors from occurring, the word combination selection module (60) uses an attribute-object feature vector (p x ) and attribute-object word vector(s y Similarity between (cos(px ,s y Along with calculating )) attribute-object word vector(s y The attribute word vector (w(a)) and object word vector (w(o)) respectively according to ), and the attribute embedding vector (p a ) and object embedding vector(p o Similarity between each (cos(p a ,w(a)), cos(p o ,w(o))) are calculated together, and each calculated similarity is summed to calculate the final similarity (c(y)) as in Equation 1, and the attribute word and object word according to the attribute-object word vector determined according to the calculated final similarity (c(y)) can be combined.

[0055]

[0056] Even though the backbone network (21) and vector transformation module (52) of the above-described attribute-object identification device utilize a pre-trained neural network, the object encoder (22), attribute encoder (32), combined feature encoder (42), and word vector encoder (53) must be additionally trained separately.

[0057] In the learning process, the object encoder (22) and the attribute encoder (32) can be learned by backpropagating the cross-entropy loss calculated by mathematical formulas 2 and 3 according to the object truth value and attribute truth value of the learning data, respectively.

[0058]

[0059]

[0060] Here, L obj and L att are object embedding vectors (p) respectively o ) and attribute embedding vector(p a Represents the cross-entropy loss for ), where O represents the object set, A represents the attribute set, and τ o wa τ arepresents the loss hyperparameters, respectively.

[0061] And the combined feature encoder (42) and the word vector encoder (53) are attribute-object feature vectors (p x ) and attribute-object word vector(s y In order to project ) into a common embedding space, it can be learned by backpropagating the cross-entropy loss calculated by Equation 4 according to the attribute-object combination truth values ​​of the training data.

[0062]

[0063] Here, Y s It represents all possible combinations between multiple attribute words and multiple object words stored in the word storage module (51).

[0064] A technique utilizing an artificial neural network to identify attribute-object combinations has been proposed in the past, and a compositional zero-shot learning technique has been proposed to enable the identification of unlearned attribute-object combinations by utilizing a backbone network (21) in common during object identification and attribute identification to reflect the association between attributes and objects. However, in the existing zero-shot learning technique, the learning proceeded by identifying attributes and objects based on the final output of the backbone network (21) and combining the identified attributes and objects. However, this existing method had a limitation in that it could not reflect the heterogeneous characteristics of attributes and objects within a single image. For example, in a combination of “blue car” in a single image, the part representing the expression “blue” and the area representing the expression “car” may be different from each other, so if the final output of the same backbone network is used, it is not easy to identify this difference. In other words, the attribute “blue” can be applied not only to “car” but also to various other objects such as “sky” within the image, and can appear in various areas unrelated to the object “car”. This resulted in limitations in attribute-object identification performance.

[0065] In contrast, the present disclosure obtains a plurality of intermediate feature maps from a backbone network (21) and an object embedding vector (p o After highlighting areas requiring attention in the intermediate feature map using ) as a guide, the attribute embedding vector (p a ) is obtained. Therefore, by highlighting regions associated with objects in advance and enabling the extraction of features for attributes, the performance of attribute-object identification can be significantly improved. In addition, the attribute-object feature vector (p) resulting from the combination of individual attribute words and individual object words x ) and attribute-object word vector(s ySince it identifies attribute-object combinations based on similarity between them, it can accurately identify even unlearned attribute-object combinations.

[0066] In the illustrated embodiments, each component may have different functions and capabilities other than those described below, and may include additional components other than those not described below. Additionally, in one embodiment, each component may be implemented using one or more physically separated devices, or by one or more processors or a combination of one or more processors and software, and may not be clearly distinguished in specific operation as in the illustrated examples.

[0067] And the attribute-object identification device illustrated in FIG. 1 may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or specific-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc. Additionally, the device may be implemented as a system-on-chip (SoC) including one or more processors and controllers.

[0068] Furthermore, the attribute-object identification device may be installed in the form of software, hardware, or a combination thereof on a computing device or server equipped with hardware elements. A computing device or server may refer to various devices that include, in whole or in part, communication devices such as communication modems for communicating with various devices or wired / wireless communication networks, memory for storing data for executing programs, and microprocessors for executing programs to perform calculations and commands.

[0069] FIG. 7 illustrates an attribute-object identification method according to one embodiment.

[0070] Referring to FIG. 7, the attribute-object identification method of the present disclosure first has a backbone network (21) including a plurality of computation layers (70 to 7N) receive an image and perform neural network computation to extract features, and a plurality of intermediate feature maps (z l , z m , z h ) and final feature map(z N ) is obtained (101). At this time, the intermediate feature map (z l , z m , z h ) is a feature map output from some computation layers (7l, 7m, 7h) placed in the middle among multiple computation layers (70 to 7N).

[0071] And the acquired final feature map (z N Encoding ) with neural network operations to obtain object embedding vector(p o ) obtain (102). object embedding vector (p o When ) is obtained, the obtained object embedding vector (p o Using ) as a guide, multiple intermediate feature maps (z l , z m , z h Each highlights the area containing the attribute that requires attention (103). And the highlighted intermediate feature map (z l , z m , z h Combine ) and encode with neural network operations to obtain attribute embedding vectors (p a ) obtains (105).

[0072] Object embedding vector (p o ) and attribute embedding vector(p a When ) is obtained, the object embedding vector (p o ) and attribute embedding vector(p a Combine ) and encode with neural network operations to obtain the attribute-object feature vector (p x Acquires ).

[0073] Meanwhile, multiple attribute words and multiple object words, which are candidate words prepared and stored in advance, are each converted into vectors to obtain multiple attribute word vectors (w(a)) and multiple object word vectors (w(o)); the obtained multiple attribute word vectors (w(a)) and multiple object word vectors (w(o)) are combined and encoded according to all possible combinations to obtain multiple attribute-object word vectors (s y ) obtains (107).

[0074] Attribute-Object Feature Vector (p x ) and multiple attribute-object word vectors(s y Calculate the similarity between ) (108). At this time, the attribute-object feature vector (p x ) and multiple attribute-object word vectors(s y Not only the similarity between ), but also the attribute embedding vectors (p a Similarity between ) and attribute word vector (w(a)) and object embedding vector (p o The similarity between ) and the object word vector (w(o)) can be calculated together, and the final similarity (c(y)) can be obtained by summing each calculated similarity.

[0075] Then, based on the calculated similarity, the combination of attribute-object words according to the attribute-object word vector with the greatest similarity is extracted and output as a result (109).

[0076] Although FIG. 7 describes each process as being executed sequentially, this is merely an illustrative description, and a person skilled in the art can apply various modifications and variations by changing the order described in FIG. 7, executing one or more processes in parallel, or adding other processes, within the scope of not departing from the essential characteristics of the embodiment of the present invention.

[0077] FIG. 8 is a drawing for explaining a computing environment including a computing device according to one embodiment.

[0078] In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those not described below. The illustrated computing environment (120) may include a computing device (121) to perform the attribute-object identification method illustrated in FIG. 7. In one embodiment, the computing device (121) may be one or more components included in the attribute-object identification device illustrated in FIG. 1.

[0079] A computing device (121) includes at least one processor (122), a computer-readable storage medium (123), and a communication bus (125). The processor (122) may enable the computing device (121) to operate according to the exemplary embodiment described above. For example, the processor (122) may execute one or more programs (124) stored in the computer-readable storage medium (123). The one or more programs (124) may include one or more computer-executable instructions, and the computer-executable instructions may be configured to enable the computing device (121) to perform operations according to the exemplary embodiment when executed by the processor (122).

[0080] The communication bus (125) interconnects various other components of the computing device (121), including the processor (122) and the computer-readable storage medium (123).

[0081] The computing device (121) may also include one or more input / output interfaces (126) and one or more communication interfaces (127) that provide interfaces for one or more input / output devices (128). The input / output interfaces (126) and the communication interfaces (127) are connected to a communication bus (125). The input / output devices (128) may be connected to other components of the computing device (121) through the input / output interfaces (126). Exemplary input / output devices (128) may include input devices such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or output devices such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (128) may be included inside the computing device (121) as a component constituting the computing device (121), or it may be connected to the computing device (121) as a separate device distinct from the computing device (121).

[0082] Although the present invention has been described in detail above through representative embodiments, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims.

Claims

Claim 1 An attribute-object identification device comprising: a memory; and a processor that executes at least a portion of an operation according to a program stored in the memory, wherein the processor uses an object embedding vector obtained by performing neural network operations on an image through a backbone network as a guide to emphasize a plurality of intermediate feature maps extracted during the process of obtaining the object embedding vector, generates an attribute embedding vector considering the object context based on the emphasized plurality of intermediate feature maps, combines and embeds the object embedding vector and the attribute embedding vector to obtain an attribute-object feature vector, wherein the processor uses the object embedding vector as a guide to emphasize and encode a plurality of intermediate feature maps extracted from a plurality of intermediate operation layers of the backbone network to generate an attribute embedding vector, scales and combines each of the plurality of intermediate feature maps and the object embedding vector to the same size to obtain a plurality of combined feature maps, and performs pooling and neural network operations on each of the plurality of combined feature maps to obtain a plurality of attribute emphasis maps. Claim 2 An attribute-object identification device according to claim 1, wherein the processor extracts a feature map from an image using a backbone network comprising a plurality of computation layers each performing neural network operations, and obtains an object embedding vector by encoding the final feature map output from the final computation layer of the backbone network. Claim 3 delete Claim 4 delete Claim 5 An attribute-object identification device according to claim 1, wherein the processor weights the plurality of attribute highlighting maps to the corresponding intermediate feature maps, adds them to the intermediate feature maps again to obtain the plurality of attribute feature maps, and combines and encodes the plurality of attribute feature maps to obtain the attribute embedding vector. Claim 6 An attribute-object identification device according to claim 1, wherein the processor averages each of the plurality of combined feature maps in the 2D spatial axis direction and the channel axis direction, respectively, performs neural network operations to obtain a channel emphasis map and a spatial emphasis map, and multiplies the channel emphasis map and the spatial emphasis map to obtain the attribute emphasis map. Claim 7 An attribute-object identification device according to claim 1, wherein the processor converts each of a plurality of candidate attribute words and a plurality of candidate object words into a vector, combines the converted vectors into a plurality of combinations and then embeddings them to obtain a plurality of attribute-object word vectors, and identifies combinations of attribute words and object words based on the similarity between each of the plurality of attribute-object word vectors and the attribute-object feature vectors. Claim 8 An attribute-object identification device according to claim 7, wherein the processor obtains a plurality of attribute word vectors and a plurality of object word vectors by performing neural network operations on a plurality of candidate attribute words and a plurality of candidate object words obtained in advance, combines the plurality of attribute word vectors and the plurality of object word vectors according to all possible combinations, and obtains the plurality of attribute-object word vectors by encoding the attribute word vectors and object word vectors combined according to each combination. Claim 9 An attribute-object identification device according to claim 7, wherein the processor calculates the similarity between each of the attribute-object feature vector and the plurality of attribute-object word vectors, determines the attribute-object word vector with the highest calculated similarity, and outputs a combination of an attribute word and an object word according to the determined attribute-object word vector among the plurality of candidate attribute words and the plurality of candidate object words. Claim 10 An attribute-object identification device according to claim 8, wherein the processor calculates and sums the similarity between each of the attribute-object feature vector and each of the plurality of attribute-object word vectors, along with the similarity between each of the attribute word vector and the object word vector according to the attribute-object word vector and each of the attribute embedding vector and the object embedding vector, determines the attribute-object word vector with the highest summed similarity, and outputs a combination of the attribute word and object word according to the determined attribute-object word vector among the plurality of candidate attribute words and the plurality of candidate object words. Claim 11 A method performed by a processor comprising: a step of emphasizing a plurality of intermediate feature maps extracted during the process of obtaining an object embedding vector by using an object embedding vector obtained by performing neural network operations on an image through a backbone network as a guide; a step of generating an attribute embedding vector considering the object context based on the emphasized plurality of intermediate feature maps; and a step of obtaining an attribute-object feature vector by combining and embedding the object embedding vector and the attribute embedding vector, wherein the step of emphasizing the intermediate feature maps involves generating an attribute embedding vector by emphasizing and encoding a plurality of intermediate feature maps extracted from a plurality of intermediate operation layers of the backbone network using the object embedding vector as a guide, obtaining a plurality of combined feature maps by scaling and combining each of the plurality of intermediate feature maps and the object embedding vector to the same size, obtaining a plurality of attribute emphasis maps by performing pooling and neural network operations on each of the plurality of combined feature maps, and obtaining a plurality of attribute feature maps by weighting the plurality of attribute emphasis maps to the corresponding intermediate feature maps and adding them again to the intermediate feature maps. Claim 12 In claim 11, the step of emphasizing the intermediate feature map extracts a feature map from an image using a backbone network comprising a plurality of computation layers each performing neural network operations, and obtains an object embedding vector by encoding the final feature map output from the final computation layer of the backbone network. Claim 13 delete Claim 14 delete Claim 15 In claim 11, the step of emphasizing the intermediate feature map comprises pooling average values ​​of each of the plurality of combined feature maps in the 2D spatial axis direction and the channel axis direction, respectively, performing neural network operations to obtain a channel emphasis map and a spatial emphasis map, and multiplying the channel emphasis map and the spatial emphasis map to obtain the attribute emphasis map, thereby forming an attribute-object identification method. Claim 16 In claim 11, the step of generating the attribute embedding vector is an attribute-object identification method that obtains the attribute embedding vector by combining and encoding the plurality of attribute feature maps. Claim 17 In claim 11, the attribute-object identification method further comprises the step of converting each of a plurality of candidate attribute words and a plurality of candidate object words into a vector, combining the converted vectors into a plurality of combinations, and then embedding them to obtain a plurality of attribute-object word vectors; and the step of identifying a combination of an attribute word and an object word based on the similarity between each of the plurality of attribute-object word vectors and the attribute-object feature vector. Claim 18 In claim 17, the step of obtaining the attribute-object word vector comprises obtaining a plurality of attribute word vectors and a plurality of object word vectors by performing neural network operations on a plurality of previously obtained candidate attribute words and a plurality of candidate object words, respectively, and combining the plurality of attribute word vectors and the plurality of object word vectors according to all possible combinations, and encoding the attribute word vectors and object word vectors combined according to each combination to obtain the plurality of attribute-object word vectors. Claim 19 In claim 17, the identifying step calculates the similarity between each of the attribute-object feature vector and the plurality of attribute-object word vectors, determines the attribute-object word vector with the highest calculated similarity, and outputs a combination of an attribute word and an object word according to the determined attribute-object word vector among the plurality of candidate attribute words and the plurality of candidate object words. Claim 20 In claim 18, the identifying step calculates and sums the similarity between each of the attribute-object feature vector and each of the plurality of attribute-object word vectors, along with the similarity between each of the attribute word vector and the object word vector according to the attribute-object word vector and each of the attribute embedding vector and the object embedding vector according to the attribute-object word vector, determines the attribute-object word vector with the highest summed similarity, and outputs a combination of the attribute word and object word according to the determined attribute-object word vector among the plurality of candidate attribute words and the plurality of candidate object words.

Images