Learning Apparatus and Method for Cross-modal Retrieval

Abstract

The present disclosure provides a learning apparatus and method for cross-modal search capable of improving cross-modal search performance by encoding a plurality of images and a plurality of sentences into an artificial neural network, calculating an explicit score representing the comprehensiveness of each sentence based on the similarity between a plurality of image descriptors and a plurality of sentence descriptors obtained therefrom, according to the ratio of images to which each sentence corresponds, and performing learning based on the explicit score.

Description

Technology Field

[0001] The present disclosure relates to a learning device and method for cross-modal search. Background Technology

[0002] Cross-modal retrieval refers to searching for the item that best matches an input query among comparison items of different modalities. Currently, the field where cross-modal retrieval is most actively utilized is the mutual search between images and sentences. In other words, it searches for cross-modal sentences (or images) against an input image (or sentence).

[0003] Generally, cross-modal search is performed by selecting the item with the highest similarity through similarity measurements between the input query and the comparison items. Furthermore, cross-modal search is primarily carried out using artificial neural networks, which are trained using training data.

[0004] Training data for learning cross-modal search includes multiple images and multiple sentences to describe each image. In this case, multiple sentences (e.g., five) may be labeled for each image. In existing training methods, if a sentence is labeled to an image and forms a pair, the image and sentence are defined as a positive pair, and if they are not labeled, they are defined as a negative pair. Training is then performed by increasing the similarity of positive pairs and decreasing the similarity of negative pairs. In this process, similar images and sentences are trained to move closer to each other in a virtual embedding space, while different images and sentences are trained to move further apart. However, this training method has a problem in that it fails to express the complex relationships between images and sentences because it causes all combinations of images and sentences to move closer or further apart with equal intensity. The problem to be solved

[0005] The object of the present disclosure is to a learning device and method for cross-modal search that can improve cross-modal search performance between images and sentences.

[0006] The object of the present disclosure is to a learning device and method for cross-modal search that can extract image and sentence descriptors to enable more detailed representation by considering the comprehensive-specific relationship between images and sentences during cross-modal search. means of solving the problem

[0007] A learning device for cross-modal search according to one embodiment of the present disclosure comprises: a memory; and a processor that executes at least a portion of an operation according to a neural network model stored in the memory, wherein the processor encodes a plurality of images and a plurality of sentences into an artificial neural network, respectively, and calculates an explicit score representing the comprehensiveness of each sentence based on the ratio of images to which each sentence corresponds, based on the similarity between a plurality of image descriptors and a plurality of sentence descriptors obtained.

[0008] The processor calculates the similarity based on the distance between a plurality of image descriptors and a plurality of sentence descriptors, and can calculate the explicit score as the ratio of the cumulative sum of similarities exceeding a threshold value among the calculated similarities relative to the number of the plurality of images.

[0009] The processor can use the explicit score to calculate a loss for training the artificial neural network so that the distance in the embedding space between a plurality of image descriptors and a plurality of sentence descriptors calculated by the similarity is adjusted.

[0010] The processor can calculate similarity adjustment weights by applying an exponential function to the negative value of the explicit score.

[0011] The processor can calculate and obtain a similarity control weight from the explicit score, and calculate a positive modal mutual loss by weighting the similarity control weight to the distance in the embedding space between the image descriptor and the sentence descriptor for positive pairs that are matched and labeled among the plurality of images and the plurality of sentences.

[0012] The processor calculates and obtains a similarity adjustment weight from the explicit score, and if the distance in the embedding space between the image descriptor and the sentence descriptor for a pair of voices that do not match among the plurality of images and the plurality of sentences is smaller than the negative margin, it can calculate the loss between voice modals by weighting a value that decreases as the similarity adjustment weight increases.

[0013] The processor can calculate a modal internal loss to adjust the distance between a multiple positive sentence representation for each image in the embedding space and a multiple positive sentence representation for a multiple positive sentence that is matched and labeled to the image according to the explicit score.

[0014] The processor can calculate a modal internal loss based on the difference between the distance ratio in the embedding space between each of the multiple positive sentence descriptors for multiple positive sentences that are matched and labeled with the same image as the image descriptor for each image, and the explicit score ratio for the multiple positive sentences.

[0015] A learning method for cross-modal search according to another embodiment of the present disclosure comprises: a process of encoding a plurality of images and a plurality of sentences into an artificial neural network, respectively, to calculate an explicit score representing the comprehensiveness of each sentence based on the similarity between a plurality of image descriptors and a plurality of sentence descriptors obtained, according to the ratio of images to which each sentence corresponds; and a process of calculating a loss to train the artificial neural network so as to adjust the distance between the plurality of image descriptors and the plurality of sentence descriptors calculated by the similarity using the explicit score. Effects of the invention

[0016] The learning device and method for cross-modal search of the present disclosure can improve cross-modal search performance between images and sentences by training a neural network to extract image and sentence descriptors by considering the comprehensive-specific relationship between images and sentences during cross-modal search. Brief explanation of the drawing

[0017] FIG. 1 shows a configuration of a learning device for cross-modal search according to one embodiment, classified according to operation. Figure 2 is a schematic diagram showing the operation of a learning device for cross-modal search of Figure 1. Figure 3 shows an example of training data. FIG. 4 is a drawing for explaining a learning concept performed by a learning device according to the present disclosure. FIG. 5 illustrates a learning method for cross-modal search according to one embodiment. FIG. 6 is a diagram illustrating a computing environment including a computing device according to one embodiment. Specific details for implementing the invention

[0018] Hereinafter, specific embodiments according to embodiments of the present disclosure will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0019] In describing the embodiments of the present disclosure, detailed descriptions of known technology related to the present invention are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the embodiments. Furthermore, terms described below are defined with consideration of their functions in the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe specific embodiments and should not be limiting. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described. Additionally, terms such as "...part," "...unit," "module," and "block" described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0020] FIG. 1 shows a configuration of a learning device for cross-modal search according to one embodiment, classified according to operation, FIG. 2 is a schematic diagram showing the operation of the learning device for cross-modal search of FIG. 1, and FIG. 3 shows an example of learning data. FIG. 4 is a diagram for explaining a learning concept performed by a learning device according to the present disclosure.

[0021] Referring to FIG. 1, a learning device for cross-modal search according to the present disclosure may include an input module (10), a descriptor acquisition module (20), an explicit score calculation module (30), and a loss calculation module (40).

[0022] The input module (10) collects and acquires training data (D) during training and applies the acquired training data (D) to the descriptor acquisition module (20). As described above, since the field where cross-modal search is most actively used is mutual search between images and sentences, it is assumed that the cross-modal search of the embodiment is also mutual search between images and sentences. Accordingly, the input module (10) can acquire data in which images and sentences are mutually matched and labeled as cross-modal items as shown in FIG. 2 as training data (D). Here, as described above, images and sentences may be matched and labeled in a one-to-one manner, but they may also be matched and labeled in a one-to-many manner. For example, as shown in FIG. 3, three sentences may be matched and labeled for each of the multiple images collected as training data, and the number of sentences matched for each image can be adjusted in various ways. In some cases, multiple images may be matched and labeled for each sentence.

[0023] That is, the training data (D) acquired by the input module (10) can be composed of an image set (V) and a sentence set (T) (D = (V, T)), and the images included in the image set (V) v The number of ) and sentences included in the sentence set( tThe number of ) may differ from each other.

[0024] And the input module (10) may acquire a portion of the large amount of training data as a mini-batch and apply it to the descriptor acquisition module (20). Generally, a large amount of training data is required to train an artificial neural network, and accordingly, the input module (10) can collect a vast amount of training data. However, the training device for cross-modal search of the present disclosure is an image ( v ) and sentence( t Training is performed to express the relationship between ) as a generic-to-specific relationship. In this process, images ( v ) and sentence( t If one intends to learn the comprehensive-specific relationship between ) at once, it requires a very large amount of computation, storage capacity, and training time. In other words, it requires a large amount of resources. To prevent such problems from occurring, the input module (10) [requires] some of the training data among the large amount of collected training data in a mini-batch ( Obtained with ), and mini-placement( The training data obtained in units may also be input to the representation acquisition module (20).

[0025] Additionally, the input module (10) can obtain an input query to search for items of a different form during a test operation after training is completed, that is, during a cross-modal search operation, and pass it to the presenter acquisition module (20). Since it is assumed here that a cross-modal search between images and sentences is performed, the input module (10) images ( v ) or sentence( t One of ) can be obtained as an input query and passed to the representation acquisition module (20). When performing a cross-modal search operation, an image ( v When ) is input, the cross-modal search device images ( v The sentence corresponding to ) tYou can select ) as a cross-modal item to output, and the sentence ( as an input query t When ) is entered, the cross-modal search device sentence( t The image corresponding to ) v You can select ) as a cross-modal item to output.

[0026] The input module (10) can be implemented as various devices such as a user interface device, a storage device, a communication device, an image acquisition device, etc.

[0027] When training data or an input query is received from the input module (10), the expression acquisition module (20) performs neural network operations on the received training data or input query to acquire expressions. Then, the expression acquisition module (20) stores the extracted expressions.

[0028] The representation acquisition module (20) is a plurality of images (V) included in the image set (V) and sentence set (T) of the training data (D = (V, T)) during training. v ) and multiple sentences( t By encoding each of them, multiple image representations( v ) and multiple sentence expressors( t Extracts ).

[0029] The representation acquisition module (20) may include a first encoder (21) and a second encoder (22). The first encoder (21) receives an image (applied from the input module (10) v ) neural network operation( F ( v Encode into )) and image representr( v = F ( v )) obtains, and the second encoder (22) obtains the sentence (applied from the input module (10) t ) neural network operation( P ( t Encoded as )) sentence descriptor( t = P ( t)) can be obtained. The first and second encoders (21, 22) can be implemented as artificial neural networks, and as shown in FIG. 2, for example, the first encoder (21) can be implemented as a convolutional neural network and the second encoder (22) can be implemented as a sequential model, but is not limited thereto and various neural networks can be used.

[0030] At this time, the first and second encoders (21, 22) are image representations ( v ) and sentence expressors( t ) can be obtained in the same format. For example, the first and second encoders (21, 22) are image descriptors ( v ) and sentence expressors( t ) can be obtained in a vector format with the same dimensions.

[0031] Meanwhile, the presenter acquisition module (20) may further include an embedding module (23). The embedding module (23) is an image ( v Image descriptor acquired for ) v ) and sentences included in the sentence set (T) t Sentence descriptors obtained for ) t ) stores. The embedding module (23) stores a plurality of image descriptors ( v ) and multiple sentence expressors( t It can be implemented as a storage device that stores ).

[0032] Here, the first and second encoders (21, 22) are image representations ( v ) and sentence expressors( t ) is obtained in a vector format having the same dimension, and the embedding module (23) obtains a plurality of image descriptors ( v ) and multiple sentence expressors( t Since it stores ), the presenter acquisition module (20) has images of different modals (as shown in FIG. 2) v) and sentence( t Image descriptor( v ) and sentence expressors( t It can also be viewed as performing the operation of acquiring ) and projecting them commonly into a virtual embedding space. Here, the image set (V) and the sentence set (T) each have B images ( v 1, v 2, … , v B ) and B sentences( t 1, t 2, … , t B Mini batch including ) It was assumed that training data (D) is applied in units of ), and accordingly, in the embedding space in Fig. 2, B image descriptors ( v 1, v 2, … , v B ) and B sentence descriptors( t 1, t 2, … , t B This represents the case where ) is projected. Here, the identifiers expressed as subscripts in the training data (D) are identical images ( v i ) and sentence( t i ) represents positive pairs that are matched and labeled with each other, and the images forming the positive pairs ( v i ) and sentence( t i Image descriptor obtained from ) v i ) and sentence expressors( t i ) also forms positive pairs. Likewise, images forming negative pairs ( v i ) and sentence( t j , where i ≠ j) image descriptor( v i ) and sentence expressors( t j ) also forms negative pairs.

[0033] The explicit score calculation module (30) obtains a plurality of image descriptors ( from the descriptor acquisition module (20) v ) and multiple sentence expressors( t Using ) multiple sentences( v Calculate a denotational score representing the generality of each.

[0034] The explicit score calculation module (30) is each sentence expression ( t i ) and multiple image presenters( v Calculate the similarity between ) and the image descriptors extracted based on the calculated similarity ( v From ) the corresponding sentences( t j By calculating the explicit score for ), each sentence ( t i Quantifies the comprehensiveness of ).

[0035] Each image included in the training data (D) v i ) is at least one sentence ( t i ) is matched and labeled and accepted as a positive pair. However, since each image generally contains many specific features and each sentence describes only some of the specific features contained in the image, a generic-to-specific relationship exists between the sentence and the image. Therefore, not only can one sentence describe multiple images, but one image can also be described by several different sentences.

[0036] In the example of Fig. 3, the first sentence of the three sentences describing the top image can also be used to describe the other two images. Similarly, the first sentence of the three sentences describing the middle image can also be used to describe the bottom image, and the first and second sentences of the three sentences regarding the bottom image can be used to describe the middle image.

[0037] In other words, this means that all three images can be used to represent the first sentence of the top image, and the middle and bottom images can also be used to represent the first sentence for each image.

[0038] In this way, the set of images described by a specific sentence is a visual denotation ( It can be defined as follows. According to this definition, the greater the number of images a sentence can explain, the greater the size of the visual specification, and a sentence with a large visual specification can be seen as a sentence that possesses comprehensiveness in explaining implied concepts.

[0039] In other words, in the case of comprehensive sentences, they can be matched to various images. Nevertheless, existing learning methods did not consider this comprehensiveness of sentences and performed learning with equal intensity by considering only whether they were labeled and matched to images as training data (D). That is, each sentence ( t i Regardless of the comprehensiveness of ), positive paired images ( v i ) and sentence( t i Image descriptor obtained from ) v i ) and sentence expressors( t i While ) positions them close to each other in the embedding space with uniform intensity, images forming voice pairs ( v i) and sentence( t j , where i ≠ j) image descriptor( v i ) and sentence expressors( t j ) was trained to move away from each other with equal intensity. As a result, although it possesses comprehensiveness, multiple images ( v A sentence that can correspond to ) t i There was a problem where cross-modal search performance was degraded because it was not labeled in the embedding space and was trained to move away from the video forming the speech pair.

[0040] To resolve these problems, the explicit score calculation module (30) of the present disclosure, during learning, a number of sentences ( t ) and multiple images( v Multiple image descriptors obtained from ) v ) and sentence expressors( t In order to make ) get closer or further away with different intensities, each sentence ( t i Calculate an explicit score indicating comprehensiveness for ).

[0041] The explicit score calculation module (30) may include a similarity calculation module (31) and a score calculation module (32). The similarity calculation module (31) includes a number of sentences ( t ) and multiple images( v Similarity between ) (s( v , t Calculate )) and the score calculation module (32) calculates the calculated similarity ( s ( v , t Based on )) multiple sentences( t Explicit scores for each Calculate )

[0042] First, the similarity calculation module (31) calculates each sentence (in the embedding space) t i ) and multiple images( vThe distance between ) d ( v , t i Based on )) according to mathematical formula 1, each sentence ( t i ) and multiple images( v Similarity between ) s ( v , t i Calculate )) according to mathematical formula 1.

[0043]

[0044] Here, ∥·∥2 represents the L2norm function.

[0045] The score calculation module (32) calculates each sentence (as in mathematical formula 2) t i Similarity calculated by the similarity calculation module (31) for ) s ( v , t i Images where )) exceeds the threshold value (τ) v Detecting ) and visual specification which is an image set ( Acquires ).

[0046]

[0047] As mentioned above, visual indication ( Images included in ) v The more ) there are, the more the corresponding sentence( t i ) can be said to be a sentence with comprehensiveness.

[0048] And each sentence ( t i Visual indication of ) The image included in ) v Similarity with ) s ( v , t i Explicit score according to )) Calculate ) according to mathematical formula 3.

[0049]

[0050] Here, κ is a scale factor set as a constant value to adjust the size of the score, and B represents the number of images included in the training data (or mini-batch). According to Equation 3, the explicit score ( ) is a similarity greater than the threshold value (τ) s ( v , t i A value obtained by dividing the cumulative sum of )) by the number of images, for sentences (v) for B images included in the training data (D) t i Visual indication of ) The image included in ) v Similarity with ) s ( v , t i It can be seen as the proportion of ))

[0051] Meanwhile, the loss calculation module (40) calculates the loss and backpropagates the calculated loss to train the first and second encoders (21, 22) implemented as artificial neural networks.

[0052] In the present disclosure, the loss calculation module (40) is the modal inter-loss (L inter ) and modal internal loss (L intra Calculate ) and the calculated intermodal loss (L inter ) and modal internal loss (L intra The first and second encoders (21, 22) can be trained by calculating the total loss by summing the ) and backpropagating. Here, the mutual loss (L inter ) is a set of labeled positive pairs (D) from the training data (D), similar to the existing one. P ) and set of voice pairs (D N Based on ), image descriptors of positive pairs in the embedding space ( v i ) and sentence expressors( t i ) get closer to each other and the image descriptors of voice pairs ( vj ) and sentence expressors( t i ) are set so that they move away from each other. At this time, as illustrated in FIG. 4 (a), each sentence ( t i Explicit score calculated for ) Using ) sentence( t i Positive images that form a positive pair with ) v i Positive image descriptor of ) v i The sound image forming a sound pair with the intensity of pulling ) close ( v i ) audio-visual descriptor( v i It controls the force that pushes ) far away. That is, the sentence ( t i Sentence descriptors for ) t i ) and positive image descriptors( v i Similarity levels between ) and sentence descriptors( t i ) and audio-visual descriptors( v j The similarity level between ) is the explicit score ( It is adjusted according to ).

[0053] In Fig. 4, the circle represents each image ( v i Image representation for ) v i ) represents, and the empty rings are each sentence( t i Sentence descriptors for ) t i It represents ). And the colors of the circle and ring are indicators to distinguish between positive or negative pairs. That is, circles and rings of the same color represent sentence descriptors that form a positive pair ( t i ) and image descriptors( v i) and circles and rings of different colors are sentence descriptors that form phonetic pairs ( t i ) and image descriptors( v j ) is. Also, the size of the ring is the sentence( t i )'s explicit score( It indicates the size of ). The solid line indicates strong pulling or pushing force, and the dotted line indicates weak pulling or pushing force.

[0054] As shown in FIG. 4(a), the modal inter-loss (L inter ) is a sentence expressor( t i For ), multiple images that are items of the cross-modal ( v ) Extracted image descriptors( v Image descriptors of positive pairs among ) v i ) is the distance in the embedding space ( d ( v i , t i While causing )) to get closer, the image descriptor of the voice pair ( v j ) is the distance in the embedding space ( d ( v , t i Make )) move away, but not make them move closer or further away with uniform intensity, and each sentence ( t i An explicit score indicating the comprehensiveness of ) By adjusting the intensity according to ) to make it closer or further away, multiple sentence descriptors ( t ) and multiple image presenters( v The location of ) is the image( v ) and sentence( t It is adjusted more finely according to the comprehensive-specific relationship between ).

[0055] Specifically, intermodal loss (Linter To calculate ) the loss calculation module (40) first calculates the explicit score ( Similarity adjustment weight (w) according to ) t Calculate ). Comprehensive sentences are highly likely to be shared and used across multiple images, whereas specific sentences are highly likely to be used only in specific images. Therefore, the similarity adjustment weight (w t When bringing positive pairs closer, the pulling force of comprehensive sentences is applied more weakly than that of specific sentences. In other words, similarity should be increased in positive pairs of specific sentences, while similarity should be decreased in positive pairs of comprehensive sentences.

[0056] And when trying to make negative pairs move apart, it is necessary to apply force with the opposite tendency. That is, while similarity should be lowered in negative pairs of specific sentences, it should be higher in positive pairs of comprehensive sentences.

[0057] Accordingly, similarity adjustment weights (w t ) can be calculated as in mathematical formula 4.

[0058]

[0059] According to mathematical formula 4, the similarity adjustment weight (w t ) is a negative explicit score( It is calculated as an exponential function value for ), and accordingly, the explicit score( As ) increases, the similarity adjustment weight (w t ) gradually approaches 0.

[0060] Similarity adjustment weight (w t When ) is calculated according to mathematical formula 4, the loss calculation module (40) calculates the modal inter-loss (L inter ) can be calculated as in mathematical formula 5.

[0061]

[0062] Here ( v ,t )~ D p represents a set of image-texts forming positive pairs, and ( v , t )~ D N represents an image-text set forming speech pairs, and α represents the negative margin. And [·] + It represents a hinge function that has an operation value only when the operation value of [·] is positive, and a value of 0 otherwise.

[0063] According to mathematical equation 5, the intermodal loss (L inter ) is a sentence in the embedding space for positive pairs( t ) and multiple images( v The distance between ) d ( v , t Explicit score on )) Similarity adjustment weight (w) that decreases as ) increases t Weight ) and for negative pairs, distance( d ( v , t Similarity adjustment weight (w) only when )) is smaller than the negative margin (α) t It is set to reflect ).

[0064] In mathematical equation 5, for convenience, the intermodal loss (L inter Although it was configured to include both the loss for positive pairs and the loss for negative pairs, the loss for positive pairs and the loss for negative pairs are respectively the mutual positive modal loss (L inter+ ) and intermodal loss (L inter- It can be calculated by separating into ).

[0065] Meanwhile, as mentioned above, each image ( in the training data (D) v i ) contains multiple sentences( t ) can be matched as positive pairs and labeled. Therefore, for the same image, multiple sentences forming a positive image ( tIt needs to be learned so that a comprehensive-specific relationship is maintained even between ). The same image ( v i Multiple sentences forming positive pairs for ) t i Among ) specific sentences( t i,a ) sentence expressor( t i,a ) is a comprehensive sentence( t i,b ) sentence expressor( t i,b Image representation (in embedding space compared to ) v i It must be located closer than ). Modal internal loss (L intra ) is an image descriptor ( v i Same modal sentence descriptor based on ) t i,a , t i,b It is applied to adjust the position of ).

[0066] In Fig. 4(b), the same image descriptor ( v i 5 sentence expressions based on ) t i ) is an explicit score( It can be seen that they are separated by different distances depending on ).

[0067] Modal internal loss (L intra To calculate ), first, each image( v i Multiple sentences forming positive pairs for ) t i Extract some sentences based on various combinations among ) to create a new set of positive pairs (D P+ ) can be obtained. Here, the same image( v i Among the multiple sentences forming positive pairs for ), 2 sentences in different combinations ( t , t+ )extract a new set of positive pairs (D P+ It is assumed that ) is obtained. Set of positive pairs (D P+ If ) is obtained, the modal internal loss (L intra ) can be calculated using mathematical formula 6.

[0068]

[0069] According to mathematical equation 6, the inter-modal loss (L intra ) is the same image( v i Positive sentences extracted in various combinations for ) t , t + Explicit score in the distance ratio with ) It can be calculated by accumulating the value obtained by subtracting the ratio of ).

[0070] The loss calculation module (40) calculates the modal mutual loss (L) according to mathematical formula 5. inter ) is calculated, and according to Equation 6, the modal internal loss (L intra When ) is calculated, the calculated intermodal loss (L inter ) and modal internal loss (L intra The total loss of the sum of ) is backpropagated to train the first and second encoders (21, 22).

[0071] Meanwhile, the cross-modal search device may be configured by adding a cross-item selection module (50) instead of using the loss calculation module (40) in the learning device for cross-modal search of FIG. 1. During a cross-modal search operation, the input module (10) receives an image or sentence as an input query and transmits it to the expression acquisition module (20). Depending on whether the input query is an image or a sentence, one of the first and second encoders (21, 22) of the expression acquisition module (20) encodes the input query to extract the query expression. Then, the embedding module (23) stores the extracted query expression. Then, the similarity calculation module (31) of the explicit score calculation module (30) calculates the similarity between the query expression and the expressions of the cross-modal that are different from the modal of the input query among the multiple expressions stored in the embedding module (23). The cross-item selection module (50) determines the expression that has the highest similarity to the query expression among the cross-modal expressions, and selects and outputs an item for the determined expression. That is, it searches for the cross-modal expression closest to the query expression in the embedding space and outputs the corresponding item.

[0072] In the illustrated embodiments, each component may have different functions and capabilities other than those described below, and may include additional components other than those not described below. Additionally, in one embodiment, each component may be implemented using one or more physically separated devices, or by one or more processors or a combination of one or more processors and software, and may not be clearly distinguished in specific operation as in the illustrated examples.

[0073] And the learning device for cross-modal search illustrated in FIG. 1 may be implemented within a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or specific-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc. Additionally, the device may be implemented as a system-on-chip (SoC) including one or more processors and controllers.

[0074] In addition, the learning device for cross-modal search may be installed in the form of software, hardware, or a combination thereof on a computing device or server equipped with hardware elements. A computing device or server may refer to various devices that include, in whole or in part, communication devices such as communication modems for communicating with various devices or wired / wireless communication networks, memory for storing data for executing programs, and microprocessors for executing programs to perform calculations and commands.

[0075] FIG. 5 illustrates a learning method for cross-modal search according to one embodiment.

[0076] Referring to FIGS. 1 to 4, the learning method for cross-modal search of the present disclosure illustrated in FIG. 5 is described as follows: First, a learning operation is performed (60). Then, during the learning operation, learning data (D) is first obtained (61). Here, the learning data (D) is data in which items that match each other among items of different modals are labeled. Each item may be labeled with cross-modal items matched one-to-one, or may be labeled with items matched one-to-many. Here, as an example, it is assumed that the data is in which images and sentences are mutually matched and labeled.

[0077] When training data (D) is acquired, an image of the acquired training data (D) v ) and sentence( t By encoding each ) using an artificial neural network, the image representation ( v ) and sentence expressors( t ) is extracted (62). Here, the image descriptor ( v ) and sentence expressors( t ) can be extracted in the same format, for example, in a vector format with the same dimension. At this time, all images of the training data (D) collected according to the training data (D) v ) and sentence( t Encoding the image representation( v ) and sentence expressors( t ) can be extracted, but if the training data is very large, some training data can be mini-batched ( Obtained with ), and mini-placement( Image descriptors () in units v ) and sentence expressors( t You can also extract ).

[0078] Image and sentence expressors ( v, t When ) is extracted, a plurality of image descriptors ( projected onto a virtual embedding space) v ) and multiple sentence expressors( t The distance between ) d ( v , t i Similarity based on )) s ( v , t Calculate )) (63). And the explicit score ( As in mathematical formula 3, the sentence ( t i Similarity calculated for ) s ( v , t iIt is calculated as the weight of the cumulative sum of similarities exceeding the threshold value (τ) among )) (64). Explicit score ( ) is the corresponding sentence( t i It is a score that quantifies whether a sentence is comprehensive enough to correspond to multiple images or specific enough to correspond to a specific image, and the more comprehensive the sentence, the larger the value.

[0079] Explicit score ( When ) is calculated, the similarity adjustment weight (w t ) as in mathematical formula 4, negative explicit score( It is calculated as an exponential function value for ) (65). Similarity adjustment weight (w t When ) is calculated, the calculated similarity adjustment weight (w t Based on ) each sentence in the embedding space ( t i Modal inter-image loss (L) is adjusted so that the distance for each of the positive and negative image pairs is adjusted, that is, so that the similarity is adjusted. inter Calculate )(66). Here, the intermodal loss (L inter ) is a large explicit score( Comprehensive sentences having ) t i Image descriptors of the cross-modal images forming positive pairs for ) v ) are sentence expressors( t i While causing it to move further away from ), specific sentences ( t i Image descriptors of the cross-modal images forming positive pairs for ) v ) are sentence expressors( t i Makes it closer to ). And a small explicit score ( A specific sentence having ) t i Image descriptors of the cross-modal images forming positive pairs for ) v) are sentence expressors( t i While making it get closer to ), specific sentences ( t i Image descriptors of images forming voice pairs for ) v ) are sentence expressors( t i Make it move further away from ). However, the image descriptor of the video forming the audio pair ( v ) are sentence expressors( t i It can be set to apply only when the distance from ) is smaller than the negative margin (α).

[0080] Also, each image representation ( v i Multiple sentence descriptors for ) t When ) forms a positive pair, sentence( t An explicit score indicating the comprehensiveness of ) Using ) image descriptors( v i Multiple sentence expressors for ) t Modal internal loss (L) that adjusts the similarity of ) intra Calculate ). Modal internal loss (L intra ) is a video representr( v i Sentence descriptors extracted with ) and various combinations ( t The distance ratio between ) and the extracted sentence( t )'s explicit score( It can be calculated as in mathematical formula 6 based on the ratio (67).

[0081] Modal mutual loss (L inter ) and modal internal loss (L intra When ) is calculated, the calculated intermodal loss (L inter ) and modal internal loss (L intra Backpropagating the total loss (summing ) to the image descriptor( v ) and sentence expressors( tThe first and second encoders (21, 22), which are artificial neural networks that extract ) are trained (68).

[0082] Afterward, it is determined whether to end the training (70). Training may end when the total loss is less than or equal to the reference loss or when the training is performed a specified number of times.

[0083] If it is determined that the learning is finished, a test operation, i.e., a cross-modal search operation (80), can be performed. For the cross-modal search operation, an input query is first received (81). One of the items of different modals may be received as the input query, for example, an image or a sentence may be received. When the input query is received, the query expression is extracted by encoding the received input query with an artificial neural network (82). Then, the similarity between the extracted query expression and the expressions of multiple cross-modal items that have already been extracted and obtained is calculated (83). Among the calculated similarities, the cross-modal item corresponding to the expression with the highest similarity is selected and output (84).

[0084] Although FIG. 5 describes each process as being executed sequentially, this is merely an illustrative description, and a person skilled in the art can apply various modifications and variations by changing the order described in FIG. 5, executing one or more processes in parallel, or adding other processes, within the scope of not departing from the essential characteristics of the embodiment of the present invention.

[0085] FIG. 6 is a diagram illustrating a computing environment including a computing device according to one embodiment.

[0086] In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those not described below. The illustrated computing environment (90) may include a computing device (91) to perform the method for cross-modal search illustrated in FIG. 5. In one embodiment, the computing device (91) may be one or more components included in the device for cross-modal search illustrated in FIG. 1.

[0087] A computing device (91) includes at least one processor (92), a computer-readable storage medium (93), and a communication bus (95). The processor (92) may enable the computing device (91) to operate according to the exemplary embodiment described above. For example, the processor (92) may execute one or more programs (94) stored in the computer-readable storage medium (93). The one or more programs (94) may include one or more computer-executable instructions, and the computer-executable instructions may be configured to enable the computing device (91) to perform operations according to the exemplary embodiment when executed by the processor (92).

[0088] The communication bus (95) interconnects various other components of the computing device (91), including the processor (92) and the computer-readable storage medium (93).

[0089] The computing device (91) may also include one or more input / output interfaces (96) and one or more communication interfaces (97) that provide an interface for one or more input / output devices (98). The input / output interfaces (96) and communication interfaces (97) are connected to a communication bus (95). The input / output devices (98) may be connected to other components of the computing device (91) through the input / output interfaces (96). An exemplary input / output device (98) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (98) may be included inside the computing device (91) as a component constituting the computing device (91), or it may be connected to the computing device (91) as a separate device distinct from the computing device (91).

[0090] Although the present invention has been described in detail above through representative embodiments, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims.

Claims

Claim 1 An apparatus for performing learning for cross-modal search, comprising: a memory; and a processor that executes at least a portion of the operation according to a neural network model stored in the memory, wherein the processor encodes a plurality of images and a plurality of sentences into an artificial neural network, respectively, calculates an explicit score representing the comprehensiveness size of each sentence based on the similarity between a plurality of image descriptors and a plurality of sentence descriptors obtained by encoding each of the images and sentences into an artificial neural network, according to the ratio of images to which each sentence corresponds, and calculates a loss for training the artificial neural network so that the distance in the embedding space between the plurality of image descriptors and the plurality of sentence descriptors calculated by the similarity is adjusted using the explicit score. Claim 2 A device according to claim 1, wherein the processor calculates the similarity based on the distance between a plurality of image descriptors and a plurality of sentence descriptors, and calculates the explicit score as the ratio of the cumulative sum of similarities exceeding a threshold value among the calculated similarities relative to the number of the plurality of images. Claim 3 delete Claim 4 In claim 1, the processor is a device that calculates a similarity adjustment weight by applying an exponential function to the negative value of the explicit score. Claim 5 A device according to claim 1, wherein the processor calculates and obtains a similarity adjustment weight from the explicit score, and calculates a positive modal mutual loss by weighting the similarity adjustment weight to the distance in the embedding space between an image descriptor and a sentence descriptor for positive pairs that are matched and labeled among the plurality of images and the plurality of sentences. Claim 6 A device according to claim 1, wherein the processor calculates and obtains a similarity adjustment weight from the explicit score, and if the distance in the embedding space between an image descriptor and a sentence descriptor for a voice pair that is not matched among the plurality of images and the plurality of sentences is smaller than a negative margin, the device calculates the voice modal mutual loss by weighting a value that decreases as the similarity adjustment weight increases. Claim 7 In claim 1, the processor is a device for calculating modal internal loss to adjust the distance between a multiple positive sentence representation for a multiple positive sentence that is matched to and labeled with the image in an embedding space according to the explicit score. Claim 8 In claim 1, the processor is a device that calculates a modal internal loss based on the difference between the distance ratio in the embedding space between each of the multiple positive sentence descriptors for a plurality of positive sentences that are matched and labeled with the same image as the image descriptor for each image, and the explicit score ratio for the plurality of positive sentences. Claim 9 A learning method for cross-modal search performed by a processor executing at least part of an operation according to a neural network model, comprising: a process of encoding a plurality of images and a plurality of sentences into an artificial neural network, and calculating an explicit score representing the comprehensiveness of each sentence based on the similarity between a plurality of image descriptors and a plurality of sentence descriptors obtained therefrom, according to the ratio of images to which each sentence can correspond; and a process of calculating a loss to train the artificial neural network so as to adjust the distance between the plurality of image descriptors and the plurality of sentence descriptors calculated by the similarity using the explicit score. Claim 10 In claim 9, the process of calculating according to the ratio of the images calculates the similarity according to the distance between a plurality of image descriptors and a plurality of sentence descriptors, and calculates the explicit score as the ratio of the cumulative sum of similarities exceeding a threshold value among the calculated similarities relative to the number of the plurality of images. Claim 11 In claim 9, the process of calculating the loss is a method of calculating similarity adjustment weights by applying an exponential function to the negative value of the explicit score. Claim 12 In claim 9, the process of calculating the loss calculates and obtains a similarity control weight from the explicit score, and calculates the positive modal mutual loss by weighting the similarity control weight to the distance in the embedding space between the image descriptor and the sentence descriptor for positive pairs that are matched and labeled among the plurality of images and the plurality of sentences. Claim 13 In claim 9, the process of calculating the loss calculates and obtains a similarity adjustment weight from the explicit score, and if the distance in the embedding space between the image descriptor and the sentence descriptor for a voice pair that is not matched among the plurality of images and the plurality of sentences is smaller than a negative margin, the method calculates the loss between voice modals by weighting a value that decreases as the similarity adjustment weight increases. Claim 14 In claim 9, the process of calculating the loss is a method for calculating modal internal loss to adjust the distance between an image representation for each image in the embedding space and a plurality of positive sentence representations for a plurality of positive sentences that are matched to and labeled in the image according to the explicit score. Claim 15 In claim 9, the process of calculating the loss is a method for calculating a modal internal loss based on the difference between the distance ratio in the embedding space between each of the multiple positive sentence expressors for each of the multiple positive sentences that are matched and labeled to the same image as the image expressor for each image, and the explicit score ratio for the multiple positive sentences.

Images