Multimodal Retrieval Method and Apparatus Using Geometric Distribution Constraint and Subspace Alignment in a Common Embedding Space

Abstract

A multimodal search method using geometric distribution constraints and subspace alignment of a common embedding space according to one technical aspect of the present invention is performed in a multimodal search device and comprises: a step of converting data of a plurality of different modalities into embedding information of a common embedding space; a step of performing learning for alignment between embedding information of the plurality of different modalities in the common embedding space, while reflecting geometric distribution constraints to control the distribution of embedding information in the common embedding space; a step of learning a multimodal embedding model by reflecting subspace alignment on the embedding information of the plurality of different modalities; and a step of generating a search result based on similarity between a search query and a search target using the learned multimodal embedding model.

Description

Technology Field

[0001] The present invention relates to multimodal search technology, and more specifically, to a multimodal search method and apparatus using geometric distribution constraints and subspace alignment of a common embedding space that improve semantic correspondence accuracy and search consistency between modalities by controlling the geometric distribution of embedding information and reflecting subspace alignment during the process of aligning data of a plurality of different modalities into a common embedding space. Background Technology

[0002] Recently, multimodal artificial intelligence technology is being utilized in various fields to integrally process data with different representation formats, such as text, images, audio, and video. In particular, multimodal embedding technology, which places data from different modalities within the same representation space, is being widely researched to enable users to search for data belonging to other modalities even when inputting a query within a single modality.

[0003] In general multimodal search techniques, data of different modalities, such as text and images, are encoded separately, and alignment between modalities is performed by training to maintain close distances between corresponding data in a common embedding space. For example, contrastive learning structures are commonly used that ensure text and images with the same meaning are adjacent in the common embedding space, while data with different meanings are moved further apart.

[0004] However, a method that sorts simply based on the distance between corresponding data has the problem of failing to sufficiently control the distribution structure of the entire common embedding space. As a result, embedding information may become excessively concentrated in specific areas, or certain semantic axes may become overly dominant, thereby reducing the distinguishability between different semantic units. Furthermore, if data distribution characteristics differ by modality, even data with the same meaning may be aligned only along a specific directional axis while distortion occurs along other axes, which can lead to a decrease in the consistency of search results.

[0005] In particular, if only uniform alignment is performed across the entire common embedding space, differences in representation structures by modality may lead to a phenomenon where certain semantic components are aligned while others are not sufficiently corresponded. In such a structure, even if search accuracy is maintained for some queries, semantic correspondence may appear unstable for other query types.

[0006] Furthermore, during the training of multimodal embedding models, distribution distortion or representation collapse may occur, where the representation of a specific modality converges to a particular region within the common embedding space, or where data from different semantic units fail to maintain a sufficient distance structure. These phenomena can accumulate due to the composition of the training data, bias in the loss function, or differences in representations across modalities, ultimately leading to a degradation in search performance.

[0007] Therefore, in the process of placing data of different modalities into a common embedding space, there is a need for a technology that goes beyond simple pairwise alignment to control the spatial distribution structure of embedding information and performs alignment between modalities based on semantically important subspaces, thereby ensuring more stable and consistent multimodal search performance. Prior art literature

[0008] Republic of Korea Published Patent Application No. 10-2025-0158570 (Date of publication: Nov. 06, 2025) The problem to be solved

[0009] One objective of the present invention is to provide a multimodal search structure capable of more stably maintaining the distribution of embedding information in a common embedding space, thereby mitigating the problem where embedding information of a specific modality is excessively concentrated or the distinguishability between data that should be semantically distinguished is reduced, in order to solve the aforementioned problems. This is achieved by addressing the difficulty in sufficiently controlling the overall distribution structure of embedding information using only alignment based on similarity between corresponding data in conventional multimodal search technologies that align data of different modalities in a common embedding space.

[0010] In addition, another objective is to resolve the problem in conventional technology where, even if data of different modalities are placed in the same space, differences in representation structures by modality can cause data corresponding to the same meaning to be aligned on some axes but distorted on others, or lead to excessive mixing or excessive separation between modalities, thereby degrading search accuracy and consistency. Furthermore, by controlling the geometric distribution of a common embedding space, this invention aims to maintain a balance between alignment between modalities and distinguishability by modality.

[0011] In addition, another objective is to mitigate the problem where conventional multimodal embedding models often perform uniform alignment based on the entire common embedding space, which fails to sufficiently reflect alignment relationships in semantically important axes or sub-regions and consequently leads to unstable search performance in specific query types or modality combinations, and to improve alignment precision for semantically important representational components by performing sub-space alignment based on the embedding information of a reference modality.

[0012] In addition, another objective is to mitigate the problem where the reliability and reproducibility of search results may be degraded due to the representation of a specific modality converging to a limited area within the common embedding space or the occurrence of distribution distortion or collapse during the training or operation of a multimodal embedding model, and to improve the stability and maintainability of search performance by providing a structure capable of control and correction that reflects the distribution state of embedding information.

[0013] In addition, another objective is to provide a multimodal search method and apparatus that can mitigate the problem of difficulty in consistently ensuring the performance of a multimodal search model in actual application environments with various modality combinations and data characteristics, and improve semantic correspondence accuracy, search consistency, and operational stability for multiple modalities by utilizing geometric distribution constraints and subspace alignment in a common embedding space.

[0014] However, the problems to be solved in this disclosure are not limited to those mentioned above, and may be expanded in various ways without departing from the spirit and scope of this disclosure. means of solving the problem

[0015] One technical aspect of the present invention provides a multimodal search method using geometric distribution constraints and subspace alignment of a common embedding space. The method is performed in a multimodal search device and includes the steps of converting data of a plurality of different modalities into embedding information of a common embedding space and calculating a search result based on similarity between a search query and a search target using a multimodal embedding model. The multimodal embedding model is learned based on geometric distribution constraints for controlling the distribution of embedding information in the common embedding space and subspace alignment of the embedding information.

[0016] In one embodiment, the multimodal search method may further include: a step of performing learning for alignment among embedding information of a plurality of different modalities in the common embedding space, while reflecting geometric distribution constraints to control the distribution of embedding information in the common embedding space; and a step of learning a multimodal embedding model by reflecting subspace alignment for the embedding information of the plurality of different modalities.

[0017] In one embodiment, the data of the plurality of different modalities may include training data of at least two different modalities corresponding to the same content, and the step of converting into embedding information of the common embedding space may include the step of encoding the training data of the at least two different modalities into feature information using a modality-specific encoder, and the step of mapping the feature information into mutually comparable embedding information in the common embedding space using a projection layer.

[0018] In one embodiment, the step of reflecting the geometric distribution constraint may include: a step of calculating a geometric distribution constraint loss that controls at least one of volume, separation, aggregation, and diversity for the distribution of the embedding information in order to control the distribution shape of the embedding information in the common embedding space; and a step of reflecting the geometric distribution constraint loss in learning for alignment between the embedding information.

[0019] In one embodiment, the step of performing learning for alignment between the embedding information may include: a step of calculating an alignment loss that increases the similarity between embedding information of different modalities corresponding to the same content and decreases the similarity between embedding information corresponding to different content; and a step of training the multimodal embedding model by combining the alignment loss and the geometric distribution constraint loss to reflect both semantic alignment and distribution stabilization of the embedding information in the common embedding space.

[0020] In one embodiment, the step of training the multimodal embedding model by reflecting the subspace alignment may include: generating a reference subspace based on embedding information of an anchor modality among a plurality of different modalities; projecting embedding information of a target modality onto the reference subspace; and calculating a subspace alignment loss that reflects the degree of alignment between the anchor modality and the target modality using the projection result on the reference subspace, and reflecting the subspace alignment loss in the training.

[0021] In one embodiment, the step of generating the reference subspace may include performing singular value decomposition or principal component analysis on a matrix composed of embedding information of the anchor modality, and selecting some of the upper components obtained as a result of the singular value decomposition or principal component analysis to set the reference subspace.

[0022] In one embodiment, the step of incorporating the subspace alignment loss into the learning process may include: a step of calculating alignment reliability based on the similarity between the embedding information of the anchor modality and the embedding information of the target modality; and a step of calculating the subspace alignment loss for embedding information where the alignment reliability is greater than or equal to a reference value, or calculating the subspace alignment loss by applying a weight corresponding to the alignment reliability.

[0023] In one embodiment, after the step of training the multimodal embedding model, the method may further include the step of monitoring a distribution indicator of embedding information generated by the multimodal embedding model, the step of detecting a distribution distortion or collapse of the embedding information in the common embedding space based on the distribution indicator and determining the projection layer as a target for correction, and the step of correcting the multimodal embedding model by updating the projection layer while keeping the modality-specific encoder fixed.

[0024] Another technical aspect of the present invention provides a multimodal search device using geometric distribution constraints and subspace alignment of a common embedding space. The device comprises at least one processor and a memory for storing instructions. When the instructions are executed by the at least one processor, the processor performs the operation of converting data of a plurality of different modalities into embedding information of a common embedding space, and the operation of generating a search result based on similarity between a search query and a search target using a multimodal embedding model. The multimodal embedding model is learned based on geometric distribution constraints for controlling the distribution of embedding information in the common embedding space and subspace alignment of the embedding information. Effects of the invention

[0025] According to various embodiments of the present invention, in the process of aligning data of multiple different modalities into a common embedding space, geometric distribution constraints for controlling the distribution of embedding information in the common embedding space are reflected together with alignment between embedding information. This provides the effect of mitigating distribution distortion, concentration in specific regions, and representation collapse, which are difficult to control solely through learning centered on similarity between corresponding data. Accordingly, the distribution stability and semantic distinction of embedding information by modality can be improved.

[0026] Furthermore, according to various embodiments of the present invention, the distribution of embedding information in a common embedding space is configured to control at least one of volume, separation, aggregation, and diversity, thereby providing the effect of ensuring sufficient distinguishability between different semantic units while maintaining appropriate aggregation between data corresponding to the same meaning. Accordingly, it is possible to mitigate the degradation of search performance caused by excessive mixing or excessive separation between modalities and to improve the accuracy and consistency of multimodal search.

[0027] Furthermore, according to various embodiments of the present invention, a reference subspace is generated based on the embedding information of an anchor modality among a plurality of different modalities, and the embedding information of a target modality is projected onto the reference subspace to perform subspace alignment. This configuration provides the effect of precisely performing alignment between modalities based on semantically important expression components that are difficult to sufficiently reflect through uniform alignment of the entire common embedding space alone. Accordingly, semantic correspondence precision can be improved even in specific modality combinations or specific query types.

[0028] Furthermore, according to various embodiments of the present invention, when configured to control whether to calculate subspace alignment loss or to control weights based on alignment reliability, it provides the effect of reflecting alignment centered on samples with high alignment reliability. Accordingly, it is possible to mitigate the phenomenon where learning becomes unstable due to samples with high noise or high alignment uncertainty, and to improve the stability and efficiency of subspace alignment.

[0029] Furthermore, according to various embodiments of the present invention, by monitoring the distribution indicator of embedding information generated by a multimodal embedding model and determining the projection layer as a target for correction when distribution distortion or collapse in the common embedding space is detected, and by updating the projection layer while keeping the modality-specific encoder fixed, the effect of responding to distribution anomalies without retraining the entire model is provided. Accordingly, the possibility of recovering and maintaining search quality can be improved while reducing the burden of responding to performance degradation during operation.

[0030] Furthermore, according to various embodiments of the present invention, the search query and the search target are converted into embedding information of the same common embedding space and search results are calculated based on their similarity, thereby providing the effect of applying a consistent search procedure to various modality combinations such as text versus image, image versus text, and voice versus text. Accordingly, the universality and scalability of the multimodal search system can be improved in various application environments.

[0031] Furthermore, according to various embodiments of the present invention, by utilizing a structure that incorporates both geometric distribution constraints and subspace alignment of a common embedding space, the effect of simultaneously achieving semantic alignment and distribution stabilization is provided compared to conventional learning structures centered on simple pairwise alignment. Accordingly, the search accuracy, search consistency, and operational stability of a multimodal search model can be comprehensively improved.

[0032] The effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects will be obvious to those skilled in the art to which the present invention belongs from the description below. Brief explanation of the drawing

[0033] FIG. 1 is a configuration diagram illustrating the overall configuration of a multimodal search system according to one embodiment of the present invention. FIG. 2 is a flowchart illustrating the overall procedure of a multimodal search method according to one embodiment of the present invention. FIG. 3 is a block diagram illustrating the detailed configuration of a multimodal search device according to one embodiment of the present invention. FIG. 4 is a diagram illustrating the concept of embedding information distribution control and subspace alignment in a common embedding space according to an embodiment of the present invention. FIG. 5 is a flowchart illustrating a geometric distribution constraint and subspace alignment-based multimodal embedding model learning procedure according to one embodiment of the present invention. FIG. 6 is a flowchart or functional flowchart illustrating a detailed processing procedure for geometric distribution constraints and subspace alignment according to an embodiment of the present invention. FIG. 7 is a flowchart illustrating the offline embedding generation and ANN indexing procedure for a search target item according to one embodiment of the present invention. FIG. 8 is a flowchart illustrating an online multimodal search execution procedure according to an embodiment of the present invention. FIG. 9 is a flowchart illustrating an operation and correction loop that detects quality degradation or distribution collapse during the operation of a multimodal search system according to an embodiment of the present invention, performs a correction operation in response thereto, and then performs verification and distribution. FIG. 10 is a flowchart illustrating a procedure for identifying a distribution collapse type of a common embedding space and selectively correcting a projection layer corresponding to the collapse type according to an embodiment of the present invention. FIG. 11 is a flowchart illustrating a hierarchical anchor subspace alignment and axis-by-axis selection alignment procedure according to one embodiment of the present invention. Specific details for implementing the invention

[0034] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0035] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, each of phrases such as A or B, at least one of A and B, at least one of A or B, A, B or C, at least one of A, B and C, and at least one of A, B, or C may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as first, second, or first or second may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as coupled or connected to another (e.g., second) component, with or without the terms functionally or communicationally, it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0036] As used in the various embodiments of this document, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0037] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., memory) readable by a machine or device. For example, the processor of the machine or device may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by a machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0038] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0039] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0041] The meanings of the key terms used in this specification are as follows, and each term is defined from a functional perspective to aid in understanding the invention. The following definitions of terms are intended to describe embodiments of the invention and should not be interpreted as unduly limiting the scope of the invention.

[0042] "Common embedding space" refers to a representation space in which embedding information generated from multiple different modalities is arranged so that it can be compared under the same distance or similarity criteria. The common embedding space may be configured to align feature representations corresponding to text, images, voice, or other modalities within the same coordinate system or dimensional structure, and may be a space in which semantic proximity, cluster structure, and distribution characteristics between representations of each modality are commonly reflected. The common embedding space may be implemented as a single continuous space, or, if necessary, as a structure in which different constraints are applied to specific partial regions.

[0043] "Embedding information" refers to information in which features extracted from input data are expressed in the form of numeric vectors in a common embedding space or a corresponding representation space. The said embedding information may be generated through a modality-specific encoder or a post-transform layer, and each dimension may reflect semantic attributes, structural characteristics, or relational information. The said embedding information may be used as a unit of operation for search, sorting, clustering, or distribution analysis.

[0044] "Geometric distribution constraints" refer to constraints imposed during the learning process to ensure that embedding information maintains a specific distribution shape or spatial structure within a common embedding space. These geometric distribution constraints can control at least one of the following characteristics: distance distribution, directionality, density, inter-cluster spacing, variance, or axial occupancy between embedding information, and can be applied to prevent representation collapse or concentration in specific regions that may occur with simple pairwise alignment alone. These geometric distribution constraints can be implemented in the form of a loss function, normalization conditions, distribution indicator-based control, or a combination thereof.

[0045] "Subspace alignment" refers to aligning embedding information of different modalities based on a sub-region composed of specific sub-dimensions or specific combinations of axes, rather than the entire common embedding space. This subspace alignment can be performed by projecting the embedding information of another modality onto principal components, singular vectors, or a selected set of axes extracted from the embedding information of a reference modality, and by evaluating or correcting the degree of alignment based on the projection results. This subspace alignment can be configured to prioritize the maintenance of consistency of specific semantic axes over full space alignment.

[0046] "Anchor modality" refers to a modality among multiple modalities that is used as a reference standard when aligning subspaces or setting a reference distribution. The anchor modality may be selected if it has relatively high semantic stability or is suitable for forming a reference axis of the data distribution, and it may be set as text data, image data, or one of other modalities. The anchor modality may be set fixedly or may be changed dynamically depending on the training stage or data type.

[0047] "Target modality" refers to a modality that is subject to alignment or correction within a common embedding space based on an anchor modality. The target modality may include embedding information generated from data having a corresponding relationship with the anchor modality, and may be subject to subspace projection or alignment loss calculation.

[0048] A "projection layer" refers to a transformation layer for mapping feature information generated by a modality-specific encoder to embedding information in a common embedding space. The projection layer may be composed of a linear transformation, a non-linear transformation, or a multi-layer structure, and may perform the role of correcting dimensional differences or representation distribution differences between modalities. The projection layer may be updated together with the modality-specific encoder during the learning process, and in certain embodiments, it may be selectively re-corrected while the modality-specific encoder is fixed.

[0049] "Alignment reliability" refers to a value that numerically represents the degree to which embedding information of different modalities corresponds to the same semantic unit. The alignment reliability can be calculated based on similarity, distance, orientation matching, subspace projection results, or multiple evaluation indicators between embedding information, and depending on the alignment reliability, whether to incorporate a specific sample into the learning process or the loss weight can be determined.

[0050] "Distribution distortion or collapse" refers to a state in which the distribution of embedding information within a common embedding space is excessively concentrated in a specific area, the distinguishability between modalities is reduced, or embedding information that needs to be semantically distinguished fails to maintain a sufficient distance structure. The aforementioned distribution distortion or collapse may be detected in the form of reduced variance, increased overlap between clusters, imbalanced occupancy by axis, or biased distance distribution; if such a state is detected, correction of the projection layer or application of additional constraints may be performed.

[0052] FIG. 1 is a configuration diagram illustrating the overall configuration of a multimodal search system according to one embodiment of the present invention.

[0053] The system illustrated in FIG. 1 may include a user terminal (10), a multimodal search device (100), a search target storage device (200), and a model management device (300). FIG. 1 schematically illustrates the overall application environment of the present invention, and the internal detailed configuration of the multimodal search device (100) can be described in more detail with reference to FIG. 3, which will be described later.

[0054] In the system according to FIG. 1, the user terminal (10) may be a device for a user to use a multimodal search service. The user terminal (10) may receive a search query in the form of text, image, audio, video, or a combination thereof and transmit it to a multimodal search device (100). Additionally, the user terminal (10) may receive search results from the multimodal search device (100) and provide them to the user through a screen, speaker, or other output means. In one embodiment, the user terminal (10) may be a smartphone, tablet terminal, laptop, desktop computer, or server terminal, but is not limited thereto.

[0055] The multimodal search device (100) is the core processing entity of the present invention and can perform a multimodal search on a search query received from a user terminal (10) and a search target stored in a search target storage device (200). More specifically, the multimodal search device (100) can convert data of multiple different modalities into embedding information of a common embedding space and calculate the similarity between the search query and the search target using a multimodal embedding model that reflects geometric distribution constraints and subspace alignment to control the distribution of embedding information in the common embedding space. Accordingly, the multimodal search device (100) can produce search results that reflect semantic correspondence relationships even between data belonging to different modalities.

[0056] The search target storage device (200) may be a storage space for storing search target data. The search target storage device (200) may store the search target itself, consisting of images, text, audio, video, or a combination thereof, and may also store embedding information corresponding to the search target and index information for improving search efficiency. The multimodal search device (100) may read the search target or the corresponding embedding information from the search target storage device (200) to perform a search, and may reflect the updated embedding information and index information in the search target storage device (200) when a new search target is added or updated.

[0057] The model management device (300) may be a device for managing a multimodal embedding model used in a multimodal search device (100). In one embodiment, the model management device (300) may store and manage learned model parameters, correction parameters of the projection layer, indicators related to the distribution of embedding information in a common embedding space, and model version information. Additionally, the model management device (300) may check the reflection status of geometric distribution constraints or the subspace alignment status based on the distribution status of embedding information or search performance information collected from the multimodal search device (100), and provide a corrected model or updated parameters to the multimodal search device (100) if necessary.

[0058] Referring to FIG. 1, a user terminal (10) and a multimodal search device (100) can be connected bidirectionally. Accordingly, the user terminal (10) can provide a search query to the multimodal search device (100), and the multimodal search device (100) can return a search result to the user terminal (10). Additionally, the multimodal search device (100) and a search target storage device (200) can be connected bidirectionally, and the multimodal search device (100) can read search target or index information from the search target storage device (200) and can store generated or updated embedding information and index information in the search target storage device (200). In addition, the multimodal search device (100) and the model management device (300) can also be connected bidirectionally, and the multimodal search device (100) can receive model parameters or correction information from the model management device (300), and the model management device (300) can collect information on the distribution status of embedding information, search quality indicators, or model operation status from the multimodal search device (100).

[0059] In one embodiment, the multimodal search device (100) may represent a search query and a search target as embedding information in the same common embedding space and perform a search using the similarity between the embedding information. In this case, the multimodal search device (100) may not only perform simple embedding alignment but may also reflect geometric distribution constraints related to cluster radius, distance between classes, suppression of collapse, and maintenance of distribution diversity of the embedding information in the common embedding space. Additionally, the multimodal search device (100) may set a reference subspace based on an anchor modality and reflect subspace alignment by projecting the embedding information of the target modality onto the reference subspace. Accordingly, semantic alignment and distribution stabilization between data of different modalities can be achieved simultaneously.

[0060] Additionally, in one embodiment, the model management device (300) may be utilized to detect distribution distortion or collapse in which the distribution of embedding information generated by the multimodal search device (100) converges excessively to a specific region or the distinguishability between modalities is reduced. In this case, the model management device (300) and the multimodal search device (100) may update the model by correcting the projection layer while keeping the modality-specific encoder fixed. With this structure, representation stability in the common embedding space can be restored or maintained without retraining the entire model.

[0061] The configuration illustrated in FIG. 1 is merely an exemplary representation of one embodiment of the present invention, and in actual implementation, the user terminal (10), multimodal search device (100), search target storage device (200), and model management device (300) may be implemented by being integrated into a single physical device or distributed across two or more devices. For example, the search target storage device (200) may be implemented in the form of a storage unit inside the multimodal search device (100), and the model management device (300) may be implemented in a separate server or cloud environment.

[0062] According to the system configuration of Fig. 1, one embodiment of the present invention can provide a multimodal search structure that aligns data of a plurality of different modalities into a common embedding space from a system perspective, and by enabling the operation and management of a multimodal embedding model that reflects geometric distribution constraints and subspace alignment of embedding information, search accuracy, search consistency, and operational stability can be improved.

[0064] FIG. 2 is a flowchart illustrating the overall procedure of a multimodal search method according to one embodiment of the present invention.

[0065] The procedure according to FIG. 2 may be a procedure of securing data of multiple different modalities, generating embedding information of a common embedding space from said data, forming a multimodal embedding model by performing alignment learning between said embedding information, reflecting geometric distribution constraints, and reflecting subspace alignment, then performing a search for a search query using said multimodal embedding model, and performing selective correction based on the distribution state of the embedding information. By such a procedure, one embodiment of the present invention can provide a multimodal search method that achieves semantic alignment and distribution stabilization in a common embedding space, while also being able to respond to distribution distortion or collapse during the search operation process.

[0066] Referring to FIG. 2, in step S210, the multimodal search device (100) can obtain data of multiple different modalities. Here, the data of multiple different modalities may include learning data of at least two different modalities corresponding to the same content. For example, one image and a text description corresponding to the image, one video and subtitle or audio data corresponding to the video, one audio clip and text transcription data corresponding to the audio, etc., may be obtained. By obtaining multiple modality data corresponding to the same content in this way, a learning foundation that reflects the semantic correspondence relationship between different modalities can be provided.

[0067] In step S210, the multimodal search device (100) may receive training data from an external storage, a search target storage device (200), a model management device (300), or a separate data collection device. The training data may include text, images, audio, video, and combinations thereof, and each data may be accompanied by metadata such as language information, domain information, quality information, or time of creation information. For example, caption text may be obtained for image data, and frame sequences, subtitles, and background audio may be obtained for video data. Due to this structure, cross-modal alignment for the same semantic unit can be performed more precisely in subsequent steps.

[0068] In step S220, the multimodal search device (100) can convert the obtained data of multiple different modalities into embedding information in a common embedding space. Specifically, the multimodal search device (100) can encode the data of the multiple different modalities into feature information using a modality-specific encoder for each, and map the feature information into embedding information that can be compared with one another in the common embedding space using a projection layer. Here, the common embedding space may be a representation space configured so that embedding information generated from multiple different modalities can be compared under the same similarity criterion or distance criterion.

[0069] In step S220, the multimodal search device (100) may perform different preprocessing processes depending on the format of the data for each modality. For example, tokenization, length normalization, or sentence segmentation may be performed on text data, resizing, patch segmentation, or normalization may be performed on image data, and segmentation, spectrum transformation, or resampling may be performed on audio data. The preprocessed data may be input into an encoder corresponding to each modality and converted into feature information, and the feature information may be converted into embedding information of a common embedding space by a projection layer. In this case, the projection layer may be implemented in the form of a projection head including a linear layer, a plurality of linear layers, a non-linear layer, or a normalization layer.

[0070] In step S230, the multimodal search device (100) can perform alignment learning between embedding information generated in a common embedding space. In this step, the multimodal search device (100) can calculate an alignment loss that increases the similarity between embedding information of different modalities corresponding to the same content and decreases the similarity between embedding information corresponding to different content. For example, when text embedding information and image embedding information describe the same content, the multimodal search device (100) can perform learning in a direction that reduces the distance or angle difference between the two embedding information, and for text and image embedding information that are unrelated to each other, it can perform learning in a direction that increases the distance or angle difference. The alignment learning can be implemented in the form of a contrast learning loss, a matching learning loss, or a combination thereof.

[0071] In step S230, the multimodal search device (100) may distinguish between positive samples and negative samples during the process of calculating alignment loss. For example, pairs of text and images corresponding to the same content may be used as positive samples, and pairs of text and images corresponding to different content may be used as negative samples. The alignment learning may serve to bring embedding information having the same meaning close to each other in a common embedding space. Although such alignment learning forms the basis for semantic alignment, there may be limitations in directly controlling the distribution structure of the entire common embedding space; therefore, it may be performed in combination with geometric distribution constraints and subspace alignment described in subsequent steps.

[0072] In step S240, the multimodal search device (100) may reflect geometric distribution constraints to control the distribution of embedding information in a common embedding space. Specifically, to control the distribution shape of embedding information in a common embedding space, the multimodal search device (100) may calculate a geometric distribution constraint loss that controls at least one of volume, separation, aggregation, and diversity for the distribution of said embedding information, and may reflect said geometric distribution constraint loss in learning for alignment between said embedding information. The geometric distribution constraint loss may be configured to control at least one of volume, separation, aggregation, and diversity for the distribution of said embedding information. By combining the alignment loss and the geometric distribution constraint loss in this way, the multimodal search device (100) may learn a multimodal embedding model to achieve semantic alignment and distribution stabilization of said embedding information together in the common embedding space.

[0073] In step S240, the multimodal search device (100) can suppress the phenomenon in which embedding information is compressed into an excessively low-dimensional portion through volume constraints. For example, the degrees of freedom or spatial occupancy of the distribution can be measured based on the Gram matrix, covariance matrix, or singular value spectrum of a matrix composed of multiple modality embedding information corresponding to the same content. If the measurement result converges to a specific threshold range or lower, the multimodal search device (100) determines that the embedding information is collapsing in a specific narrow direction and can reflect geometric distribution constraint loss in a direction that suppresses the collapse. Through such a structure, the representation collapse phenomenon in which nearly identical embedding information is generated for different queries can be mitigated. At this time, the multimodal search device (100) can calculate the volume constraint loss using at least one of the determinant value, singular value sum, or rank value corresponding to the distribution shape.

[0075] In step S240, the multimodal search device (100) can suppress excessive mixing between modalities through separation constraints. For example, by calculating the difference in distance, angle, or covariance between the center vectors of embedding information for each modality, it can control the process so that different modalities do not converge to a completely identical distribution. Even if text embedding information and image embedding information correspond to the same semantic unit, if they are completely overlapped in a way that eliminates both the descriptive information of the text and the visual structural information of the image, subsequent search precision or explainability may be reduced. Through the separation constraints, the multimodal search device (100) can preserve the difference in characteristics for each modality to a certain level while maintaining semantic alignment.

[0076] In step S240, the multimodal search device (100) can control embedding information corresponding to the same content or the same semantic unit to be gathered near a single semantic unit in the common embedding space through aggregation constraints. For example, if there are multiple caption sentences corresponding to a single image, multiple frames or subtitle segments corresponding to a single video, or multiple text transcription variations corresponding to a single audio, the multimodal search device (100) can reflect a loss of aggregation constraints by lowering the average radius or pair distance value between the embedding information. Through such a structure, the problem of data expressing the same semantic unit from various perspectives being excessively dispersed in the common embedding space can be mitigated.

[0077] In step S240, the multimodal search device (100) can maintain distribution diversity through diversity constraints so that different samples can be sufficiently distinguished. For example, by using mean cosine similarity, covariance rank, spectral entropy, or uniformity indicators between different samples within the same modality, the phenomenon of embedding information being excessively concentrated in a specific narrow direction can be suppressed. By reflecting diversity constraints in this way, the multimodal search device (100) can more stably maintain the discriminability between samples within a common embedding space.

[0078] In step S250, the multimodal search device (100) can reflect subspace alignment for the embedding information of a plurality of different modalities. Specifically, the multimodal search device (100) can generate a reference subspace based on the embedding information of an anchor modality among a plurality of different modalities, project the embedding information of a target modality onto the reference subspace, calculate a subspace alignment loss that reflects the degree of alignment between the anchor modality and the target modality using the projection result on the reference subspace, and reflect the subspace alignment loss in learning. The subspace alignment loss can be calculated using the distance (e.g., L2 distance), angle difference (e.g., cosine distance), or axial error between the projection result of the target modality and the reference subspace of the anchor modality. In one embodiment, the subspace alignment loss may be redefined to minimize the relational geometric structure error between pairs of embeddings by calculating the difference between the Gram matrices of the projected target embeddings and the Gram matrices of the reference embeddings. Here, the anchor modality may refer to a modality used for setting the reference axis among a plurality of modalities, and the target modality may refer to a modality corrected to be aligned to the reference subspace.

[0079] In one embodiment, the anchor modality may include text or vision data of a High Resource Language (HRL) that is relatively rich in training data, and the target modality may include text data of a Low Resource Language (LLL) that is relatively prone to collapsing expressiveness.

[0080] In step S250, the multimodal search device (100) may perform singular value decomposition or principal component analysis on a matrix composed of embedding information of an anchor modality to generate the reference subspace, and select some of the upper components obtained as a result to set the reference subspace. For example, if an image modality is selected as the anchor modality, the upper principal component directions of the matrix composed of image embedding information may be set as the reference subspace. Due to the above structure, the multimodal search device (100) may not align the entire common embedding space equally, but may perform alignment of the target modality based on semantically important axes or representative direction structures.

[0081] In step S250, the multimodal search device (100) can use alignment reliability in the process of reflecting the subspace alignment loss in learning.

[0082] Specifically, the multimodal search device (100) can calculate alignment reliability based on the similarity between the embedding information of the anchor modality and the embedding information of the target modality. For example, based on the cosine similarity between reference embedding a and anchor modality embedding v, the gating weight g can be calculated as shown in [Equation 1] below.

[0083] [Mathematical Formula 1]

[0084]

[0085] Here, σ is the sigmoid function, t is the threshold value, and τ is the temperature parameter.

[0086] Subsequently, the multimodal search device (100) may calculate a subspace alignment loss only for embedding information where the alignment reliability (or gating weight) is greater than or equal to a reference value, or may calculate a subspace alignment loss by applying a weight corresponding to the alignment reliability.

[0087] For example, even if text and images correspond to the same content, some samples may have low description quality or contain misalignment, so the multimodal search device (100) may assign a greater alignment weight to samples with high reliability. In this case, the weight corresponding to the alignment reliability can be directly reflected when calculating the subspace alignment loss. With such a structure, the problem of subspace alignment learning becoming unstable due to samples with high noise can be reduced.

[0088] In step S260, the multimodal search device (100) can generate a multimodal embedding model by reflecting the alignment loss, the geometric distribution constraint loss, and the subspace alignment loss, and can generate embedding information of a search target and store the search target using the generated multimodal embedding model. At this time, the multimodal search device (100) can determine a multimodal embedding model that generates embedding information of multiple modalities in a common embedding space using a parameter set reflecting the alignment loss, the geometric distribution constraint loss, and the subspace alignment loss. In this step, the multimodal search device (100) inputs the search target data into the multimodal embedding model to generate embedding information of the common embedding space, and can store the embedding information and the original data of the search target or the search target identifier in the search target storage device (200). In this case, index information for improving search efficiency for the embedding information may be stored together in the search target storage device (200).

[0089] In step S260, the multimodal search device (100) may store embedding information for each modality separately according to the search target storage method, or may store embedding information for multiple modalities combined. For example, embedding information for images, text, and audio can be constructed as individual indices, and for a single search target, embedding information for multiple modalities can be stored as a single embedding information combined according to an average, weighted combination, or selection rule. With such a storage structure, various search strategies can be adopted depending on the form of the search query or the response speed requirements of the search system.

[0090] In step S270, the multimodal search device (100) can generate a search result based on the similarity between a search query and a search target using a learned multimodal embedding model. Specifically, the multimodal search device (100) receives a search query input from a user terminal (10), converts the search query into embedding information in a common embedding space, and then calculates the similarity with the embedding information of a search target stored in a search target storage device (200). Based on the similarity, the multimodal search device (100) can extract top candidate search targets and provide the search result to the user terminal (10). For example, if a text query is input, text embedding information can be generated and compared with image or video embedding information, and if an image query is input, image embedding information can be generated and compared with text or audio embedding information.

[0091] In step S270, the multimodal search device (100) may optionally perform reordering on the initial search candidates. For example, after extracting a primary group of candidates based on similarity in a common embedding space, a more precise matching evaluation model or a cross-encoder-based evaluation may be applied to the group of candidates to readjust their rankings. In this case, the primary search may serve to ensure a high response speed, and the reordering step may serve to improve semantic precision for the top group of candidates. Since the reordering step may be applied optionally, it may be omitted or enabled depending on the system resource status or service quality requirements.

[0092] In step S280, the multimodal search device (100) may perform selective correction based on the distribution state of embedding information generated after the step of learning the multimodal embedding model or during the search execution process. The distribution state may be calculated for the embedding information generated after the step of learning the multimodal embedding model. Specifically, the multimodal search device (100) may monitor the distribution indicator of the embedding information generated by the multimodal embedding model, detect distribution distortion or collapse of the embedding information in the common embedding space based on the distribution indicator, and then determine the projection layer as a target for correction. The distribution indicator may include mean similarity, variance, center distance between modalities, cluster radius, anisotropy indicator, rank indicator, or search performance indicator.

[0093] In step S280, if the distribution distortion or collapse is detected, the multimodal search device (100) can correct the multimodal embedding model by updating the projection layer while keeping the modality-specific encoder fixed. For example, the multimodal search device (100) can retrain or fine-tune only the projection layer responsible for mapping to the common embedding space while maintaining the parameters of the text encoder, image encoder, or audio encoder. In this case, the multimodal search device (100) can utilize the reference subspace or alignment confidence information used in step S250 to configure the correction of the projection layer so that it stably restores the semantic structure of the common embedding space. By this structure, it is possible to efficiently respond to the degradation of distribution stability without retraining the entire encoder.

[0094] In step S280, the multimodal search device (100) can verify the correction results by re-measuring the embedding information distribution or search performance after correction. For example, by comparing the search precision, recall, cluster radius, center distance between modalities, or anisotropy indicators before and after correction, it can be evaluated whether the projection layer correction actually contributes to distribution stabilization and search performance improvement. Based on the verification results, the multimodal search device (100) can reflect the corrected model as a service model or repeat the additional correction procedure.

[0095] According to the procedure of FIG. 2, one embodiment of the present invention converts data of a plurality of different modalities into embedding information of a common embedding space, and by reflecting geometric distribution constraints and subspace alignment along with alignment learning between the embedding information, semantic alignment and distribution stabilization can be achieved simultaneously. In addition, by including not only the process of storing search targets and performing searches, but also selective correction based on the distribution state of the embedding information, the accuracy, consistency, and operational stability of multimodal search can be improved.

[0097] FIG. 3 is a block diagram illustrating the detailed configuration of a multimodal search device according to one embodiment of the present invention.

[0098] The "geometric control" used in Fig. 3 corresponds to the application function of the geometric distribution constraint loss described in Fig. 2, and the "correction control" may correspond to a structure that selectively updates only the projection layer in the projector-only correction step.

[0099] Referring to FIG. 3, the multimodal search device (100) may include an embedding generation unit (110), an alignment learning execution unit (120), a geometry control unit (130), a search execution unit (140), and a correction control unit (150).

[0100] The embedding generation unit (110) can receive data of multiple different modalities and generate initial embedding information for mapping to a common embedding space. To this end, the embedding generation unit (110) may include a modality-specific data encoding function and a mapping function to a common embedding space. Specifically, the embedding generation unit (110) can generate feature information by applying a text encoder to text data, applying an image encoder to image data, and applying a corresponding modality-specific encoder to other modalities such as voice data or sensor data. Subsequently, the embedding generation unit (110) can generate modality-specific embedding information by projecting each feature information into a common embedding space of the same dimension.

[0101] In one embodiment, the embedding generation unit (110) can place a text embedding corresponding to the text sentence "industrial robot arm failure" and an image embedding generated by receiving an industrial robot arm failure image as input on the same embedding dimension. In this case, subsequent alignment learning can be performed so that they are placed in semantically similar positions despite having different input formats.

[0102] The alignment learning execution unit (120) can train a multimodal embedding model by performing alignment learning between embedding information generated by the embedding generation unit (110). To this end, the alignment learning execution unit (120) may include an alignment learning function between embedding information and a multimodal embedding model training function.

[0103] The alignment learning execution unit (120) can perform contrastive alignment by forming positive pairs for modality pairs that correspond semantically to each other and negative pairs for sample pairs that are semantically unrelated. In this process, the alignment loss can be calculated in a direction that decreases the distance between positive pairs and increases the distance between negative pairs.

[0104] In one embodiment, the alignment learning execution unit (120) may perform batch-unit learning by setting product description text and product images for the same product as positive pairs and combinations with other product images as negative pairs. Additionally, a loss structure based on cosine similarity, temperature scaling, and margin may be applied during the learning process.

[0105] The alignment learning execution unit (120) can update the parameters of the entire multimodal embedding model by reflecting not only the alignment loss but also the constraint loss result of the geometry control unit (130) described later. At this time, the multimodal search device (100) can determine a multimodal embedding model that generates embedding information of multiple modalities in a common embedding space using a parameter set in which the alignment loss, geometric distribution constraint loss, and subspace alignment loss are all reflected.

[0106] The geometric control unit (130) may reflect geometric distribution constraints and additionally perform subspace alignment to control the distribution state of the embedding information. To this end, the geometric control unit (130) may include a function for reflecting geometric distribution constraints and a function for reflecting subspace alignment.

[0107] The geometric control unit (130) can calculate a geometric distribution constraint loss that controls at least one of volume, separation, aggregation, and diversity for the distribution of the embedding information. For example, if the entire embedding shrinks excessively in a specific direction, a volume constraint can be applied to maintain the distribution dimension, and if the center points for each modality are excessively separated or overlap, a separation constraint or an aggregation constraint can be applied.

[0108] In one embodiment, the geometry control unit (130) can calculate the distribution volume using the Gram matrix or covariance matrix of the embedding matrix, and can calculate the volume constraint loss using at least one of the determinant value, the sum of singular values, or the rank value. For example, if only the upper singular value becomes excessively large and the lower singular value decreases rapidly in a specific learning interval, the geometry control unit (130) can detect the possibility of rank collapse and increase the constraint term to mitigate it.

[0109] Additionally, the geometry control unit (130) can control the distribution difference between modalities by using the center distance or covariance difference for each modality. For example, if the distance deviation between the text embedding center and the image embedding center increases above a reference value, the distribution alignment weight can be increased to induce re-alignment.

[0110] The subspace alignment reflection function may be a function that defines a subspace based on an anchor modality and aligns other modalities to the corresponding subspace. In one embodiment, after setting a text embedding as an anchor modality, a reference subspace may be generated by performing singular value decomposition or principal component analysis on a set of text embeddings.

[0111] Subsequently, the geometry control unit (130) can project an image embedding or other modality embedding onto the reference subspace and calculate a subspace alignment loss using the distance, angle difference, or axial error between the projection result and the reference subspace.

[0112] Additionally, when performing subspace alignment, alignment confidence can be calculated for each sample to apply different weights. For example, noisy image samples or incomplete descriptive sentences can be assigned low alignment confidence and given a low weighting when calculating subspace alignment loss.

[0113] The search execution unit (140) can manage search target embedding information using a trained multimodal embedding model and perform a similarity-based search for an input query. To this end, the search execution unit (140) may include a search target embedding information management function and a similarity-based search execution function.

[0114] The search execution unit (140) can generate and store embedding information for documents, images, video frames, or other data stored in the search target database in advance. Subsequently, when an input query is given, it can convert to the same common embedding space and then perform a nearest neighbor search.

[0115] In one embodiment, the search execution unit (140) can extract top candidates using cosine similarity, inner product, or Euclidean distance, and can improve large-scale search speed using an approximate nearest neighbor index.

[0116] The correction control unit (150) can monitor the distribution status of embedding information and perform projection layer correction when necessary. To this end, the correction control unit (150) may include an embedding information distribution status monitoring function and a projection layer correction function.

[0117] The correction control unit (150) can detect distribution drift between modalities by periodically analyzing the embedding distribution in operation after training is completed. For example, if the variance of a specific modality increases rapidly due to the influx of new domain data, only the projection layer can be updated without retraining the entire existing encoder.

[0118] At this time, the encoders for each modality remain in a fixed state, and only the projection layer can be selectively updated. The above distribution state can be calculated for the embedding information generated after the step of training the multimodal embedding model.

[0119] In one embodiment, the correction control unit (150) can perform new domain adaptation by gradient updating only the last projection matrix while maintaining the text encoder backbone and the image encoder backbone. Accordingly, the speed of new data adaptation can be increased while maintaining existing representation stability.

[0120] In addition, the structure according to Fig. 3 can simultaneously secure alignment stability, geometric distribution maintenance, subspace consistency, and search accuracy without retraining the entire encoder, so it has the effect of improving domain scalability while suppressing performance degradation of search systems operated for a long time in a multimodal environment.

[0122] FIG. 4 is a diagram illustrating the concept of embedding information distribution control and subspace alignment in a common embedding space according to an embodiment of the present invention.

[0123] FIG. 4 conceptually illustrates how the geometric distribution constraint reflection step and the subspace alignment reflection step can be implemented within a common embedding space. Text data (410), image data (420), and audio data (430) in FIG. 4 illustrate examples of data of multiple different modalities, and the common embedding space (440) may represent a representation space in which embedding information generated from the data of multiple different modalities can be compared under the same similarity criterion or distance criterion. Additionally, the subspace alignment (460) and reference subspace (461) in FIG. 4 may be understood as conceptually illustrating the reference axis structure used in the subspace alignment reflection step.

[0124] Referring to FIG. 4, text data (410), image data (420), and audio data (430) may be data of multiple different modalities input to a multimodal search device (100). The multimodal search device (100) may convert the text data (410), image data (420), and audio data (430) into feature information using a modality-specific encoder, and map the feature information to embedding information in a common embedding space (440) using a projection layer. At this time, the text data (410), image data (420), and audio data (430) may each have separate representation formats and statistical characteristics, but the multimodal search device (100) may be configured so that the data are converted into embedding information that can be compared with each other in the common embedding space (440).

[0125] In the common embedding space (440), embedding information corresponding to different modalities may be placed together. In this case, embedding information corresponding to semantically identical or similar content may be placed in close proximity to each other, and embedding information corresponding to semantically different content may be placed in spaced apart from each other. For example, text data (410) describing the same event, image data (420) capturing the event, and audio data (430) corresponding to a voice description of the event may be placed to form a semantic cluster within the common embedding space (440). On the other hand, data representing unrelated events or objects may be placed in a separate area separated from the cluster. Due to this structure, the multimodal search device (100) can treat semantically corresponding data as the same search candidate group even if the modalities are different.

[0126] The "cluster radius limit," "maintaining distance between classes," "suppressing collapse," and "maintaining distribution diversity" within the common embedding space (440) illustrated in FIG. 4 may represent the distribution control direction aimed at by the geometric distribution constraint of the present invention. In the operation corresponding to step S240, the multimodal search device (100) may calculate a geometric distribution constraint loss to control the distribution shape of embedding information in the common embedding space (440) and reflect this in learning. Specifically, the multimodal search device (100) may limit the cluster radius so that embedding information corresponding to the same semantic unit does not spread excessively, and may control the distance between embedding information corresponding to different semantic units or classes to be maintained at a certain level or higher. In addition, the multimodal search device (100) may suppress the collapse phenomenon in which embedding information is excessively concentrated in a specific few directions or specific regions, and may secure distribution diversity so that the overall distribution maintains sufficient directionality and dimensionality.

[0127] In one embodiment, the multimodal search device (100) may apply aggregation constraints to maintain the average radius or pair distance value between multiple caption embeddings, image embeddings, and audio embeddings corresponding to the same content within a reference range in order to implement the cluster radius limit. Additionally, the multimodal search device (100) may apply separation constraints that reflect the difference in distance, angle, or covariance between center vectors by modality or class to implement class-to-class distance maintenance. To suppress collapse and maintain distribution diversity, the multimodal search device (100) may measure distribution volume and diversity using determinant values, sum of singular values, rank values, mean cosine similarity, spectral entropy, or uniformity indicators, and set constraint losses to mitigate excessive contraction or bias in a specific direction. By such a structure, the common embedding space (440) can be formed not merely as a space that brings corresponding data pairs closer together, but as a structured representation space in which search accuracy and distribution stability are secured.

[0128] The partial space alignment (460) illustrated on the right side of FIG. 4 is intended to explain the concept of partial space alignment, which is distinct from uniform alignment over the entire common embedding space (440). In the operation corresponding to step S250, the multimodal search device (100) may set a reference partial space (461) based on the embedding information of an anchor modality among a plurality of different modalities, and project the embedding information of a target modality onto the reference partial space (461) to calculate the degree of alignment. The reference partial space (461) illustrated in FIG. 4 may conceptually represent a representative directional structure or principal component axis structure formed by the embedding information of the anchor modality. In this case, the expression "partial space alignment (460)" may be understood as having the same technical concept as the partial space alignment used in the claim and FIG. 2, and "reference partial space (461)" may specifically illustrate a partial space that serves as a reference for the partial space alignment.

[0129] The "target embedding projection direction" illustrated in FIG. 4 may indicate the direction in which the embedding information of the target modality is projected so as to be aligned with the reference subspace (461). For example, when image data (420) is set as the anchor modality, the multimodal search device (100) may set the reference subspace (461) by performing singular value decomposition or principal component analysis on the set of image embedding information, and may project the target embedding information generated from text data (410) or audio data (430) onto the reference subspace (461). Using the projection result, the multimodal search device (100) may calculate the distance, angle difference, or axial error between the target embedding information and the reference subspace (461) and reflect this in the calculation of the subspace alignment loss.

[0130] In one embodiment, text data (410) and image data (420) may appear to be in close proximity throughout the entire common embedding space (440), but alignment errors may still exist in some semantically important axis directions. In this case, the multimodal search device (100) can correct subtle semantic axis discrepancies that are difficult to identify based on the total distance alone by separately evaluating the alignment state on the reference subspace (461). For example, even if a text description and an image of the same product are sufficiently similar based on the total cosine similarity criterion, deviations may exist in specific semantic axes related to changes in the product's state, defect types, or location information. The multimodal search device (100) can reinforce the consistency in the specific semantic axes through subspace alignment. Due to this structure, more precise semantic correspondence can be achieved compared to simple alignment over the entire common embedding space.

[0131] Additionally, the multimodal search device (100) may also consider alignment reliability during the subspace alignment process. For example, if the embedding information of the target modality already shows high similarity with the reference subspace (461), a larger weight may be applied to calculate the alignment loss, and a lower weight may be applied to samples with low explanation quality or high noise. When subspace alignment based on alignment reliability is performed in this way, the problem of the semantic structure of the reference subspace (461) being distorted by incorrect samples or low-quality samples can be reduced.

[0132] The structure according to FIG. 4 can be distinguished from conventional simple pairwise alignment-based multimodal search structures in that geometric distribution constraints and subspace alignment are combined. In conventional technology, the focus may be on placing data with the same meaning close together in a common space, but in this case, the entire distribution may collapse in a specific direction, the unique structure of some modalities may be excessively lost, or alignment with respect to semantically important axes may not be sufficiently secured. On the other hand, the multimodal search device (100) according to one embodiment of the present invention controls the cluster radius, distance between classes, distribution collapse, and diversity together in a common embedding space (440), and additionally reflects subspace alignment using a reference subspace (461), thereby securing semantic alignment and distribution stabilization simultaneously.

[0133] According to the structure of FIG. 4, one embodiment of the present invention can form a common embedding space (440) not as a simple projection result space, but as a structured representation space in which geometric distribution constraints and subspace alignment are reflected together. Accordingly, semantic correspondence precision between multiple different modalities, consistency of search results, and long-term operational stability of embedding information can be improved.

[0135] FIG. 5 is a flowchart illustrating a geometric distribution constraint and subspace alignment-based multimodal embedding model learning procedure according to one embodiment of the present invention.

[0136] The procedure according to FIG. 5 may be a procedure for training a multimodal embedding model by configuring learning settings for data of multiple different modalities, generating embedding information based on a mini-batch, calculating alignment loss, geometric distribution constraint loss, and subspace alignment loss to combine the total loss, and optionally updating only the projection layer or updating all parameters. In this process, subspace alignment using a reference subspace and weighting based on alignment confidence may be performed together to achieve both stabilization of the distribution of embedding information in a common embedding space and precise alignment between modalities.

[0137] The expression "projector" used in FIG. 5 can be understood as a concept corresponding to the "projection layer" described earlier. Additionally, the "anchor subspace" in FIG. 5 may be an expression corresponding to the "reference subspace" described earlier, and "gating" may correspond to a structure that controls the reflection intensity of subspace alignment loss using weights corresponding to alignment reliability. Therefore, the expressions used in the conception of FIG. 5 can be understood as representing technical concepts that are functionally identical to the terms used in the claims and prior art.

[0138] Referring to FIG. 5, in step S501, the multimodal search device (100) can configure learning settings. Specifically, the multimodal search device (100) can set the type of alignment loss, the configuration of geometric distribution constraint loss, whether to reflect subspace alignment loss, whether to activate a correction mode that updates only the projection layer, the learning rate, temperature parameters, a threshold related to alignment reliability, loss term weights, and the number of learning iterations. For example, the contrast learning loss and the matching loss can be used together as alignment loss, and multiple loss terms related to volume, separation, aggregation, and diversity can be combined as geometric distribution constraint loss. The learning settings can be set based on a predefined policy provided by the model management device (300), and may be adjusted according to previous learning history or distribution indicators collected during operation.

[0139] In step S502, the multimodal search device (100) may sample a mini-batch. In this step, the multimodal search device (100) may configure the mini-batch to include at least two different modalities of data corresponding to the same content. For example, if images, text, and audio correspond to the same event or the same object, the data may be grouped into a single semantic unit and included in the mini-batch. In this case, if there are multiple descriptions of the same content, such as multiple captions or multiple explanatory sentences, they may be included together to more reliably learn the aggregation characteristics within the same semantic unit. The mini-batch may be configured to simultaneously include positive samples and negative samples used for alignment loss calculation.

[0140] In step S503, the multimodal search device (100) can encode data of each modality included in the minibatch. Specifically, the multimodal search device (100) can generate feature information corresponding to each input by applying a text encoder to text data, an image encoder to image data, and an audio encoder to audio data. For example, a text token sequence can be converted into a contextual representation vector, image patch or frame information can be converted into a visual feature vector, and audio waveform or spectrum information can be converted into an acoustic feature vector. In this process, the modality-specific encoder may be updated together according to the learning mode, or may remain in a fixed state in the correction mode described later.

[0141] In step S504, the multimodal search device (100) can project the feature information into a common embedding space. Specifically, the multimodal search device (100) can input the feature information generated by the modality-specific encoder into a projection layer to map it into embedding information in the common embedding space. The projection layer may include a single linear layer, multiple linear layers, a normalization layer, or a non-linear transformation layer, and can transform feature information of different modalities so that it can be compared under the same dimensional structure and similarity criteria. For example, even if text feature vectors and image feature vectors have different raw statistical characteristics, they can be placed in a common embedding space of the same dimension through the projection layer to enable mutual similarity calculation.

[0142] In step S505, the multimodal search device (100) can calculate an alignment loss. Specifically, the multimodal search device (100) can calculate an alignment loss that increases the similarity between embedding information of different modalities corresponding to the same content and decreases the similarity between embedding information corresponding to different content. The alignment loss can be implemented as a contrast learning loss, a matching loss, or a combination thereof. For example, embedding information of the same image and descriptive sentence can be learned to be close to each other in a common embedding space, and embedding information of unrelated descriptive sentences and images can be learned to be far apart from each other. Although the alignment loss forms the basis of multimodal semantic correspondence, it does not directly control distribution collapse, excessive mixing, or specific axis bias, so it can be used in combination with geometric distribution constraints and subspace alignment in subsequent steps.

[0143] In step S506, the multimodal search device (100) can calculate a geometric distribution constraint loss. Specifically, the multimodal search device (100) can calculate a geometric distribution constraint loss that controls at least one of volume, separation, aggregation, and diversity for the distribution of said embedding information in order to control the distribution shape of said embedding information in a common embedding space. For example, regarding volume constraints, the multimodal search device (100) can evaluate the distribution degrees of freedom of a specific modality or the entire batch using at least one of the Gram matrix, covariance matrix, determinant value, sum of singular values, or rank value of the embedding matrix, and suppress the phenomenon of the embedding information shrinking in an excessively low-dimensional direction. Regarding separation constraints, the multimodal search device (100) can mitigate excessive overlap between modalities using the distance, angle, or covariance difference between center vectors for each modality. Regarding cohesion constraints, mean radius or pair distance-based constraints may be applied to ensure that multiple narrative views, frame views, or variant views corresponding to the same content are clustered in a single semantic vicinity. Regarding diversity constraints, distribution diversity may be maintained using mean cosine similarity, spectral entropy, or uniformity indicators so that different samples are sufficiently distinguishable.

[0144] In step S507, the multimodal search device (100) can estimate an anchor subspace. The anchor subspace may correspond to the reference subspace described above and may be established based on the embedding information of the anchor modality among a plurality of different modalities. Specifically, the multimodal search device (100) may perform singular value decomposition or principal component analysis on a matrix composed of the embedding information of the anchor modality, and select some of the upper components obtained as a result to form the reference subspace. For example, when an image modality is set as the anchor modality, principal components reflecting the major dispersion direction of the image embedding information set may be set as the reference subspace. Through such a structure, a basis can be formed to perform alignment based on semantically important axis structures rather than uniform alignment over the entire common embedding space.

[0145] In step S508, the multimodal search device (100) can perform subspace projection, alignment, and gating. Specifically, the multimodal search device (100) can project the embedding information of the target modality onto the reference subspace estimated in step S507 and calculate a subspace alignment loss that reflects the degree of alignment between the anchor modality and the target modality using the projection result. The subspace alignment loss can be calculated using the distance, angle difference, or axial error between the projection result of the target modality and the reference subspace. Additionally, the multimodal search device (100) can calculate alignment reliability based on the similarity between the embedding information of the anchor modality and the embedding information of the target modality, and apply the subspace alignment loss only to embedding information where the alignment reliability is greater than or equal to a reference value, or directly reflect a weight corresponding to the alignment reliability when calculating the subspace alignment loss. For example, text with high descriptive quality and images with clear visual correspondences can have high alignment reliability, and larger alignment weights can be assigned to these samples. Conversely, distortion of the reference subspace can be reduced by applying lower weights to samples that are noisy or have a high probability of misalignment.

[0146] In step S509, the multimodal search device (100) can combine the total loss. Specifically, the multimodal search device (100) can form the total loss by combining the alignment loss calculated in step S505, the geometric distribution constraint loss calculated in step S506, and the subspace alignment loss calculated in step S508. At this time, different weights may be assigned to each loss term, and said weights may be set as fixed values ​​or may be dynamically adjusted in the manner described in steps S514 and S515 described later. For example, if the likelihood of distribution collapse is evaluated as high, the weight of the volume or diversity-related loss term may be increased, and if the alignment error between modalities is large, the reflection ratio of the subspace alignment loss may be increased.

[0147] In step S510, the multimodal search device (100) can determine whether to use a correction mode. The correction mode may be a mode in which only the projection layer is selectively updated while the modality-specific encoder is fixed, in order to mitigate distribution distortion or collapse in the common embedding space. For example, if the existing encoder already possesses sufficient semantic representation capabilities but a specific modality converges excessively in a narrow direction in the projection layer, the multimodal search device (100) can activate the correction mode to correct only the projection layer without retraining the entire encoder.

[0148] In step S511, if the judgment result of step S510 is in correction mode, the multimodal search device (100) can update only the projector. Here, "projector" may have the same technical meaning as the projection layer described above. Specifically, the multimodal search device (100) may fix the parameters of the text encoder, image encoder, or audio encoder and set only the projection layer responsible for mapping to the common embedding space as the target for update. In this case, the multimodal search device (100) can correct distribution distortion in the common embedding space at a lower cost while maintaining the existing semantic representation structure as much as possible. For example, if new domain data is introduced and the center distance relationship between text embeddings and image embeddings changes, the alignment relationship can be restored by adjusting only the projection matrix without retraining the entire encoder.

[0149] In step S512, if the judgment result of step S510 is not a correction mode, the multimodal search device (100) can update all parameters. Specifically, the multimodal search device (100) can perform learning on the entire set of parameters, including encoders and projection layers for each modality. In this case, if it is an initial learning stage or if a large-scale distribution change occurs and modifying only the projection layer is insufficient, the entire parameter update mode may be adopted. For example, if a new modality is added or a new service domain significantly different from the existing data distribution is applied, the multimodal search device (100) can update the encoder and projection layer together to form a new, more fundamental alignment structure.

[0150] In step S513, the multimodal search device (100) can perform backpropagation and updates. Specifically, the multimodal search device (100) can calculate a gradient based on the total loss formed in step S509 and perform backpropagation on the parameters to be updated determined in step S511 or step S512. At this time, in correction mode, the gradient may be applied only to the projection layer, and in full parameter update mode, the gradient may be applied to the entire encoder and projection layer. For example, if the alignment loss is good for the same mini-batch but the diversity-related indicator is degraded, a larger update amount may be applied in the direction of a specific parameter according to the corresponding loss contribution.

[0151] In step S514, the multimodal search device (100) may determine whether to update dynamic weights or thresholds. This step may be a step for determining whether to adjust the reflection strength or alignment confidence threshold value based on the current learning state or distribution state, instead of using it as a fixed value for each loss term. For example, the policy may be changed such that the weight of alignment loss is increased during the early stages of learning, and the weight of geometric distribution constraint loss or subspace alignment loss is increased during the later stages of learning. Alternatively, the sample range to which subspace alignment is applied may be adjusted by relaxing or strengthening the alignment confidence threshold value.

[0152] In step S515, if dynamic updating is required based on the judgment result of step S514, the multimodal search device (100) may estimate geometric indicators. The geometric indicators may be indicators that numerically represent the distribution state of embedding information, and may include, for example, average similarity, center distance between modalities, cluster radius, anisotropy indicator, rank value, spectral characteristics, degree of separation, or degree of aggregation. The multimodal search device (100) may use the geometric indicators to determine whether the current distribution is in an overmixed state, an over-separated state, a state where only specific modalities are collapsing, or a state where diversity is lacking overall. In this case, the multimodal search device (100) may adjust the weights of loss terms related to volume, separation, aggregation, diversity, or subspace alignment based on the judgment result. For example, if specific modality embeddings are excessively concentrated in a narrow cone shape, the volume or diversity constraint weights may be increased, and if the center distance deviation between modalities becomes excessively large, the reflection ratio related to separation or subspace alignment may be adjusted.

[0153] In step S516, the multimodal search device (100) can record checkpoints and logs. Specifically, the multimodal search device (100) can store the current number of training iterations, total loss value, detailed value per loss term, alignment confidence distribution, geometric indicators, search performance indicators, parameter update status, or whether correction mode is applied as logs. Additionally, the model parameters at the current point in time can be stored as checkpoints and used for subsequent evaluation, retraining, or model comparison. For example, by recording the recall value, anisotropy indicator, and change in center distance between modalities at a specific epoch together, it is possible to track whether alignment accuracy and distribution stabilization are simultaneously improved.

[0154] In step S517, the multimodal search device (100) can determine whether to terminate training. This determination may be made based on whether the target number of iterations is reached, whether the total loss converges, whether the improvement in search performance has stalled, or whether the distribution indicators have stabilized. For example, if the improvement in search precision for the validation dataset is minimal and the cluster radius and distance between classes remain stable within a reference range, the multimodal search device (100) may determine that training has ended. Conversely, if the conditions for training termination are not met, the process may be repeated by returning to step S502 and sampling a new mini-batch. Through such an iterative structure, a multimodal embedding model can be formed that integrally reflects alignment loss, geometric distribution constraint loss, and subspace alignment loss.

[0155] The learning procedure according to FIG. 5 can be distinguished from a conventional learning structure that simply adjusts the distance between positive pairs and negative pairs. In a conventional structure, by focusing on optimizing relative similarity across the entire common embedding space, it may be difficult to directly control distribution collapse, excessive modality mixing, or failure of alignment of specific semantic axes. On the other hand, a multimodal search device (100) according to an embodiment of the present invention directly controls the geometric distribution shape of the common embedding space in step S506, performs precise alignment using a reference subspace in steps S507 and S508, and selectively corrects only the projection layer in steps S510 to S513. With such a structure, alignment accuracy, distribution stability, and operational efficiency can be improved together.

[0156] According to the procedure of FIG. 5, one embodiment of the present invention maintains semantic alignment in a common embedding space while reflecting geometric distribution constraints regarding volume, separation, aggregation, and diversity, performs subspace alignment using a reference subspace and weighted control based on alignment reliability, and can selectively correct only the projection layer if necessary. Accordingly, the learning stability, search precision, and operational adaptability of a multimodal embedding model for multiple different modalities can be comprehensively improved.

[0159] FIG. 6 is a flowchart or functional flowchart illustrating a detailed processing procedure for geometric distribution constraints and subspace alignment according to an embodiment of the present invention.

[0160] FIG. 6 more specifically illustrates how the geometric distribution constraint loss calculation step and the subspace alignment step among the learning procedures described in FIG. 5 can be internally composed of detailed operations. In particular, the "batch common embedding" used in FIG. 6 may be a concept corresponding to the embedding information of the common embedding space described in FIG. 2 to FIG. 5, and the "anchor modal reference subspace" may be a concept corresponding to the anchor subspace or reference subspace described earlier. Therefore, the expressions used in the conception theory of FIG. 6 can be understood as representing technical concepts that are functionally identical to the terms used in the existing drawings and claims.

[0161] The procedure according to FIG. 6 may be a procedure for calculating modality-specific or sample-specific statistics from batch common embeddings, calculating geometric distribution constraint losses corresponding to volume constraints, separation constraints, aggregation constraints, and diversity constraints based on said statistics, calculating subspace alignment losses using the projection results of target modalities on reference subspaces set based on anchor modalities, and then aggregating said losses to output final constraint losses and diagnostic indicators. By this structure, one embodiment of the present invention can precisely control the distribution of embedding information in the common embedding space by including not only simple pairwise alignment levels but also geometric structures at the batch level and alignment structures at the subspace level.

[0162] Referring to FIG. 6, in step S610, the multimodal search device (100) can calculate statistics for batch common embeddings. Specifically, the multimodal search device (100) can calculate a center vector, a covariance matrix, a Gram matrix, spectral characteristics, low-rank span information, or an inter-sample distance distribution from a set of embedding information corresponding to each modality or each semantic group within the batch. For example, a modal center vector can be calculated for a set of embedding information corresponding to an image modality, a covariance structure can be estimated for a set of embedding information corresponding to a text modality, and an inter-sample mean distance and direction distribution can be measured for a set of embedding information corresponding to an audio modality. These statistics can be used as basic information for calculating geometric distribution constraints regarding volume, separation, aggregation, and diversity, respectively, in subsequent steps.

[0163] In step S610, the multimodal search device (100) can calculate statistics by dividing the batch common embeddings not only by modality but also by unit of sample set corresponding to the same content or by unit of the same semantic group. For example, when a single image and multiple text captions and multiple audio descriptions corresponding thereto form a single semantic group, the multimodal search device (100) can calculate the distribution of embedding information within the group and the distribution between groups by distinguishing them. Due to this structure, the cohesion constraint and separation constraint described later can be configured to reflect not only the distance between simple modalities but also the cohesion within the same semantic group and the dispersion between groups.

[0164] In step S621, the multimodal search device (100) can calculate a volume constraint loss. Specifically, the multimodal search device (100) can measure the distribution degrees of freedom or spatial occupancy of embedding information using at least one of a Gram matrix, a covariance matrix, a determinant value, a sum of singular values, or a rank value for a batch common embedding or a set of specific modality embeddings. If the measurement result indicates a direction of excessive contraction into a specific low-dimensional structure, the multimodal search device (100) can set the volume constraint loss in a direction that suppresses said contraction. For example, if a large number of text embedding information are clustered in a narrow conical area within the common embedding space, the multimodal search device (100) can determine a lack of distribution volume based on a decrease in the determinant value or rank value and increase the volume constraint loss to expand the distribution degrees of freedom.

[0165] In step S621, the multimodal search device (100) can be controlled to preserve distribution degrees of freedom within a range that maintains semantic dispersion through volume constraint loss. For example, since search accuracy may be reduced if samples corresponding to the same meaning are distributed independently of each other, the multimodal search device (100) can be configured to maintain distribution volume only within a range that does not compromise semantic consistency by combining volume constraints with cohesion constraints. With such a structure, a common embedding space can be formed in which search precision is maintained while suppressing representation collapse.

[0166] In step S620, the multimodal search device (100) can calculate a separation constraint loss. Specifically, the multimodal search device (100) can measure the degree of separation between different modalities or different semantic groups using the distance between center vectors for each modality, the angle, the difference in covariance, the difference in the principal angle or variance direction between low-rank spans. For example, if the center of a text modality and the center of an image modality are too close together, weakening mutual distinction, the multimodal search device (100) determines that the degree of separation between the modalities is insufficient and can reflect a separation constraint loss in a direction that maintains the boundary between the modalities. Conversely, if the center distance or principal angle between modalities becomes excessively large, weakening mutual alignment, the loss reflection ratio can be adjusted to prevent excessive separation.

[0167] In step S620, the multimodal search device (100) can avoid both "complete overlap between modalities" and "excessive separation between modalities" through separation constraints. For example, if text embedding information and image embedding information describing the same object are completely mixed into one block, the unique representation axis of each modality may be lost, and conversely, if the two are excessively separated, the cross-modal search precision may be reduced. The multimodal search device (100) can form a balanced state in which mutual alignment is maintained while distinguishability is preserved by utilizing the above separation constraint loss.

[0168] In step S620, the multimodal search device (100) can calculate a cohesion constraint loss. Specifically, the multimodal search device (100) measures the average radius, average pair distance, or deviation from the reference center for multiple modality embedding information corresponding to the same content or multiple view embedding information belonging to the same semantic group, and can apply a cohesion constraint loss if the deviation becomes excessively large. For example, if there are keyframe images, subtitle text, background audio descriptions, and summary sentences corresponding to a single video content, the multimodal search device (100) can apply a cohesion constraint loss so that the embedding information gathers near a single semantic in a common embedding space. Through such a structure, the phenomenon in which various expressions for the same semantic are dispersed far apart from each other can be mitigated.

[0169] In step S620, the multimodal search device (100) may reflect not only the simple average distance but also quality differences by modality or reliability differences by view when calculating the cohesion constraint loss. For example, if one of three captions corresponding to the same image contains noise, the multimodal search device (100) may set the radius contribution of the corresponding caption embedding information low or assign a different weight during group-centered calculation. With such a structure, the cohesion constraint can group identical semantic groups into a single semantic vicinity while reducing distortion caused by low-quality views.

[0170] In step S631, the multimodal search device (100) can calculate a diversity constraint loss. Specifically, the multimodal search device (100) can evaluate the diversity of the entire distribution using the mean cosine similarity, spectral entropy, uniformity index, covariance rank, or variance direction distribution among different samples within the same modality. As a result of the evaluation, if different samples are excessively concentrated in a specific direction within a common embedding space, the multimodal search device (100) can increase the diversity constraint loss to alleviate the concentration and ensure discrimination between samples. For example, if different images form nearly identical embedding directions or different text queries converge excessively into similar vectors, the multimodal search device (100) can determine a lack of diversity based on an increase in mean cosine similarity or a decrease in spectral entropy.

[0171] In step S631, the multimodal search device (100) can maintain distribution diversity in a direction that contributes to improving search discriminability rather than simply expanding variance through diversity constraint loss. For example, if only the distance between classes is maintained and the internal distribution within classes becomes excessively monotonous, it may be difficult to distinguish similar but different search queries; therefore, the multimodal search device (100) can reflect diversity constraint loss in a direction where different samples are sufficiently distinguishable. With such a structure, the fine-grained ranking quality of the search results can be improved.

[0172] In step S622, the multimodal search device (100) can perform subspace alignment and gating. Specifically, the multimodal search device (100) can set an anchor modal reference subspace by performing singular value decomposition or principal component analysis on a set of embedding information of an anchor modality, and project the embedding information of a target modality onto the reference subspace. Subsequently, the multimodal search device (100) can calculate a subspace alignment loss using the distance, angle difference, or axial error between the projection result and the projection result of the reference subspace or the anchor modality. For example, when an image modality is set as the anchor modality, a reference subspace can be formed in the major dispersion direction of the set of image embedding information, and text or audio embedding information can be projected onto the reference subspace to separately measure the alignment state for semantically important axes.

[0173] In step S622, the multimodal search device (100) can use gating to select the target for reflecting subspace alignment loss or adjust the reflection intensity. Specifically, the multimodal search device (100) can calculate alignment reliability based on the similarity between the embedding information of the anchor modality and the embedding information of the target modality, the projection fit for the reference subspace, or reliability per modality. If the alignment reliability is greater than or equal to a reference value, the subspace alignment loss can be actively reflected for the sample, and if the alignment reliability is low, the reflection ratio of the loss can be lowered or its application omitted. For example, a high gating value can be applied when an image-text pair accurately describes the same object, and a low gating value can be applied when the text description is ambiguous or the audio contains noise. Through such a structure, the problem of the semantic structure of the reference subspace being damaged by noisy samples can be reduced.

[0174] In step S640, the multimodal search device (100) can aggregate constraint losses. Specifically, the multimodal search device (100) can form a geometric distribution constraint loss or a total constraint loss by weightedly combining the volume constraint loss calculated in step S602, the separation constraint loss calculated in step S603, the aggregation constraint loss calculated in step S604, the diversity constraint loss calculated in step S605, and the subspace alignment loss calculated in step S606. At this time, the weights assigned to each loss term may be fixed values ​​or may be dynamically adjusted according to the geometric indicators of the current batch or the learning progress status. For example, if the likelihood of collapse in a particular batch is evaluated as high, the reflection ratio of the volume or diversity-related loss term may be increased, and if the alignment error between modalities is evaluated as high, the reflection ratio of the subspace alignment loss may be increased.

[0175] In step S640, the multimodal search device (100) may combine the constraint loss aggregation result with the alignment loss to form a final learning loss. For example, in the total loss combining step S509 described in FIG. 5, the alignment loss and the constraint loss formed in this step may be used together. With such a structure, one embodiment of the present invention can simultaneously optimize not only simple pairwise alignment accuracy but also batch-level distribution structure, cohesion of same semantic groups, diversity of different samples, and alignment precision on a reference subspace.

[0176] In step S640, the multimodal search device (100) may generate output information. Specifically, the multimodal search device (100) may output a geometric distribution constraint loss, a subspace alignment loss, a constraint loss aggregation result, a volume-related diagnostic indicator, a separation-related diagnostic indicator, an aggregation-related diagnostic indicator, a diversity-related diagnostic indicator, or an alignment confidence distribution. The output information may be provided to the total loss combination step of FIG. 5, the geometric indicator estimation step of FIG. 5, or the distribution monitoring and correction step during operation of FIG. 9. For example, if the distribution diversity indicator of a specific modality is consistently kept low, the multimodal search device (100) may use the information to increase the weight of the diversity constraint loss in subsequent learning or determine the target for projection layer correction during operation.

[0177] The structure according to FIG. 6 can be distinguished from conventional methods that form a common embedding space with only alignment loss. In conventional methods, even if it is successful to place pairs of data with the same meaning close together, the overall distribution may collapse in a specific low-dimensional direction, excessive mixing between modalities may occur, multiple views of the same meaning may be excessively dispersed, or alignment inconsistencies may remain in semantically important axis directions. On the other hand, a multimodal search device (100) according to an embodiment of the present invention can independently calculate placement-level statistics, volume control, separation control, aggregation control, diversity maintenance, and subspace alignment through steps S601 to S608, and combine them to control the geometric structure of the common embedding space itself. With such a structure, not only can search accuracy be improved, but distribution stability, domain change responsiveness, and search consistency during long-term operation can also be improved.

[0178] According to the processing structure of FIG. 6 as described above, one embodiment of the present invention treats the distribution of embedding information in a common embedding space not as a simple alignment result but as a geometric structure that is a direct control target, and can reflect a plurality of geometric distribution constraints including volume, separation, aggregation, and diversity, along with subspace alignment for an anchor modal reference subspace. Accordingly, semantic alignment and distribution stabilization can be achieved simultaneously, and the precision, consistency, and representation stability of multimodal search for a plurality of different modalities can be improved.

[0180] FIG. 7 is a flowchart illustrating the offline embedding generation and ANN indexing procedure for a search target item according to one embodiment of the present invention.

[0181] The procedure according to FIG. 7 may be a procedure that collects multiple multimodal items to be searched, performs preprocessing, encoding, and projection for each modality to generate vectors in a common embedding space, stores the generated embeddings by modality or by item according to a storage policy, builds or updates an ANN index for approximate nearest neighbor search, and stores distribution indicators together as needed, thereby enabling fast candidate search and distribution stability monitoring during subsequent online search.

[0182] Referring to FIG. 7, in step S710, the multimodal search device (100) can collect search target items. Here, the search target items may include images, videos, audio, documents, subtitles, descriptive text, metadata, or a combination thereof. For example, for a single video item, a representative frame, audio segment, subtitle text, and title information may be collected together, and additional metadata such as language information, domain information, quality rating, creation time, or category may be associated with each item. This metadata may subsequently be used as index splitting or search range limiting conditions.

[0183] In step S720, the multimodal search device (100) can perform modality-specific preprocessing on the collected search target items. For example, for video data, representative frames or clip-unit segments can be extracted, for audio data, resampling, silence removal, or segment splitting can be performed, and for text data, normalization, token alignment, language determination, or sentence splitting can be performed.

[0184] In one embodiment, when an item includes multiple captions or multiple language descriptions, the multimodal search device (100) may maintain each description sentence as a separate segment to enable subsequent generation of multiple embeddings.

[0185] In step S730, the multimodal search device (100) can perform encoding and projection on each preprocessed modality data. Specifically, text data can be converted into text feature vectors through a text encoder, image data can be converted into image feature vectors through a vision encoder, and audio data can be converted into acoustic feature vectors through an audio encoder. Subsequently, each feature vector can be mapped to a common embedding space through a projection layer. The embeddings generated at this time can be normalized; for example, by applying L2 normalization to normalize the vector length, a structure suitable for cosine similarity-based search can be formed. Additionally, in one embodiment, the multimodal search device (100) can perform a quality check immediately after generating the embeddings to detect vectors where the norm value deviates from a reference range or is excessively skewed in a specific direction, and if the quality criteria are not satisfied, the corresponding vector can be regenerated or excluded.

[0186] In step S735, the multimodal search device (100) can select a storage method for the generated embeddings. This step can be branched according to search accuracy, storage capacity, search speed, and operation policy.

[0187] In step S740, if the first method is selected as a result of selecting the storage method, the multimodal search device (100) can store embeddings individually by modality or by view. For example, for a single item, text embeddings, vision embeddings, and audio embeddings can be stored as separate index entries, and if multiple captions exist, multiple vectors can be maintained for each caption. Such a structure can have the advantage of enabling more precise searching for modalities identical to the query modality and allowing only specific modalities to be selectively updated. For example, if a descriptive sentence is added to the same image, only the text embedding can be updated while maintaining the existing image embedding.

[0188] In step S741, if the second method is selected as a result of selecting the storage method, the multimodal search device (100) can generate and store an item combined embedding. For example, text embeddings and image embeddings can be weighted averaged or fused into a single representative vector through an attention-based combining operation. In this case, since one item is stored as a single vector, the index structure can be simplified and the search speed can be improved.

[0189] In one embodiment, different weights may be applied according to the reliability or quality score for each modality, and the audio embedding reflection ratio may be lowered when low-quality audio is included.

[0190] In step S750, the multimodal search device (100) can construct an ANN index or update an existing index using stored embeddings. Here, the ANN index may be an approximate nearest search structure, and may use, for example, an IVF-based structure, a graph-based structure, or a hierarchical search structure.

[0191] In one embodiment, when the data scale increases, the index can be constructed by separating it into shard units, and rebuilding units can be distinguished by category or language. Additionally, when new data is added, incremental updates can be performed without a full rebuild, and in this case, the state of the previous version of the index can be saved as a rollback point.

[0192] In step S760, the multimodal search device (100) may optionally store distribution indicators. Here, the distribution indicators may include centroids for each modality, covariance, variance magnitude, anisotropy-related indicators, principal component directions, or class-specific aggregation. These distribution indicators may be used as criteria to determine whether there is an abnormality in embedding quality during the online search step or to determine the need for projector-only correction. For example, if the centroid of specific language data consistently deviates from the reference position, the subsequent correction control unit may control the performance of projector re-correction for that language.

[0193] According to the procedure of Fig. 7, the data to be searched is structured and stored based on a common embedding space prior to an online query, and since the storage method itself can be selectively configured according to modality characteristics and operational purposes, it is possible to reduce the computational load during searching while simultaneously ensuring search consistency between modalities and operational flexibility. In addition, by maintaining the ANN index and distribution indicator together, the degradation of search accuracy or the collapse of the embedding distribution can be stably controlled even in a long-term operating environment.

[0195] FIG. 8 is a flowchart illustrating an online multimodal search execution procedure according to an embodiment of the present invention.

[0196] The procedure according to FIG. 8 may be a procedure for continuously managing search quality and system status by receiving a query input from a user, generating a query embedding, extracting candidate search results using a pre-built ANN index, performing precise reordering as needed or determining the final result using only the embedding score, and then recording the result return and operation log.

[0197] Referring to FIG. 8, at step S810, the multimodal search device (100) may receive a search query from a user. Here, the query may be a text-based sentence and may optionally include an image or audio. For example, if the user enters a query as a natural language sentence, the multimodal search device (100) may normalize internal search conditions by analyzing the language, domain, length, or included named entities of the sentence.

[0198] In one embodiment, when multiple language expressions having the same meaning are input, the query language can be determined and preprocessing rules suitable for the characteristics of the corresponding language can be applied, and excessively long queries can be compressed around core tokens.

[0199] In step S820, the multimodal search device (100) can generate a query embedding corresponding to the received query. Specifically, the query text can be converted into a feature vector by passing through a query encoder and then mapped to a common embedding space through a projection layer. At this time, the same projector structure can be used to maintain the same coordinate system as the common embedding space used during training, and a corrected projector version can be applied as needed. For example, if a recent distribution shift is detected in a specific language or a specific domain, the multimodal search device (100) can reduce the positional bias of the query embedding by applying updated projector parameters in the correction control unit.

[0200] In step S830, the multimodal search device (100) can perform an ANN-based candidate search using the generated query embedding. Here, the ANN search may be a process of rapidly extracting Top-N candidates from a pre-built approximate nearest search index. For example, top candidate vectors can be searched based on cosine similarity or inner product, and if the storage structure is separated into modality-specific indices, candidates can be searched from the text index, image index, and audio index, respectively, and then merged. In one embodiment, if the query is text, a search is performed first based on the text index, and then image-based candidates can be additionally combined.

[0201] In step S835, the multimodal search device (100) may determine whether precise reordering is necessary for the candidate search results. This determination may be based on the score difference between candidates, the type of query, the response time limit, or the service level. For example, if it is determined that the difference in similarity between top candidates is very small and the possibility of misjudgment is high, a precise reordering procedure may be performed, and in an environment where large-scale requests are concentrated, only the embedding score may be used without reordering to reduce response delay.

[0202] In step S840, if precise reordering is required, the multimodal search device (100) can perform reranking based on ITM or a cross-encoder. Specifically, for each of the top-N candidates obtained through ANN search, a query and a candidate item can be input as pairs to recalculate the precise matching score. For example, a query sentence and an image candidate can be input together into a cross-encoder to calculate a sentence-image matching score, which can then be combined with the existing embedding score to reorder the ranking. This method can reflect subtle semantic differences even when there are many hard negatives. For example, if there are many images that are visually similar but semantically different, it may be difficult to distinguish them based solely on embedding-based distance, but the cross-encoder can correct the ranking by reflecting context-unit matching.

[0203] In step S841, if precise reordering is not required, the multimodal search device (100) can determine the final ranking using only the embedding scores. In this case, the candidates can be sorted using the cosine similarity or inner product values ​​as they are, and since the result can be returned immediately without separate recalculation, it may be advantageous in low-latency environments or environments with large-scale concurrent requests. For example, in a mobile service environment, the S841 path may be selected first to prioritize response time.

[0204] In step S850, the multimodal search device (100) may return the final search result to the user. The returned result may be a Top-K search result, and each result may include a descriptive phrase, highlight information, a representative image, or summary information. In one embodiment, when returning an image search result corresponding to a text query, the main descriptive sentence in which the image was selected may be included along with the image.

[0205] In step S860, the multimodal search device (100) may record log and monitoring information regarding the search execution process. This may include query processing time, number of candidates, whether reordering was performed, whether a final click was made, or user response information. Additionally, the distribution indicator may optionally be updated based on the embedding location where the current query is projected, and may be used as a projector-only correction trigger for the correction control unit in the event of repeated centroid bias, reduced cohesion, or increased mixing in a specific language or specific domain query. For example, if search accuracy is repeatedly reduced in a specific language query, the need for projector correction may be determined based on the corresponding query log.

[0206] According to the procedure of Fig. 8, rapid candidate extraction based on ANN and selective precision reordering can be performed in parallel during the online search phase, allowing for balanced control of response speed and search accuracy depending on the situation. In addition, by continuously monitoring the query distribution and signs of quality degradation even after the search results are returned, the stability of the common embedding space and search performance can be maintained even during long-term operation.

[0208] FIG. 9 is a flowchart illustrating an operation and correction loop that detects quality degradation or distribution collapse during the operation of a multimodal search system according to an embodiment of the present invention, performs a correction operation in response thereto, and then performs verification and distribution.

[0209] The procedure according to FIG. 9 may be a procedure that continuously monitors service and embedding metrics, diagnoses the cause when quality degradation or collapse is detected, performs correction work centered on the projection layer first, verifies the correction results to determine whether improvement is needed, and if improvement is insufficient, re-explores parameters or policies. Through such a structure, one embodiment of the present invention can actively respond to distortion in the distribution of the common embedding space or degradation in search quality that may occur during the long-term operation of a multimodal embedding model in an actual service environment.

[0210] The "calibration" used in FIG. 9 may be a concept corresponding to the calibration mode or projection layer selective update described in FIG. 5. Additionally, the "collapse" in FIG. 9 may be understood as a concept including distribution distortion or representation collapse in which embedding information in the common embedding space is excessively concentrated in a specific direction or specific area, the distinguishability between modalities is reduced, or samples that need to be semantically distinguished fail to maintain a sufficient distance structure. Accordingly, the expressions used in the conception of FIG. 9 may be understood as representing technical concepts functionally identical to the distribution distortion or collapse of the common embedding space and projection layer calibration described in the claims and the preceding drawings.

[0211] Referring to FIG. 9, at step S910, the multimodal search device (100) can monitor service and embedding indicators. Specifically, the multimodal search device (100) can monitor service indicators such as search accuracy, recall, click rate, and response latency, as well as embedding indicators representing the distribution state in a common embedding space. The embedding indicators may include anisotropy-related indicators by modality or language, mean cosine similarity, spectral characteristics, distance between center vectors, cluster radius, distribution cohesion, distribution separation, over-mixing indicators, or over-separation indicators. For example, if the search quality for a specific language query is continuously degraded or if specific modality embeddings are excessively concentrated in a narrow direction, the multimodal search device (100) can detect the phenomenon as a change in indicators.

[0212] In step S910, the multimodal search device (100) can calculate the indicator using real-time query logs collected during the online search process, search result click history, evaluation results for offline verification datasets, or batch-unit embedding statistics. For example, by analyzing the recall change over a certain window period and the center shift amount by modality together, it is possible to more accurately determine whether a change in the embedding distribution structure itself has occurred, rather than a simple increase in service load. Due to such a structure, one embodiment of the present invention can track the structural stability of the common embedding space during operation, going beyond simple search score monitoring.

[0213] In step S915, the multimodal search device (100) can determine whether quality degradation or collapse has been detected. This determination can be made based on whether the service indicators and embedding indicators calculated in step S910 fall outside the reference range. For example, if the search recall drops below the reference value and, at the same time, the anisotropy indicator of a specific modality increases sharply, the multimodal search device (100) can determine that distribution distortion or collapse has occurred in the common embedding space. Conversely, if the indicators remain stable within the reference range, it can be determined that the device is in a normal operating state.

[0214] In step S921, if no quality degradation or collapse is detected as a result of the judgment in step S915, the multimodal search device (100) can maintain normal operation. In this case, the multimodal search device (100) can maintain the current service model as is, continue recording metrics, and repeat the monitoring procedure of step S910 for subsequent new queries or operational data. For example, if both the search accuracy and embedding distribution metrics are stable within a specific time interval, only multi-window-based monitoring can be performed while maintaining the current state without separate correction. With such a structure, stable operation can be maintained without unnecessary model updates or index rebuilds.

[0215] In step S920, if quality degradation or collapse is detected as a result of the judgment in step S915, the multimodal search device (100) can perform a diagnostic analysis. Specifically, the multimodal search device (100) can analyze whether the cause of the problem lies on the encoder side by modality, on the projection layer side, or is limited to a specific language, a specific domain, or a specific item cluster. For example, if the vision encoder itself remains stable but only the center distance from the text embedding after projection increases abnormally, the multimodal search device (100) can estimate the failure location as a bottleneck on the projection layer side. Additionally, if distribution collapse occurs only in a specific language group, the possibility of failure in language-specific projector alignment can be evaluated as high.

[0216] In step S920, the multimodal search device (100) can check the selection status of the anchor modal or anchor language, the alignment reliability threshold value, the gating threshold value, or the loss weight setting together during the diagnostic analysis process. For example, if the quality of a specific low-signal modality is degraded, it can be determined that the anchor modal selection is inappropriate and the subspace alignment criteria are not sufficiently stable. Alternatively, if the alignment reliability threshold value is excessively strict and valid samples are excessively excluded, it can be determined that the subspace alignment loss is not sufficiently reflected. Through such a diagnostic structure, it is possible to distinguish response paths by cause, going beyond merely recognizing performance degradation.

[0217] In step S930, the multimodal search device (100) can perform a correction operation.

[0218] Specifically, the multimodal search device (100) can perform a projector-only correction mode that updates only the projection layer while keeping the modality-specific encoder fixed based on the diagnosis results. In this case, similar to the correction mode described in FIG. 5, only the mapping relationship to the common embedding space can be adjusted while maintaining the semantic representation structure possessed by the existing encoder. For example, if the relative positions of a specific language text embedding and an image embedding are drifted, the multimodal search device (100) can fix the text encoder and the vision encoder and readjust only the text side or the common projector.

[0219] In step S930, the multimodal search device (100) may incorporate subspace alignment and geometric distribution constraints together during the correction operation. For example, the reference subspace of the anchor modality may be re-estimated, and the subspace alignment loss may be included in the correction loss in a direction that reduces the projection error of the target modality. Additionally, if indicators related to cluster radius, inter-class distance, distribution diversity, or collapse suppression change abnormally, the weights of the loss terms related to volume, separation, aggregation, or diversity may be readjusted and reflected in the correction operation.

[0220] In one embodiment, if it is determined that projector-only correction is insufficient, the multimodal search device (100) may extend to full parameter fine-tuning. For example, if a large amount of new domain data is introduced and the encoder representation itself must change, limited fine-tuning including some encoder blocks may be performed.

[0221] In step S940, the multimodal search device (100) can verify the results of the correction operation and, if necessary, deploy them. Specifically, the multimodal search device (100) can verify the effectiveness of the corrected model by performing performance evaluations on an offline verification dataset and online A / B tests. For example, it can compare recall, precision, click-through rate, average response time, center distance by modality, cluster radius, or anisotropy metrics before and after correction. If the verification results satisfy the criteria, the multimodal search device (100) can deploy the corrected model or updated projector parameters to the actual service path. Additionally, if it is determined that the index reference space has changed, it can also determine whether an offline index rebuild is necessary.

[0222] In step S945, the multimodal search device (100) can determine whether there is improvement after correction. This determination can be made based on the verification results of step S940. For example, if the recovery of search accuracy, normalization of distribution indicators, and service stabilization are all confirmed, it can be determined that there is improvement. Conversely, if only some indicators are improved and the core quality indicators still fall short of the standard, it can be determined that the improvement is insufficient.

[0223] In step S951, if the improvement of the judgment result in step S945 is insufficient, the multimodal search device (100) may re-search parameters or policies. Specifically, the multimodal search device (100) may reset the weights of geometric distribution constraint losses, alignment confidence thresholds, gating criteria, anchor modal selection policies, or combination methods per modality. For example, if a specific modality consistently converges into a narrow cone shape, the reflection ratio of volume or diversity-related loss terms may be increased, and if excessive separation persists, the separation-related threshold may be relaxed or the anchor modal may be replaced. Additionally, if necessary, the modal configuration itself may be adjusted to exclude specific modalities or add new auxiliary modalities. The results of the re-search may be reflected back into the correction operation of step S930.

[0224] If the improvement is confirmed as a result of the judgment in step S945, the multimodal search device (100) may terminate the correction loop or return to a normal operating state.

[0225] For example, if the calibrated projector operates stably and the distribution indicator is maintained within the reference range, the multimodal search device (100) can subsequently monitor the operating status again through the monitoring procedure of step S910. Through such a repeating loop structure, one embodiment of the present invention can implement a quality management system during continuous operation rather than one-time calibration.

[0226] The procedure according to FIG. 9 can be distinguished from the conventional response method of simply retraining the entire model after service quality degradation occurs. In the conventional method, since the cause of search quality degradation—whether it is embedding distribution distortion, a projector bottleneck, or an encoder representation limit—can be clearly distinguished, the entire model may be retrained, resulting in high costs and high uncertainty during operation. On the other hand, the multimodal search device (100) according to one embodiment of the present invention monitors service indicators and embedding indicators together through the illustrated steps S910 to S951, detects distribution distortion or collapse early, diagnoses the cause, performs projector-only correction first, verifies the results, and can even perform policy re-search if necessary. With such a structure, the structural stability of the common embedding space and search quality can be efficiently restored without retraining the entire encoder.

[0227] According to the operation and correction loop of Fig. 9, problems such as distribution collapse, alignment failure between modalities, excessive mixing, or excessive separation that may occur in the actual service environment of a multimodal search system can be actively addressed. Accordingly, the long-term stability of the common embedding space, the sustainability of search accuracy, and service operation efficiency can be improved together.

[0229] FIG. 10 is a flowchart illustrating a procedure for identifying a distribution collapse type of a common embedding space and selectively correcting a projection layer corresponding to the collapse type according to an embodiment of the present invention.

[0230] The procedure according to FIG. 10 may be a procedure that collects embedding results for input modality data, evaluates the distance between classes and cluster distribution in a common embedding space, determines whether collapse occurs relative to a reference geometric structure, selects a projection layer to be corrected corresponding to the type of collapse, calculates a correction value reflecting subspace alignment conditions to update the projection layer parameters, and re-evaluates the embedding distribution after the update to determine whether to maintain it. Here, the geometry used in FIG. 10 can be understood as a concept corresponding to the geometric distribution structure or geometric distribution constraint of the common embedding space described above, and the projector can be understood as a concept corresponding to the projection layer responsible for mapping to the common embedding space. Therefore, the expressions used in the conception theory of FIG. 10 can be understood as representing technical concepts that are functionally identical to the terms described in the claims and FIGs. 2 through 9.

[0231] Referring to FIG. 10, in step S1010, the multimodal search device (100) may collect embedding results for input modality data. Here, the input modality data may include text, images, audio, video, or a combination thereof, and the embedding results may be embedding information generated in a common embedding space, a center vector per modality, a cluster radius, a dispersion structure, or a history of similarity between a search query and a search target. The multimodal search device (100) may use embedding results of online queries collected during service operation, or may use batch data for verification or a set of samples accumulated over a recent period. For example, by collecting text embeddings for a specific language query together with embeddings of image items frequently selected in response to that query, the embedding distribution in the actual service environment can be reflected more directly.

[0232] In one embodiment of step S1010, the multimodal search device (100) can collect embedding results separately by modality, language, or domain. For example, by collecting Korean text query sets, English text query sets, image query sets, and audio query sets separately from one another, it is possible to identify not only system-level anomalies but also local breakdowns occurring only in specific subgroups. In this case, even when using the same common embedding space, it is possible to separately detect phenomena where only the embeddings of a specific language group are concentrated in a narrow conical area, or where only the cluster radius of a specific modality increases excessively.

[0233] In step S1020, the multimodal search device (100) can evaluate the distance between classes and the cluster distribution in a common embedding space. Specifically, the multimodal search device (100) can measure the distance between center vectors corresponding to semantically different classes or groups, the cluster radius by modality or class, the direction of dispersion, the covariance structure, the principal component distribution, or the mean similarity distribution. The evaluation may include not only the simple average distance, but also the degree of overlap between clusters, the degree of bias toward a specific axis direction, the degrees of freedom of distribution, and the local concentration of a specific modality. For example, even if image and text embeddings belonging to the same electronic product category are aligned with each other, if the distance between subclasses that need to be distinguished according to product condition or defect type decreases rapidly, the multimodal search device (100) may determine that a phenomenon of degraded separation is occurring.

[0234] In one embodiment of step S1020, the multimodal search device (100) may calculate volume-related indicators, separation-related indicators, aggregation-related indicators, and diversity-related indicators together when evaluating cluster distribution. For example, determinant values, sum of singular values, or rank values ​​may be used as indicators representing distribution volume, distance between class centers and principal angles may be used as separation indicators, mean radius within same semantic groups may be used as aggregation indicators, and mean cosine similarity and spectral entropy may be used as diversity indicators. By using multiple indicators in parallel in this way, structural anomalies that are difficult to identify solely by search scores can be identified at an earlier stage.

[0235] In step S1030, the multimodal search device (100) can determine whether collapse has occurred relative to a reference geometric structure. Here, the reference geometric structure may refer to a reference distribution, reference center distance, reference cluster radius, reference anisotropy level, or reference rank structure obtained at a normal learning time or a previous stable operation time. The multimodal search device (100) can detect whether distribution collapse or distortion has occurred by determining whether the current evaluation result deviates from an acceptable range when compared to the reference geometric structure. For example, if the center vector of a text embedding suddenly moves away from the center vector of an image embedding, or if the average cosine similarity of a specific language text embedding rises sharply so that different sentences converge in almost the same vector direction, the multimodal search device (100) can determine that collapse has occurred relative to the reference geometric structure.

[0236] In one embodiment of step S1030, the multimodal search device (100) may not only determine whether a collapse has occurred through binary judgment, but may also calculate it in the form of a collapse risk score. For example, a collapse risk score may be formed by weighted combination of the distribution volume reduction rate, the cluster radius increase rate, the center distance deviation, and the diversity degradation rate, and a collapse may be determined to have occurred when the score is greater than or equal to a threshold value. With such a structure, potential distribution anomalies can be detected early even before serious quality degradation appears as an external service indicator.

[0237] In step S1040, the multimodal search device (100) can classify the collapse type. Specifically, the multimodal search device (100) can determine whether the current abnormal state is an excessively mixed type, an excessively separated type, a low-rank collapse type, a specific modality bias type, a specific language bias type, or a specific group cohesion failure type by using the indicators calculated in steps S1020 and S1030. For example, if the center distance between classes is maintained but only the radius within the same class increases excessively, it can be classified as a cohesion failure type; and if all samples are excessively clustered in a specific axis direction and the rank value decreases rapidly, it can be classified as a low-rank collapse type (e.g., a phenomenon where low-resource language embeddings converge into a narrow cone structure) or a bifurcation phenomenon where the representation splits rapidly by modality.

[0238] In one embodiment of step S1040, the multimodal search device (100) may set a subsequent correction strategy differently depending on the classified collapse type. For example, in the case of a low-rank collapse type, volume and diversity recovery may be set as the priority goal, in the case of an over-separation type, alignment enhancement for the anchor reference subspace may be set as the priority goal, and in the case of a specific modality bias type, only the projection layer connected to the corresponding modality may be designated as the priority correction target. With such a structure, unlike a method of applying uniform correction to the entire model, a correction path consistent with the cause of the anomaly can be selected.

[0239] In step S1050, the multimodal search device (100) can select a projection layer to be corrected corresponding to the type of collapse. Specifically, the multimodal search device (100) can determine which projection layer contributes most significantly to the current distribution distortion based on the classified type of collapse and the influence of each modality. For example, if the text modality embeddings are concentrated in a narrow direction and only the image-text alignment collapses, the text-side projection layer can be selected as the priority target for correction, and if the cluster radius expands abnormally only in a specific multimodal combined path, the combined projection layer can be selected. With such a structure, only the transformation path causing the problem can be precisely corrected without retraining the entire modality-specific encoder.

[0240] In one embodiment of step S1050, the multimodal search device (100) can select a plurality of correction target projection layers with priority. For example, if the main cause of collapse is the text projector but the image projector also has a secondary effect, the first correction target and the second correction target can be designated sequentially. In this case, the first correction is performed only on the first correction target, and can be extended to the second correction target only if the effect is insufficient. This sequential correction structure can enable stable correction while reducing unnecessary parameter changes.

[0241] In step S1060, the multimodal search device (100) can calculate a correction value that reflects subspace alignment conditions for the selected projection layer. Specifically, the multimodal search device (100) can reset the reference subspace of the anchor modality or retrieve the existing reference subspace, project the embedding information of the modality to be corrected onto the reference subspace, and then calculate a correction value in a direction that reduces the distance, angle difference, or axial error between the projection result and the reference subspace. At this time, the correction value can be calculated such that geometric distribution constraints and subspace alignment conditions are simultaneously satisfied. For example, since strengthening only simple alignment while the distribution volume is reduced may lead to more severe excessive concentration, the multimodal search device (100) can set the correction value by considering both the volume recovery term and the subspace alignment term together.

[0242] In one embodiment of step S1060, the multimodal search device (100) may apply alignment reliability or gating values ​​together when calculating the correction value. For example, the correction direction can be stabilized by assigning a larger weight to samples that show high alignment with the reference subspace, and assigning a smaller weight to samples that are noisy or have low alignment with the reference axis. Additionally, if only a specific class has collapsed, the set of samples belonging to said class may be reflected first, and the normal class may be processed with a lower weight in the correction calculation. With such a structure, the problem area can be precisely corrected without the correction value unnecessarily shaking the entire distribution.

[0243] In step S1070, the multimodal search device (100) can update projection layer parameters by applying a correction value. In this step, the multimodal search device (100) can keep the modality-specific encoders fixed and update only the selected projection layer. For example, while keeping the text encoder, image encoder, and audio encoder as they are, only the weight matrix of the text projector or combined projector can be modified according to the correction value. This structure may be technically advantageous in that it can efficiently correct only the relative positional relationships and distribution structures in the common embedding space without damaging the existing semantic representation structure.

[0244] In one embodiment of step S1070, the multimodal search device (100) may implement a correction value application method using a direct substitution method, a gradient-based fine-tuning method, or a limited-range residual correction method. For example, it may be implemented by adding a small residual matrix to the current projector parameters to reduce axial errors, or by performing only a few fine-tuning steps on the correction loss to achieve stable convergence. In this case, the amount of computation and service downtime can be significantly reduced compared to full parameter retraining.

[0245] In step S1080, the multimodal search device (100) may re-evaluate the embedding distribution after the update and determine whether to maintain it. Specifically, the multimodal search device (100) may measure the inter-class distance, cluster radius, distribution volume, diversity index, center deviation, or search performance index again after the correction to determine whether the corrected projection layer has actually restored distribution stability and search quality. For example, if the recall of a specific language query was low and the anisotropy was high before the correction, but the recall is restored and the distribution spreads widely again after the correction, the multimodal search device (100) may decide to maintain the updated projector. Conversely, if the distribution is partially restored but the search quality deteriorates or side effects occur in other modalities, the update may be discarded or additional correction may be performed.

[0246] In one embodiment of step S1080, the multimodal search device (100) may perform offline verification and online verification in parallel when determining whether to maintain. For example, after evaluating recall, precision, and clustering metrics for a verification dataset, an online experiment on a small amount of traffic may be performed to verify whether the actual click-through rate or user response is improved. Through this dual verification structure, it is possible to more reliably verify whether geometric improvements actually lead to improvements in service quality.

[0247] The procedure according to FIG. 10 can be distinguished from a method of simply retraining the entire model after quality degradation. In conventional structures, since the entire parameter can be modified in the same way without subdividing the types of distribution collapse, the cause analysis is insufficient, the computational cost is high, and there is a possibility of unnecessarily disrupting the existing alignment structure. On the other hand, a multimodal search device (100) according to one embodiment of the present invention can determine whether collapse has occurred using actual embedding results collected from input modality data, subdivide the types of collapse, select a projection layer to be corrected correspondingly, calculate a correction value by reflecting alignment conditions for a reference subspace, and update only the selected projection layer. With such a structure, local correction consistent with the cause of collapse becomes possible, and the semantic structure of the common embedding space can be stably restored while reducing the correction cost.

[0248] For example, in a multilingual e-commerce search service, it can be assumed that the alignment between English queries and product images is maintained, but only the Korean query embeddings converge in a narrow direction, so that different product descriptions are represented by similar vectors. In this case, the multimodal search device (100) can detect the reduction of the cluster radius and the increase in anisotropy of the Korean text embeddings in steps S1020 and S1030, and classify it as a language-biased low-rank collapse in step S1040. Subsequently, in step S1050, a Korean text-side projector is selected, in step S1060, a correction value is calculated to reduce the projection error of the Korean embeddings for the image anchor reference subspace, and in step S1070, only the projector can be updated. As a result, the existing English path can be maintained while selectively recovering only the Korean search quality. This embodiment can clearly demonstrate the technical excellence of the present invention in that it can rapidly correct partial quality degradation without retraining the entire multimodal model.

[0249] As another embodiment, it can be assumed that in an industrial equipment inspection system, thermal images, inspection text reports, and voice memos are search targets together. In this case, the cluster radius of only the embeddings corresponding to the voice memos may become excessively large during a specific period, which may reduce cohesion for the same type of failure. The multimodal search device (100) may classify this as a cohesion failure type collapse and select the voice projector as a target for correction. Subsequently, a correction value that reduces the projection error of the voice embeddings is calculated for a reference subspace using the thermal image embeddings as an anchor modality, and only the voice projector may be updated. In this case, the image and text paths are maintained as they are, while only the search precision based on voice queries can be selectively improved.

[0250] According to the procedure of FIG. 10, one embodiment of the present invention goes beyond simply detecting the collapse of a common embedding space to structurally classify the collapse type and selectively correct only the projection layer corresponding to the collapse type by reflecting the structured reference subspace alignment condition. Accordingly, while preserving the existing semantic structure of the entire multimodal embedding model as much as possible, only the local path where distribution stability has deteriorated can be precisely corrected, thereby significantly improving the long-term operational stability, correction efficiency, and service continuity of multimodal search.

[0252] FIG. 11 is a flowchart illustrating a hierarchical anchor subspace alignment and axis-by-axis selection alignment procedure according to one embodiment of the present invention.

[0253] The procedure according to FIG. 11 may be a procedure in which the reference subspace of an anchor modality is divided into multiple axis groups, the embedding information of a target modality is decomposed for each axis group, the difference in distribution for each axis group is calculated to determine the alignment priority, the subspace alignment transformation is selectively performed only on the axis groups with high priority, and for the non-selected axis groups, the alignment results for each axis group are combined and reflected in the final projection layer output while maintaining the existing geometric structure. With such a structure, unlike a method that performs alignment of the same strength uniformly across the entire common embedding space, alignment can be selectively strengthened only on semantically important axis groups while preserving the remaining axis groups, thereby simultaneously achieving semantic correspondence precision between modalities and preservation of the modality's unique representation.

[0254] The "reference subspace" used in FIG. 11 can be understood as a concept corresponding to the reference subspace or anchor subspace described previously in FIG. 4, 5, and 6. Additionally, "geometry" in FIG. 11 can be understood as a concept corresponding to a distribution state formed by a geometric distribution structure or geometric distribution constraints in a common embedding space, and "projector output" can be understood as a concept corresponding to the embedding information of the common embedding space generated by the projection layer. Accordingly, the expressions used in the conception of FIG. 11 can be understood as representing technical concepts that are functionally consistent with the terms used in the claims and the preceding drawings.

[0255] Referring to FIG. 11, in step S1110, the multimodal search device (100) can divide the reference subspace of the anchor modality into a plurality of axis groups. Specifically, the multimodal search device (100) can form a reference subspace by performing singular value decomposition or principal component analysis on the set of embedding information of the anchor modality, and then divide the reference subspace into a plurality of axis groups according to the nature of the semantic axes or the contribution of the variance. For example, upper principal component axes can be classified into a first axis group that strongly reflects common semantics, axes with intermediate levels of variance can be classified into a second axis group that reflects detailed attributes or auxiliary semantics, and axes with relatively low variance can be classified into a third axis group that reflects modality-specific expressions or residual expressions. By such a structure, the reference subspace can be treated as a structure having multiple semantic hierarchies rather than a single alignment target.

[0256] In one embodiment of step S1110, the multimodal search device (100) may consider the contribution to dispersion, axis stability, commonality between modalities, or sensitivity to search performance when splitting axis groups. For example, when using image modality as an anchor modality, axes that strongly describe object types or scene categories may be set as a higher priority group, and axes that describe auxiliary visual elements, such as color deviation, background texture, or shooting conditions, may be set as a lower group. This splitting structure may serve as a basis for performing selective alignment only on semantically important axes in subsequent steps, rather than aligning all axes equally.

[0257] In step S1120, the multimodal search device (100) can decompose the target modality embedding by axis group. Specifically, the multimodal search device (100) can project the embedding information of the target modality onto each axis group divided in step S1110 to produce a partial representation corresponding to each axis group. For example, if the text embedding is the target modality, the text embedding can be decomposed into a projection component in the direction of the first axis group, a projection component in the direction of the second axis group, and a projection component in the direction of the third axis group. Due to this structure, even if the embedding of the target modality appears similar to the anchor modality overall, it is possible to subdivide and identify which axis group actually has a large discrepancy.

[0258] In one embodiment of step S1120, the multimodal search device (100) may process the target modality embedding using a simple orthogonal decomposition method or a weighted projection method by axis group. For example, if the axis group boundaries are clear, coordinate components separated by orthogonal axis groups may be used, and if there is interdependence between axis groups, decomposition reflecting the contribution rate for each axis may be performed using a soft assignment method. With such a structure, stable analysis by axis group can be performed even if there is incompleteness in the actual service data or correlation between axes.

[0259] In step S1130, the multimodal search device (100) can calculate the distribution difference for each axis group. Specifically, the multimodal search device (100) can calculate the distance difference, angle difference, dispersion difference, cluster radius difference, center displacement amount, or distribution volume difference between the anchor modality and the target modality for each axis group. For example, if the center direction of the text embedding and the image embedding aligns well in the first axis group, but the average angle difference appears significantly in the second axis group, and only specific text samples show excessively skewed dispersion in the third axis group, the multimodal search device (100) can determine that a different distribution difference pattern exists for each axis group. Due to this structure, detailed discrepancies that are difficult to identify solely by the average similarity of the entire embedding vector can be analyzed at the axis group level.

[0260] In one embodiment of step S1130, the multimodal search device (100) may calculate the distribution difference by axis group not merely as a simple numerical difference, but as an influence score combined with the contribution to search failure. For example, if the actual search ranking changes significantly even with only a small angle difference in a specific axis group, the axis group may be evaluated as having high search sensitivity. Conversely, if the difference in a specific axis group is large but has little effect on the search results, the need for alignment of the axis group may be evaluated as low. By such a structure, the priority determination of subsequent steps can be configured to reflect the contribution to actual search quality beyond the level of simple statistical differences.

[0261] In step S1140, the multimodal search device (100) can determine the alignment priority. Specifically, the multimodal search device (100) can determine which axis group to prioritize for alignment by synthesizing the distribution difference per axis group, search sensitivity, commonality between modalities, and the current geometric distribution constraint state calculated in step S1130. For example, the highest priority can be assigned if the first axis group directly reflects common meaning and has a large distribution difference, and a low priority can be assigned if the third axis group corresponds to the unique characteristics of the modality and has low search sensitivity. Due to this structure, instead of applying the same alignment to all axes, the axis groups that significantly affect actual meaning correspondence and search precision can be selectively aligned first.

[0262] In one embodiment of step S1140, the multimodal search device (100) may determine the priority fixedly or dynamically in conjunction with the collapse type classification results described in FIG. 10. For example, in a specific domain, the importance of the color axis or the background axis is low, but in a domain where detailed visual differences are important, such as medical image search or defect detection search, the importance of a specific intermediate axis group may increase. In this case, the multimodal search device (100) may dynamically adjust the axis group priority according to a domain-specific policy. This structure can increase adaptability to various application environments compared to a fixed alignment structure.

[0263] In step S1150, the multimodal search device (100) can select a high-priority axis group. For example, it may select only the top 1 axis group, or it may simultaneously select multiple axis groups whose cumulative importance is greater than or equal to a reference value. In this case, the selected axis group becomes a target for alignment reinforcement, and the unselected axis group can be treated as a target for preservation in a subsequent step. With this structure, it is possible to intensively correct only the core axis group that determines the common meaning, while ensuring that the modality-specific information expressed in the remaining axes is not excessively modified.

[0264] In one embodiment of step S1150, the multimodal search device (100) can adaptively adjust the number of selected axis groups. For example, in the initial learning stage or when strong collapse is detected, multiple top axis groups can be selected simultaneously, and in the stabilization stage or fine correction stage, only one or two top axis groups can be selected. With such a structure, the balance between correction strength and representation preservation can be controlled on a situational basis.

[0265] In step S1160, the multimodal search device (100) can perform a subspace alignment transformation for a selected axis group. Specifically, the multimodal search device (100) can perform adjustments to a transformation matrix, a correction vector, or local weights within a projector so that the projection result of the target modality is better aligned with the reference subspace of the anchor modality only for the direction of the selected axis group. For example, if the text embedding in the first axis group corresponding to the common semantic axis shows an angular difference from the image reference subspace, the multimodal search device (100) can fine-tune the projection position by modifying the weights in the direction of the corresponding axis of the text projector. With such a structure, alignment errors in the core semantic axis can be reduced without relocating the entire embedding vector in bulk.

[0266] In one embodiment of step S1160, the multimodal search device (100) may apply alignment reliability or sample-specific gating values ​​together during subspace alignment transformation. For example, for samples showing high similarity with the anchor modality, the alignment transformation of the selected axis group may be strongly reflected, and for samples with high noise or unclear semantic correspondence, it may be weakly reflected. In addition, the alignment strength may be set differently for each class even within the same axis group. Due to such a structure, alignment for the selected axis group can be performed more stably and precisely.

[0267] In step S1170, the multimodal search device (100) may maintain the existing geometric structure of the non-selected axis groups. Specifically, the multimodal search device (100) may not apply a separate alignment transformation to axis groups that have low priority or are primarily responsible for modality-specific expressions, or may allow only minimal changes. For example, if a specific axis group reflects modality-specific information such as hue differences, background textures, or tone differences, the multimodal search device (100) can reduce the problem of excessive loss of modality-specific information during the common semantic alignment process by maintaining the existing geometry of said axis group. Such a structure can solve the problem that if all axes are aligned equally, the expressiveness becomes homogenized, which may actually reduce search discriminability.

[0268] In one embodiment of step S1170, the multimodal search device (100) may use a fully fixed method or a limited preservation method for the non-selected axis group. For example, the dispersion structure of the non-selected axis group may be maintained, while allowing only minimal adjustments associated with full vector normalization. Alternatively, a preservation normalization term may be added so that the correction of the selected axis group does not excessively shake the relative distribution of the non-selected axis group. With such a structure, overall distribution stability can be maintained even if strong alignment is performed for the selected axis group.

[0269] In step S1180, the multimodal search device (100) can combine alignment results by axis group and reflect them in the final projector output. Specifically, the multimodal search device (100) can combine the aligned components from the selected axis group and the components maintained from the unselected axis group to form final embedding information, which can be used as a projection layer output or a subsequent common embedding space input. For example, the first axis group may be reflected in a strongly aligned state, while the second and third axis groups may be combined while maintaining the existing geometry. With such a structure, the final projector output may have a balanced structure that preserves representational diversity in modality-specific axes while increasing alignment precision in common semantic axes.

[0270] In one embodiment of step S1180, the multimodal search device (100) may re-normalize the combined results by axis group or input them into the correction loop described in FIGS. 5 and FIGS. 10 to utilize for additional learning or additional projector correction. For example, if the total distribution volume is excessively reduced even after the alignment results by axis group are combined, the multimodal search device (100) may stabilize the final output by reapplying volume or diversity constraints. Due to this structure, the hierarchical axis selection alignment may be used as an independent correction procedure or may be used in a combined form within the overall learning or operational correction loop.

[0271] The procedure according to FIG. 11 can be distinguished from a conventional method that performs alignment of the same strength using only a single reference subspace for the entire common embedding space. In the conventional method, as the overall alignment strength increases, common semantic alignment can be improved, but at the same time, modality-specific information may be excessively lost, which may reduce the ability to distinguish details in search. Conversely, if the alignment strength is set weakly, modality-specific information is preserved, but inconsistencies may remain in the core semantic axes. On the other hand, a multimodal search device (100) according to an embodiment of the present invention divides the reference subspace into a plurality of axis groups, analyzes the distribution difference for each axis group to determine priority, and then selectively performs subspace alignment transformation only on the axis group with high priority, while maintaining the existing geometry of the non-selected axis group. With such a structure, alignment precision can be increased in the core axes reflecting common semantics, while expression diversity can be maintained in the modality-specific axes.

[0272] For example, one can assume a case where an image modality is used as an anchor modality and a text description is used as a target modality in an e-commerce multimodal search service. In this case, an axis group describing product type, brand, and category may have a high priority, while an axis group related to color nuances, shooting environment, or marketing phrases may have a low priority. The multimodal search device (100) can reduce mismatches in major semantic axes such as "smartphone," "laptop," and "earphone" by precisely aligning text embeddings to the image-based subspace for the high axis group, and can preserve the diversity of additional expressions such as "silver," "matte," and "premium sentiment" by maintaining the low axis group. Such an embodiment can clearly demonstrate the technical effect of the present invention in that it offers a superior balance between actual search quality and expression preservation compared to simple overall alignment.

[0273] As another embodiment, it can be assumed that a medical image search system uses CT image embeddings as anchor modalities and diagnostic report text as target modalities. In this case, axis groups related to the presence or absence of lesions, lesion locations, and lesion types may be set to a high alignment priority, while axis groups related to narrative styles or additional explanations may be given a low priority. The multimodal search device (100) can enhance the consistency of core medical meanings between images and reports by performing selective alignment only on axis groups related to lesions, and preserve document expression diversity by maintaining axis groups related to expression styles. Such a structure can be very useful in that it can increase the accuracy of core lesion searches in actual clinical searches while maintaining the flexibility of report expressions.

[0274] According to the procedure of FIG. 11, one embodiment of the present invention hierarchically decomposes the reference subspace of the anchor modality and evaluates the alignment requirements for each axis group for the target modality embedding, and then selectively performs subspace alignment only on the axis group with high priority, while maintaining the existing geometric structure of the remaining axis groups. Accordingly, alignment precision in the common semantic axis is improved, and representational diversity and distinguishability in the modality-specific axis can be maintained, thereby comprehensively improving the accuracy, explainability, and long-term operational stability of multimodal search.

[0276] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0277] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0278] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0279] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0280] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

[0281] Although specific embodiments have been described in the detailed description of this document, it will be obvious to those skilled in the art that various modifications are possible within the scope of this document. Explanation of the symbols

[0282] 10 : User terminal 100 : Multimodal search device 110: Embedding generation unit 120: Sorting learning execution unit 130: Geometric control unit 140: Search execution unit 150 : Correction control unit 200 : Storage device to search 300: Model Management Unit

Claims

Claim 1 A multimodal search method performed in a multimodal search device, comprising: a step of converting data of multiple different modalities into embedding information of a common embedding space; and a step of generating a search result based on similarity between a search query and a search target using a multimodal embedding model; wherein the multimodal embedding model is characterized by being learned based on geometric distribution constraints for controlling the distribution of embedding information in the common embedding space and subspace alignment for the embedding information. Claim 2 A multimodal search method using geometric distribution constraints and subspace alignment of a common embedding space, wherein the multimodal search method further comprises: a step of performing learning for alignment between embedding information of a plurality of different modalities in the common embedding space, while reflecting geometric distribution constraints for controlling the distribution of embedding information in the common embedding space; and a step of learning a multimodal embedding model by reflecting subspace alignment for the embedding information of a plurality of different modalities. Claim 3 In claim 2, the data of the plurality of different modalities includes training data of at least two different modalities corresponding to the same content, and the step of converting into embedding information of the common embedding space comprises: a step of encoding the training data of the at least two different modalities into feature information using a modal-specific encoder for each; and a step of mapping the feature information into mutually comparable embedding information in the common embedding space using a projection layer; a multimodal search method using geometric distribution constraints and subspace alignment of a common embedding space. Claim 4 A multimodal search method using geometric distribution constraints and subspace alignment of a common embedding space, wherein the step of reflecting the geometric distribution constraint comprises: a step of calculating a geometric distribution constraint loss that controls at least one of volume, separation, aggregation, and diversity for the distribution of the embedding information in order to control the distribution shape of the embedding information in the common embedding space; and a step of reflecting the geometric distribution constraint loss in learning for alignment between the embedding information. Claim 5 In claim 4, the step of performing learning for alignment between the embedding information comprises: a step of calculating an alignment loss that increases the similarity between embedding information of different modalities corresponding to the same content and decreases the similarity between embedding information corresponding to different content; and a step of training the multimodal embedding model to reflect both semantic alignment and distribution stabilization of embedding information in the common embedding space by combining the alignment loss and the geometric distribution constraint loss. A multimodal search method using geometric distribution constraints and subspace alignment of a common embedding space. Claim 6 In paragraph 2, the step of training the multimodal embedding model by reflecting the subspace alignment comprises: a step of generating a reference subspace based on the embedding information of an anchor modality among the plurality of different modalities; a step of projecting the embedding information of a target modality onto the reference subspace; and a step of calculating a subspace alignment loss that reflects the degree of alignment between the anchor modality and the target modality using the projection result on the reference subspace, and reflecting the subspace alignment loss in the training; a multimodal search method using geometric distribution constraints of a common embedding space and subspace alignment. Claim 7 A multimodal search method using geometric distribution constraints and subspace alignment of a common embedding space, wherein, in claim 6, the step of generating the reference subspace comprises: a step of performing singular value decomposition or principal component analysis on a matrix composed of embedding information of the anchor modality; and a step of setting the reference subspace by selecting some of the upper components obtained as a result of the singular value decomposition or principal component analysis. Claim 8 A multimodal search method using geometric distribution constraints of a common embedding space and subspace alignment, wherein the step of incorporating the subspace alignment loss into the learning process comprises: a step of calculating alignment reliability based on the similarity between the embedding information of the anchor modality and the embedding information of the target modality; and a step of calculating the subspace alignment loss for embedding information where the alignment reliability is greater than or equal to a reference value, or calculating the subspace alignment loss by applying a weight corresponding to the alignment reliability. Claim 9 A multimodal search method using geometric distribution constraints and subspace alignment of a common embedding space, further comprising: a step of monitoring a distribution index of embedding information generated by the multimodal embedding model after the step of learning the multimodal embedding model; a step of detecting a distribution distortion or collapse of the embedding information in the common embedding space based on the distribution index and determining the projection layer as a target for correction; and a step of correcting the multimodal embedding model by updating the projection layer while keeping the modality-specific encoder fixed. Claim 10 A computer-readable storage medium storing a program for executing the method of any one of paragraphs 1 through 9 on a computer. Claim 11 A multimodal search device using geometric distribution constraints and subspace alignment of a common embedding space, comprising: at least one processor; and a memory for storing instructions, wherein when the instructions are executed by the at least one processor, the processor performs the operation of converting data of multiple different modalities into embedding information of a common embedding space; and the operation of producing a search result based on similarity between a search query and a search target using a multimodal embedding model, wherein the multimodal embedding model is characterized by being learned based on geometric distribution constraints for controlling the distribution of embedding information in the common embedding space and subspace alignment for the embedding information.

Images