Multimodal response generation method for enterprise-customized chemical formulation and process optimization recommendation, and electronic device for performing the same

Abstract

A method of operation of an electronic device providing information specialized to a chemical domain according to one embodiment may include: receiving a user input query and user profile information; generating an extended input query by combining the user profile information and the input query; obtaining a weight vector for each of a plurality of search models (Retrievers) based on the extended input query; extracting at least one document chunk from a multimodal vector database using a plurality of search models to which the weight vector is applied; and generating a response including formulation, physical property prediction values, and recommended process conditions along with the basis, based on the document chunk and the input query, using a chemical-specialized sLLM (smaller Large Language Model).

Description

Technology Field

[0001] The following disclosure relates to a method for generating a multimodal response, and more specifically, to a method for generating a multimodal response for recommending corporate-specific chemical formulations and process optimizations, and to an electronic device for performing the same. Background Technology

[0003] In the traditional chemical industry, securing a competitive edge requires in-depth expertise in chemicals, the acquisition of vast amounts of experimental data, and formulation and process optimization technologies to enhance physical properties. However, systematizing this knowledge and internalizing core technologies necessitates a massive investment of time and capital. In particular, small and medium-sized enterprises (SMEs), which lack research infrastructure and skilled personnel, face serious limitations in identifying the latest research trends or deriving advanced process conditions.

[0004] Existing information retrieval methods suffer from inefficiency due to the need to individually search fragmented literature data, and general-purpose AI models face the problem of being difficult to apply in actual industrial settings because they lack the precise reasoning capabilities specialized for chemical domains. In particular, current AI technologies related to chemicals tend to rely excessively on limited publicly available datasets, exhibiting distinct limitations in terms of flexibility and scalability. For instance, when new external data or specialized internal data is introduced, models must be retrained every time, or prediction accuracy drops sharply.

[0005] Recently, while domain-specific Small-Large Language Models (sLLM) have been actively adopted and yielding results in other industries such as finance, biotechnology, and law, the adoption of artificial intelligence in the chemical sector has been relatively delayed due to high levels of technical protection and a large amount of unstructured image data (charts, tables, etc.). Consequently, there is a growing demand for specialized solutions based on Search Augmented Generative Engineering (RAG) that can organically combine confidential internal data with vast external data of papers and patents without security incidents, and expand knowledge in real-time without the need for separate, full-scale retraining. Prior art literature

[0007] 1. Korean Registered Patent 2784194 B12. Korean Registered Patent 2851868 B13. Korean Registered Patent 2823155 B14. Korean Registered Patent 2799924 B1 The problem to be solved

[0008] The first problem that this invention aims to solve is to establish a customized knowledge system optimized for the research and production environments of individual companies without concerns about data leakage, by utilizing private experimental data and internal technical documents held by each company. Through this, the invention seeks to digitize latent internal know-how and provide a security-oriented knowledge internalization environment where artificial intelligence models can immediately reference and utilize this knowledge to provide answers.

[0009] The second task is to precisely structure not only text-based information but also complex table and chart data contained in papers and patents, and convert them into a form that enables semantic search. By integrating and managing internal corporate data and external professional academic knowledge within the same vector space, the goal is to enable sLLM to proactively grasp vast amounts of external information and perform comparative analysis with internal data to generate highly reliable physical and chemical prediction results.

[0010] Finally, we aim to implement a system that minimizes model response errors and continuously enhances performance through an autonomous learning mechanism that incorporates expert feedback into the learning process. The objective is to provide a next-generation multimodal response generation system capable of going beyond merely listing information in response to complex user queries to present practical technical solutions—such as proposing formulation ratios, calculating predicted physical property values, and deriving recommended process conditions—along with supporting literature. means of solving the problem

[0012] A method of operation of an electronic device providing information specialized to a chemical domain according to one embodiment may include: receiving a user input query and user profile information; generating an extended input query by combining the user profile information and the input query; obtaining a weight vector for each of a plurality of search models (Retrievers) based on the extended input query; extracting at least one document chunk from a multimodal vector database using a plurality of search models to which the weight vector is applied; and generating a response including formulation, physical property prediction values, and recommended process conditions along with the basis, based on the document chunk and the input query, using a chemical-specialized sLLM (smaller Large Language Model).

[0013] According to one embodiment, the method may further include, as a preliminary preparation step, a step of constructing the multimodal vector database including text, table summary, and chart summary data by collecting chemical-related literature; a step of storing keyword candidate sets by industry group by extracting and filtering entity names from the metadata of the multimodal vector database; a step of forming query clusters by applying clustering to a plurality of extended input queries; and a step of calculating and storing cluster-specific weight vectors by measuring the response consistency of each of a plurality of search models (Retrievers) for the query clusters.

[0014] According to one embodiment, the step of storing the keyword candidate group in advance may include: applying a named entity recognition model to the metadata for text chunks, table summary text, and chart summary text to extract named entities corresponding to raw materials, physical properties, and processes; and filtering the named entities based on their frequency of occurrence, literature distribution, and domain fit within the metadata and storing them as the keyword candidate group.

[0015] According to one embodiment, the step of generating the extended input query may include: converting industry group information included in the user profile information into an industry group embedding vector; converting a sentence describing the purpose of use included in the user profile information into a purpose semantic vector; generating a single keyword embedding vector by applying one of average pooling, importance-weighted average, or attention-based integration to a plurality of interest keywords selected by the user; and integrating the industry group embedding vector, the purpose semantic vector, and the single keyword embedding vector into a user profile vector.

[0016] According to one embodiment, the step of obtaining the weight vector may include: mapping the embedding of the extended input query to a specific cluster among the query clusters formed in the preliminary preparation step that is most similar in terms of industry group, query purpose, and keyword distribution characteristics; and calling the weight vector for the mapped cluster.

[0017] According to one embodiment, the step of calculating and storing the weight vector for each cluster may include: a step of defining a reference response distribution by calculating the arithmetic mean of the response tokens for each of the plurality of search models; a step of quantifying response consistency by measuring the KL-divergence between the response token distribution of each search model and the reference response distribution; a step of adjusting the weight sensitivity according to the KL-divergence value by applying a temperature parameter; and a step of determining the weight vector based on a final objective function including an entropy regularization term to prevent excessive concentration of weights on a specific search model.

[0018] According to one embodiment, the multimodal vector database integrates and stores text, table, and chart data within a document so that they can be searched in the same semantic space, and for search efficiency, only the summary data of the text, table, and chart is embedded and stored in a vector space, while the raw data of the table and chart is stored separately in individual fields of metadata.

[0019] According to one embodiment, the step of constructing the multimodal vector database may include: recognizing a table area within a document using a document layout recognition model and converting the table area into a markdown format; designating an area containing a predetermined tag within the markdown format as a table candidate and extracting a caption of the table within a predetermined line before or after the table candidate; generating a table-caption pair by matching the caption with the table, wherein a table without a caption is classified as administrative or bibliographic data and excluded; separating a header area and a data row from the table-caption pair and reconstructing them into semantic-based serialized text; generating a summary by extracting key scientific information from the serialized text through a first LLM specialized in content summarization; and analyzing the summary through a second LLM specialized in text semantics to determine whether the table corresponds to research data including experimental results, physical property values, or composition ratios, and finally storing the result.

[0020] According to one embodiment, the step of constructing the multimodal vector database may include: identifying images within a document through a document layout recognition model and sequentially assigning unique identifiers according to the order of appearance within the document and storing them in a directory; generating image caption candidates by extracting sentences containing predetermined keywords from a document converted into Markdown format; generating image-caption pairs by sequentially matching the unique identifiers and the image caption candidate group; analyzing the content of the image captions through an LLM to determine whether the images are related to a chart of experimental results and filtering them; applying a Chart-to-Text generation model to image-caption pairs determined to be related to the chart to convert the visual information of the chart into summary text; and embedding the summary text and storing it in a vector space, and storing the original data of the chart corresponding to the summary text in individual fields of the metadata.

[0021] According to one embodiment, the method further includes the step of updating a model based on expert feedback to maintain the answer quality of the sLLM, wherein the update may be performed when at least one of the following conditions is met: when new feedback data has accumulated more than a preset number, when the negative response rate in the user satisfaction evaluation is greater than a preset percentage, or when a domain imbalance is detected by detecting a quality degradation in a specific chemical domain. Effects of the invention

[0023] According to the present invention, by combining sLLM fine-tuned with chemical expertise data and RAG technology, hallucination in large language models is mitigated and the reliability of answers is improved. In particular, the autonomous learning mechanism through RLHF provides an environment in which the precision of answers can be enhanced in real time by continuously reflecting expert judgment criteria into the model.

[0024] According to the present invention, knowledge extraction optimized for each company's research environment becomes possible through a customized RAG search module that reflects private internal data. This minimizes the time and cost of information retrieval for practitioners and accelerates the internalization of technology in small and medium-sized enterprises by systematizing the knowledge acquisition process, which previously relied on highly specialized personnel. Users can directly perform the practical optimization process based on the guides provided by the system.

[0025] According to the present invention, a multimodal vector database schema that stores table and chart summaries and original data separately overcomes the limitations of text-based search systems and provides an innovative means of transforming complex visual data into knowledge. This construction methodology can be extended beyond the chemical industry to various industrial sectors possessing vast amounts of unstructured data, such as bio, cosmetics, finance, and manufacturing. Brief explanation of the drawing

[0027] FIG. 1 is a schematic block diagram of a multimodal response generation system according to one embodiment. FIG. 2 is a diagram illustrating an artificial intelligence model according to one embodiment. FIG. 3 is a flowchart of a multimodal response generation method according to one embodiment. FIG. 4 is a diagram illustrating the construction of a chemical corpus and a chemical-specific sLLM according to one embodiment. FIG. 5 is a diagram illustrating the construction of a multimodal vector DB according to one embodiment and the application of RAG technology accordingly. FIGS. 6a and 6b are drawings for explaining the operation of building a multimodal vector DB according to one embodiment. FIGS. 7 and FIGS. 8 are drawings for explaining the schema of data stored in a chemical database according to one embodiment. FIG. 9 is a diagram illustrating a multimodal response generation method according to one embodiment. FIG. 10 is a diagram illustrating a user profile vector integration step according to one embodiment. FIG. 11 is a diagram illustrating the step of obtaining a weight vector according to one embodiment. FIG. 12 is a diagram illustrating the RAG step and the response generation step according to one embodiment. FIG. 13 is a diagram illustrating the learning of a chemically specialized sLLM according to one embodiment. Specific details for implementing the invention

[0028] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0029] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0030] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or joined to that other component, or that there may be other components in between.

[0031] Singular expressions include plural expressions unless the context clearly indicates otherwise. In this document, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may each include any one of the items listed together with the corresponding phrase, or all possible combinations thereof. In this specification, terms such as “comprising” or “having” are intended to designate the existence of the described feature, number, step, action, component, part, or combination thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0032] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0033] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0034] As used in this document, the term "part" refers to a software or hardware component, such as an FPGA or ASIC, that performs certain roles. However, "part" is not limited to software or hardware. "Part" may be configured to reside in an addressable storage medium or configured to operate one or more processors. For example, "part" may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." Furthermore, components and "parts" may be implemented to operate one or more CPUs within a device or secure multimedia card. Additionally, '~part' may include one or more processors.

[0035] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0037] FIG. 1 is a schematic block diagram of a multimodal response generation system according to one embodiment.

[0038] Referring to FIG. 1, a multimodal response generation system (hereinafter referred to as System (10)) may be a system provided for recommending corporate-specific chemical formulations and process optimizations. Chemical formulations may refer to data including manufacturing process information such as the type, content, mixing order, and temperature conditions of a composition (e.g., main component and auxiliary component), and the final physical form (formulation). System (10) may be a system utilizing compound-specific sLLM (e.g., 230) and RAG technology.

[0039] The system (10) may include a user terminal (100) and an electronic device (200). The user terminal (100) and the electronic device (200) may be connected via a network (11) (e.g., a mobile radio communication network, a local area network (LAN), a wide area network (WAN), a value added network (VAN), a satellite communication network, or a combination thereof). The user terminal (100) and the electronic device (200) may communicate with each other via a wired communication method or a wireless communication method (e.g., Wi-Fi, Bluetooth, Bluetooth Low Energy, ZigBee, WiFi Direct (WFD), Ultra Wide Band (UWB), Infrared Data Association (IrDA), Near Field Communication (NFC)).

[0040] A user terminal (100) (e.g., a smartphone) may be a user terminal that seeks to obtain a response including the formulation of a chemical, predicted physical property values, and recommended process conditions. The user terminal (100) may include a memory (110) and a processor (120). The memory (110) may store various data used (or collected) by at least one component of the user terminal (100) (e.g., the processor (120)). The processor (120) (e.g., an application processor) may access the memory (110) to execute one or more instructions.

[0041] An electronic device (200) (e.g., a server) can generate information (e.g., a response) about a compound based on a user's input query and user profile information by utilizing sLLM (e.g., 230) and RAG technology. The electronic device (200) (e.g., a server) may include memory (210), a processor (220), and an artificial intelligence model (230). Memory (210) may store various data used (or collected) by at least one component of the electronic device (200) (e.g., a processor (220)). The processor (220) (e.g., an application processor) may access memory (210) to execute one or more instructions. The artificial intelligence model (230) may be stored in the electronic device (200) after training is completed. The artificial intelligence model (230) will be described in detail through FIG. 2.

[0043] FIG. 2 is a diagram illustrating an artificial intelligence model according to one embodiment.

[0044] In the present disclosure, the artificial intelligence model (230) may be a single artificial intelligence model or a plurality of artificial intelligence models. Anything referred to as a “model” in the present disclosure may be an artificial intelligence model.

[0045] Referring to FIG. 2, the artificial intelligence model (230) may include a plurality of artificial intelligence models (31, 232).

[0046] Artificial intelligence models can be composed of neural networks (or artificial neural networks) and may include statistical learning algorithms in machine learning and cognitive science that mimic biological neurons. A neural network can refer to a model that possesses problem-solving capabilities, where artificial neurons (nodes) forming a network through synaptic connections change the strength of these connections through learning. The neurons in a neural network may include combinations of weights or biases. A neural network may include one or more layers composed of one or more neurons or nodes. For example, a neural network may include an input layer, a hidden layer, and an output layer. By changing the weights of neurons through learning, a neural network can infer a desired output from an arbitrary input.

[0047] The electronic device (200) can generate a neural network, train or learn a neural network, perform operations based on received input data, generate an information signal based on the results of the operation, or retrain a neural network. The neural network models may include, but are not limited to, various types of models such as Convolutional Neural Network (CNN), Region with Convolutional Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restructured Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory Network (LSTM), Classification Network, etc. The electronic device (200) may include one or more processors (e.g., 220 of FIG. 1) for performing operations according to models of a neural network.

[0048] Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto) Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), GAN (Generative Adversarial Network), LSM (Liquid State Machine), ELM (Extreme Learning Machine), ESN (Echo It will be understood by a person skilled in the art that any neural network may be included, but is not limited to, State Network, Deep Residual Network, Differential Neural Computer, Neural Turning Machine, Capsule Network, Kohonen Network, and Attention Network.

[0049] According to an exemplary embodiment of the present disclosure, the electronic device (200) is a Convolutional Neural Network (CNN) such as GoogleNet, AlexNet, VGG Network, Region with Convolutional Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restructured Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory Network (LSTM), Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA for natural language processing, Text Analysis, Dialog System, GPT-3, GPT-4, GPT-5, Visual Analytics, Visual Understanding, Video Synthesis for vision processing, Anomaly Detection for ResNet data intelligence, Prediction, Various artificial intelligence structures and algorithms, such as time-series forecasting, optimization, recommendation, and data creation, may be used, but are not limited thereto.

[0051] FIG. 3 is a flowchart of a multimodal response generation method according to one embodiment.

[0052] In FIG. 3, steps S310 to S350 can be understood as being performed by the processor (220) of the electronic device (200).

[0053] In step S310, the electronic device (200) may receive user input queries and user profile information. User input queries may include requests for chemical formulations, predicted physical property values, and recommended process conditions. That is, user input queries may correspond to user input. User profile information may correspond to a company profile. User profile information may be collected to provide a response optimized for the company.

[0054] In step S320, the electronic device (200) can generate an extended input query by combining user profile information and an input query. Steps S310 and S320 will be explained in detail through FIG. 10.

[0055] In step S330, the electronic device (200) can obtain weight vectors for each of the multiple retrievers based on the extended input query. Step S330 will be explained in detail through FIG. 11.

[0056] In step S340, the electronic device (200) can extract at least one document chunk from a multimodal vector database using a plurality of search models to which weight vectors are applied.

[0057] In step S350, the electronic device (200) can generate a response including formulation, predicted physical property values, and recommended process conditions with supporting evidence based on document chunks and input queries using a chemically specialized sLLM (smaller Large Language Model). Steps S340 and S350 will be described in detail with reference to FIG. 12.

[0058] Before generating a multimodal response in response to a user request, preliminary steps such as building a multimodal vector database, establishing the foundation for RAG technology, and building a chemical-specific sLLM are required. Below, the preliminary steps will be explained first through Figures 4 to 8.

[0060] FIG. 4 is a diagram illustrating the construction of a chemical corpus and a chemical-specific sLLM according to one embodiment.

[0061] Referring to FIG. 4, the chemical-specific sLLM construction according to one embodiment can be broadly divided into a chemical corpus construction step (a) for acquiring data and a chemical-specific sLLM construction step (b) for generating an optimal model.

[0062] First, referring to Figure 4(a), in the chemical corpus construction stage, field-specific literature can be collected from authoritative academic literature related to chemicals, such as Springer, Elsevier, and the Royal Society of Chemistry (RSC), as well as from external sites. The collected data in various formats undergoes a purification and structuring process through a data preprocessing module, and can finally be constructed into a chemical corpus classified by detailed fields, such as plastics, household chemicals, and cement / concrete.

[0063] Next, referring to Fig. 4(b), in the chemical substance-specific sLLM construction step, a specific base model (Selected Base LLM) that meets the purpose can be selected from among a plurality of pre-trained base models (Base LLM 1, 2, 3, 4, etc.). Domain-specific learning is performed by inputting the previously constructed chemical substance corpus as training data into the selected base model. At this time, to maximize learning efficiency, various fine-tuning techniques such as LoRA (Low-Rank Adaptation) and Prefix-Tuning can be applied, in addition to the Full-Fine tuning method that learns all parameters.

[0064] Subsequently, the results of the applied learning techniques can be compared and analyzed through a fine-tuning selector to determine the model with the best performance as the final chemical-specialized sLLM. The chemical-specialized sLLM generated in this way can perform precise analysis and inference on complex chemical data by internalizing chemical domain knowledge beyond simple language generation.

[0065] Consequently, this process can serve as a foundation for providing advanced chemical formulation and property prediction services by refining fragmented external knowledge, converting it into specialized knowledge assets required by companies, and utilizing this for training optimal AI models.

[0067] FIG. 5 is a diagram illustrating the construction of a multimodal vector DB according to one embodiment and the application of RAG technology accordingly.

[0068] The electronic device (200) can then, as a preliminary step, 1) collect chemical-related literature to construct a multimodal vector database containing text, table summary and chart summary data, 2) extract and filter entity names from the metadata of the multimodal vector database to store industry-specific keyword candidates, 3) apply clustering to multiple extended input queries to form query clusters, and measure the response consistency of each of multiple search models (Retrievers) for the query clusters to calculate and store cluster-specific weight vectors.

[0069] Referring to Fig. 5, we will first describe the step of 1) collecting chemical-related literature to construct a multimodal vector database containing text, table summary, and chart summary data.

[0070] Referring to FIG. 5(a), the steps for constructing a multimodal vector database can be identified. The electronic device (200) may begin by collecting basic data from chemical-related sites and academic literature. The collected data is input into an object recognition module and can be identified and classified into detailed elements such as charts, text, and table images. The identified multimodal data is converted and refined into a searchable text form through a Multi-modal to text module and a data preprocessing module within a Multi-modal embedding model, and finally, it can be converted into numerical vector data through a text embedding process. The knowledge data thus converted can be stored and managed in a chemical information vector database classified by chemical domains, such as plastics, household chemicals, and cement / concrete.

[0071] Referring to Figure 5(b), the application step of RAG technology utilizing a multimodal vector database can be observed. The application step of RAG technology utilizing the constructed database may include a process of organically combining the acquired high-quality knowledge assets with search augmentation generative technology. When a user provides an input query regarding a specific chemical substance, the retriever can extract the knowledge chunk that is semantically most similar to the query within the multimodal vector database. The extracted relevant knowledge data can be transmitted to a chemical substance-specific sLLM along with the user's input query.

[0072] Finally, sLLM can generate a final response by performing inference based on actual literature evidence and database information provided by the search model. This structure prevents the hallucination phenomenon that occurs when relying solely on training data within the model, and can significantly improve the accuracy and reliability of the response by directly referencing the professional domain database built in (a) of Fig. 5. In other words, the precise database construction in step (a) can serve as an essential technical foundation for maximizing the performance of enterprise-customized RAG technology performed in step (b).

[0073] In Fig. 6, the construction of the multimodal vector database will be described further later, and the preliminary preparation step - 2) the step of extracting and filtering entity names from the metadata of the multimodal vector database to store keyword candidate sets by industry group - will be explained next.

[0074] The electronic device (200) can perform the step of storing a set of keyword candidates in advance as follows.

[0075] The electronic device (200) can extract entity names corresponding to raw materials, physical properties, and processes by applying an entity recognition model to metadata (e.g., metadata stored in a multimodal vector database) for text chunks, table summary text, and chart summary text. For example, when extracting entity names, the electronic device (200) can identify raw material entity names such as 'Polyethylene' and 'Sodium Hydroxide (NaOH)' based on IUPAC nomenclature or conventional chemical names, extract physical property units and items such as 'Viscosity', 'Tensile Strength', and 'Glass Transition Temperature (Tg)', and classify process parameters such as 'Agitation', 'Calcination', and 'Extrusion'. In this case, rather than simply extracting words from the text, the accuracy of named entity identification can be improved by utilizing structural positional information, such as whether the word is a column header of a table or an axis label of a chart, as a weight.

[0076] The electronic device (200) can store keyword candidates by filtering for named entities based on frequency of occurrence in metadata, literature distribution, and domain relevance. The electronic device (200) can select a final keyword candidate set by performing filtering based on the following quantitative / qualitative indicators.

[0077] First, in the case of the frequency of occurrence criterion, generality within the industry can be determined by checking whether a specific entity name (e.g., 'heat stabilizer') is found more than a specified threshold number of times (e.g., 100 times) within the entire collected dataset.

[0078] Second, regarding the literature distribution criterion, it is possible to analyze whether a specific physical property (e.g., 'chemical resistance') appears intensively in only a single paper or is commonly mentioned in multiple patents and research reports (e.g., more than 10 different sources). The more evenly distributed it is across multiple documents, the higher the likelihood that it will be adopted as a core keyword representing the relevant industry group.

[0079] Third, regarding the domain relevance criterion, to exclude general vocabulary such as 'result,' 'analysis,' and 'increase,' only terms with proven expertise, such as 'degree of polymerization' and 'crosslinking,' can be selectively stored in the keyword candidate pool by comparing them with the Chemical Technology Standard Glossary (Ontology) or existing internal technical document dictionaries.

[0080] The keyword candidate pool constructed through this multi-faceted filtering process can be utilized as foundational data to configure search modules optimized for user input queries or to enhance the search performance of customized RAG systems for each company.

[0081] Next, we will explain the preliminary preparation step - 3) the step of applying clustering to multiple extended input queries to form query clusters, and measuring the response consistency of each of the multiple search models (Retrievers) for the query clusters to calculate and store weight vectors for each cluster.

[0082] The electronic device (200) may have an input query clustering-based search model selection structure.

[0083] Specifically, the electronic device (200) can apply a K-means clustering algorithm to a set of extended query embeddings generated from a combination of multiple user profile information and user input queries. Through this, the electronic device (200) can divide queries with similar characteristics of specific industry group, query purpose, and keyword distribution into K clusters. Subsequently, the electronic device (200) can input the queries belonging to each cluster into a respective search model Ri and generate a response token distribution, which is a conditional probability distribution for which the LLM will generate a response y, based on the extracted document set Di. The response token distribution (Pi) can be calculated through Equation 1.

[0085] [Mathematical Formula 1]

[0086]

[0088] Next, the electronic device (200) can define a reference response distribution by calculating the arithmetic mean of response tokens for multiple search models. This reference distribution can provide a reference point that is not biased toward a specific search model even in real environments where there is no correct answer data through a zero-shot learning method. The reference response distribution (Q) can be calculated using Equation 2.

[0090] [Mathematical Formula 2]

[0091]

[0093] The electronic device (200) can quantify response consistency by measuring the KL-divergence between the response token distribution of each search model and the reference response distribution. A smaller KL-divergence value indicates that the results of the corresponding search model induce responses that are more consistent with the reference distribution, and based on this, the search model weight vector for the cluster It can be calculated. The weight vector can be calculated through Equation 3.

[0095] [Mathematical Formula 3]

[0096]

[0098] As disclosed in mathematical formula 3, the electronic device (200) can adjust the weight sensitivity according to the KL-divergence value by applying a temperature parameter (A).

[0099] The electronic device (200) can determine a weight vector based on a final objective function that includes an entropy regularization term to prevent excessive weight concentration on a specific search model. To prevent the phenomenon of excessive weight concentration on a specific search model in a zero-shot environment and to ensure the stability of the system, the electronic device (200) includes an entropy regularization term It may be that was introduced. Final objective function can be defined as the sum of the loss value of each search model and the regularization term (see Equation 4).

[0101] [Mathematical Formula 4]

[0102]

[0104] The electronic device (200) is the final objective function The weight vector can be finally determined and stored in a direction that minimizes [the value]. The cluster-specific weight vectors constructed through this process can be utilized to configure the optimal search model combination that matches the user's profile and query characteristics in real time during actual service.

[0106] FIGS. 6a and 6b are drawings for explaining the operation of building a multimodal vector DB according to one embodiment, and FIGS. 7 and 8 are drawings for explaining the schema of data stored in a multimodal vector DB according to one embodiment.

[0107] Referring to FIG. 6a, the electronic device (200) can convert unstructured data collected through various paths into high-dimensional vector data that a machine learning model can understand and search.

[0108] The electronic device (200) can collect documents in various formats, such as XML and PDF, from specialized literature related to chemical substances or chemical substance-related websites. The electronic device (200) can apply a dual collection method combining keyword-based web scraping and semantic-based literature verification to secure literature specialized in the chemical substance domain. At this time, the keywords used for collection can be set based on main materials related to the chemical substance domain; for example, when collecting documents related to the concrete domain, keywords such as cement, sand, clinker, limestone, gypsum, and steel may be used. The electronic device (200) collects literature by entering main related keywords from Google Patent and arXiv, etc., and can perform a method of searching for the same document by combining 'keyword AND domain name' and a method of determining the meaning of the title and abstract through LLM. In particular, when determining the meaning, the electronic device (200) can control the domain-relatedness of the text by entering a specific prompt into an LLM set as a chemical and materials science expert to determine it as True or False. Subsequently, the electronic device (200) determines whether the documents are identical based on the patent number or arXiv ID, and can secure the reliability of the collected data by confirming only the documents derived in common from the two methods as the final domain-related documents.

[0109] The documents collected in this way can be analyzed through a 'pre-trained document object recognition model' included in the electronic device (200), and the electronic device (200) can classify the information within the documents into three types—charts, text, and table images—according to their characteristics and process them through different paths.

[0110] The electronic device (200) can input chart elements among the classified data into a 'pre-trained Chart-to-Text model' to convert them into 'chart content summary' text that describes the visual data. Additionally, the electronic device (200) can input a table image into an 'Attention-based table structure recognition model' and a 'pre-trained Table-to-Text model' to identify the structure of the table and generate 'table content summary' information. At the same time, the electronic device (200) can extract general text data.

[0111] The electronic device (200) can perform 'text preprocessing and chunking' by integrating extracted plain text and summary text generated from charts and tables. Subsequently, the electronic device (200) converts the data into numerical high-dimensional vectors through a 'text embedding' model and can finally store them in a 'multimodal vector DB'. Based on the vector DB thus constructed, the electronic device (200) can provide contextual and professional chemical information search results in response to a user's query.

[0112] Referring to FIG. 6b, the detailed configuration of an attention-based table structure recognition model can be seen. The electronic device (200) can receive table data in the form of an image and perform a series of encoding and decoding processes to convert it into a digital structure.

[0113] The electronic device (200) can transmit the input table image to a 'CNN Backbone Encoder' to extract features, and in this process, the main features of the image can be obtained by passing through multiple ResNet Blocks and Spatial Attention modules. 'Positional embedding' to maintain the spatial order of visual information can be combined with the extracted feature data.

[0114] The electronic device (200) can input feature data combined with location information into a 'Transformer encoder' to globally learn the correlation between components within the table. Subsequently, the electronic device (200) can process the encoded data using a 'Transformer decoder', and at this time, the data can be split into two branches to perform parallel decoding. Each decoder branch can repeat the Cross Attention, Add & Norm, and FFN steps.

[0115] The electronic device (200) can generate data for different purposes through branched decoders. Through the upper branch, it can generate 'HTML Structure Tags' that define the logical framework of a table, such as , etc., and through the lower branch, it can derive 'Bounding boxes location' which are coordinate values ​​of the area where individual text or components within the table are located. As a result, the electronic device (200) can obtain highly refined table data by combining the visual location information of the table image with the structural tag information.

[0116] Not only that, the electronic device (200) can perform a series of processes to structure table data and select data that is of actual research value.

[0117] First, the electronic device (200) can recognize a table area within a document using a document layout recognition model and convert the table area into a markdown format. Specifically, the electronic device (200) can receive a document collected in PDF format or the like and separate text areas, table areas, and image areas within the document using an OCR-based document layout recognition model (e.g., DeepSeek-OCR). At this time, the electronic device (200) converts the text and table into a markdown format and returns them, and the image data can be saved and managed as a separate file.

[0118] The electronic device (200) can designate an area containing a predetermined tag within a Markdown format as a table candidate and extract a caption for the table within a predetermined line before or after the table candidate. Specifically, the electronic device (200) within a converted Markdown document

[0119]

[0120]

[0121]

[0122]

[0123]

[0124]

[0125]

[0126]

[0127]

[0128]

[0129]

[0130]

[0131]

[0132]

[0133]

[0134]

[0136]

[0137]

[0138]

[0140]

[0141]

[0142]

[0143]

[0144]

[0145]

[0147]

[0148]

[0149]

[0150]

[0152]

[0153]

[0154]

[0155]

[0156]

[0157]

[0158]

[0159]

[0161]

[0162]

[0163]

[0164]

[0165]

[0167]

[0168]

[0169]

[0170]

[0171]

[0172] , An area containing a predetermined tag, such as the table, can be designated as a table candidate, and a table caption can be extracted by searching for a sentence containing the keyword 'table_caption' within a predetermined line in front of or behind the area. The electronic device (200) generates table-caption pairs by matching the caption with the table, but can exclude tables without captions by classifying them as administrative or bibliographic data. For example, the electronic device (200) determines that a subsequent caption existing within 4 lines after the appearance of the table is valid, and tables without captions can be excluded from analysis by classifying them as administrative or bibliographic data rather than research data. Subsequently, the electronic device (200) can match the extracted table and caption and store them in a metadata format such as 701 in FIG. 7. The electronic device (200) can separate the header area and the data row from the table-caption pair and reconstruct them into semantic-based serialized text. Referring to 702 in FIG. 7, the electronic device (200) can create an environment in which a Large Language Model (LM) can more accurately grasp the structure and content of a table by converting the existing complex Markdown format into a serialized text format distinguished by 'CAPTION', 'COLUMNS', 'ROW n', etc. In this process, the electronic device (200) can ensure data integrity by utilizing the 'Attention-based table structure recognition model' illustrated in FIG. 6b to precisely match the visual position information of the table with HTML structure tags. The electronic device (200) can generate a summary by extracting key scientific information from the serialized text through a first LLM specialized in content summarization. At this time, the electronic device (200) can control the extraction of the core meaning of the table based on relevant domain knowledge, such as polymer and plastic materials, by providing a prompt set as a materials scientist to the first LLM.Next, the electronic device (200) can determine whether the table corresponds to actual research data, including experimental results, physical property values, or composition ratios, by analyzing the summary through a second LLM specialized in understanding text meaning. Finally, the electronic device (200) can prevent unnecessary token consumption and effectively exclude tables containing simple numerical listings or equipment specifications by performing the summary step and the meaning determination step separately. The electronic device (200) can build a high-quality knowledge base specialized in chemical domains by selecting only the final table data that can be utilized in actual research and development based on the determination result and storing it in a vector database (DB), etc. As part of the process of building a multimodal vector database, the electronic device (200) can perform a chart data structuring and sequential matching process. First, the electronic device (200) can perform a step of identifying images within a document through a document layout recognition model, and sequentially assigning unique identifiers (e.g., image_1, image_2, etc.) according to the order of appearance within the document and storing them in a directory. At this time, the OCR-based document layout recognition model can accurately identify the sequential relationship of how images are visually arranged in the document and match them with sequential identifiers to systematically record them in the image storage directory. Subsequently, the electronic device (200) can perform the step of generating image caption candidates by extracting sentences containing a specific keyword (e.g., image_caption) from the document converted into Markdown format. The electronic device (200) can proceed with the step of generating image-caption pairs by sequentially matching the extracted unique identifiers and the image caption candidate group. This utilizes the characteristic that images and image captions correspond to each other based on the order of appearance within the document, and the electronic device (200) can automatically generate pairs between images and captions through this.The electronic device (200) can perform a filtering step by analyzing the content of the image caption through an LLM to determine whether the image is related to the chart of the experimental results. Through this process, the electronic device (200) can exclude simple illustrations unrelated to the experimental results and select only chart-related image-caption pairs that have value as actual research data, and store them in a metadata format. Next, the electronic device (200) can perform a step of converting the visual information of the chart into summary text by applying a Chart-to-Text generation model to the image-caption pairs determined to be related to the chart. The summary text generated at this time includes specific numerical values ​​or trends of the chart and can be managed in combination with source information, etc. Finally, the electronic device (200) can perform a step of embedding the summary text and storing it in a vector space, and storing the original data of the chart corresponding to the summary text in individual fields of the metadata. Through this, the electronic device (200) can vectorize the visual meaning of the chart to provide an environment for efficient searching, and at the same time, complete a data structure that allows for immediate reference to the original chart data when necessary. Referring to Fig. 8, an example of the schema of data stored in a multimodal vector DB can be seen. The electronic device (200) may have constructed a multimodal vector database that integrates and stores text, table, and chart data within a document so that they can be searched in the same semantic space. In this case, for search efficiency, the electronic device (200) may embed only the summary data of the text, table, and chart and store it in a vector space, while the original data (Raw Data) of the table and chart may be stored separately in individual fields of metadata. However, it does not stop there, and the summary data of the text, table, and chart may also be stored in metadata.The electronic device (200) can divide the final selected table data into text chunks having a predetermined size, and can vectorize each chunk through a sentence embedding model. The electronic device (200) manages each data within the vector DB in chunk units divided by semantic units, and while the raw data of the table data and chart data is stored separately in the metadata area, summary content can be stored in the main data area of ​​the vector DB. The vector DB constructed by the electronic device (200) may have an integrated schema such as 801 in FIG. 8. Specifically, each chunk may have a unique vector identifier (vectorDB_id) that combines the original document identifier, data type, and chunk sequence number. The unique identifier may have a format such as {document_id}text{chunk_id}, {document_id}title{chunk_id}, {document_id}table{chunk_id}, or {document_id}chart{chunk_id}. Additionally, each data item may include actual text content (document) and a corresponding embedding value. The electronic device (200) may separately configure a metadata field corresponding to the chunk unit data. The metadata may include a document identifier (source_id), the nature of the data (section; e.g., abstract, title, chart_summary, table_summary, table_raw, etc.), a chunk index (chunk_index), a document source type (source; e.g., patent, paper), and chemical domain information (domain). Through this metadata configuration, the electronic device (200) can perform a precise search on multimodal chemical data and process semantic-based queries specialized for chemical domains.FIG. 9 is a diagram illustrating a method for generating a multimodal response according to one embodiment. Referring to FIG. 9, an electronic device (200) may receive and generate a response (903) including the formulation of a chemical, predicted physical property values, and recommended process conditions based on a user's input query (901) and user profile information (902). In this process, the electronic device (200) may utilize a chemical-specific sLLM (910) and an enterprise-customized RAG module (920). The electronic device (200) may generate an 'extended input query containing profile information' by organically combining the user's input query and detailed user profile information to extract specific information. For example, if a user inputs a query such as 'recommendation for a formulation with a Flexural Strength of 100 MPa or more,' the electronic device (200) may reflect the industry group (plastic) and specific purpose of use (formulation optimization and physical property prediction for new product development) set in the profile into the query. At this time, the electronic device (200) can enhance the expertise of the query by merging keywords of interest that include raw materials such as Polyetherimide, physical property information such as Flexural Modulus, and process conditions such as compound time and Temperature. Finally, the electronic device (200) can obtain base data for deep semantic analysis beyond simple text matching by inputting the generated extended input query into a pre-trained sentence embedding model and converting it into a high-dimensional query embedding vector. This will be explained in more detail in FIG. 10. FIG. 10 is a diagram illustrating the user profile vector integration step according to one embodiment. Referring to FIG. 10, steps S1010 to S1040 may be performed by the processor (220) of the electronic device (200). In step S1010, the electronic device (200) can convert industry group information included in the user profile information into an industry group embedding vector.The electronic device (200) can define the domain context for subsequent search and analysis by mapping a specific industrial field selected by the user (e.g., secondary batteries, specialty chemicals, construction materials, etc.) into a high-dimensional vector space. For example, if the user selects the 'concrete domain', the electronic device (200) can generate an embedding vector having numerical features specialized for that industrial group and use it as basic data to narrow down the range of keyword candidates to be provided in subsequent steps. In step S1020, the electronic device (200) can convert the sentence describing the purpose of use included in the user profile information into a purpose semantic vector. The electronic device (200) can semantically analyze the research purpose or data utilization intention entered by the user in a free sentence format (e.g., "extraction of experimental data for the development of high-heat-resistant polymer additives") through natural language processing technology. Through this, the electronic device (200) can identify the user's actual research intention as a quantified vector beyond simple keyword search, and this can be a key element in reflecting the user's specific needs when constructing the final user profile vector. In step S1030, the electronic device (200) can generate a single keyword embedding vector by applying one of average pooling, importance-weighted average, or attention-based integration to multiple interest keywords selected by the user. The electronic device (200) provides the user with interest keywords classified by raw material, physical property, and process tags, and the keyword candidate group may be constructed based on text, table, and chart summary metadata generated in advance when building the vector DB. As described above, the electronic device (200) can apply a named entity recognition model (e.g., materialBERT) to automatically extract named entities such as 'cement' (raw material), 'compressive strength' (physical property), 'hydration' (process), etc., and store them as a candidate group by filtering based on frequency of occurrence or domain suitability.When a user selects multiple keywords from among these candidate groups, the electronic device (200) can convert them into a single interest keyword embedding vector through a method such as attention-based integration that reflects the relationship between the keywords. In step S1040, the electronic device (200) can integrate the industry group embedding vector, the purpose semantic vector, and the single keyword embedding vector into a user profile vector. The electronic device (200) can complete each vector generated in the preceding steps into a single integrated user profile vector through concatenation, linear combination, or attention combination methods. For example, this vector, in which a specific researcher's field of expertise, current research purpose, and interest keyword information are all fused, can be provided as an input to the 'Retriever Selector' along with the input query embedding. Consequently, the electronic device (200) can perform a personalized search function that prioritizes searching for and selecting research data (experimental result charts, physical property tables, etc.) most optimized for the researcher within the constructed multimodal vector DB. FIG. 11 is a diagram illustrating the weight vector acquisition step according to one embodiment. Referring to FIG. 11, steps S1110 to S1120 may be performed by the processor (220) of the electronic device (200). In step S1110, the electronic device (200) can map the embedding of the extended input query to a specific cluster among the query clusters formed in the preliminary preparation step that is most similar in terms of industry group, query purpose, and keyword distribution characteristics. The extended input query embedding is mapped to a cluster with the most similar vector distance within a pre-learned query cluster space, and the electronic device (200) can identify the professional context implied by the query through this.For example, when a complex query asking about changes in molecular weight according to experimental conditions of a specific polymer material is input, the electronic device (200) can map the query to a specific query cluster dense with expertise of the type "polymer property prediction and formulation optimization." In step S1120, the electronic device (200) can call a weight vector for the mapped cluster. The called weight vector represents the performance contribution of each search model calculated in advance for the cluster, and the electronic device (200) can maximize search accuracy by selecting the search model with the highest weight as the primary search model. For example, if a Fine-tuned Contriever-based search model advantageous for semantic search according to a pre-set weight vector has the highest weight, a Recency-based search model for reflecting the latest literature has an auxiliary weight, and a BM25-based search model for keyword matching has a relatively low weight, the electronic device (200) can derive a final search result by utilizing the Fine-tuned Contriever as the primary searcher and reflecting the remaining search models in the mixed search with a low weight. FIG. 12 is a diagram illustrating the RAG step and the response generation step according to one embodiment. Referring to Step 1 of FIG. 12, the electronic device (200) can extract at least one document chunk from a multimodal vector database using a plurality of search models to which weight vectors are applied. The multimodal vector DB searched by the electronic device (200) may store not only text chunks containing the body of a paper or patent description, but also table summary text such as Flexural Strength measurement results and chart summary text including Stress-Strain Curve or Flexural Test result graphs.The selected search model searches for vectors with high similarity to the input query embeddings and can produce results such as a summary of experimental result tables summarizing Polyetherimide-Polycarbonate blend composition ratios and Flexural Strength, or a summary of charts explaining changes in physical properties according to process temperature and compounding conditions. The searched documents can be reordered based on the degree of match with raw materials, physical properties, process tags, etc., set in the user profile, and the electronic device (200) can finally select the top N document chunks, such as a paragraph describing a case where Flexural Strength of 100 MPa or more was achieved in a specific formulation, and use them as input for Retrieval-Augmented Generation (RAG). Referring to Step 2 of FIG. 12, the electronic device (200) can use a chemical-specific Smaller Large Language Model (sLLM) to generate a response including formulation, predicted physical property values, and recommended process conditions, along with supporting evidence, based on the document chunks and input queries. The electronic device (200) combines the searched sentences and input queries to generate a prompt for the chemical It is input into a specialized sLLM, and the output of the sLLM passes through a formulation optimization downstream task layer to be derived as a concrete result. Accordingly, the system can generate a final response including five plastic formulation combinations satisfying the Flexural Strength 100 MPa condition, predicted physical property values ​​for each combination, and optimal recommended process conditions. At this time, the electronic device (200) may provide referenced experimental table summary data, chart-based analysis of physical property changes, and literature sources for similar cases as grounds for recommendation to increase the reliability of the response. FIG. 13 is a diagram illustrating the learning of a chemical-specialized sLLM according to one embodiment.The electronic device (200) can update the model based on expert feedback to maintain the quality of the sLLM's answers. The update may be performed when at least one of the following conditions is met: when new feedback data has accumulated more than a preset number, when the negative response rate in the user satisfaction evaluation is greater than a preset percentage, or when a domain imbalance is detected due to quality degradation in a specific chemical domain. The electronic device (200) can build a 'Performance Dataset' for sLLM enhancement based on data collected from chemical industry professionals and chemical QA documents, as illustrated in FIG. 13. The electronic device (200) can collect expert preference feedback on multiple answer candidate groups generated by the sLLM, and the evaluation may be performed based on four criteria: domain suitability, response completeness, consistency of evidence, and industrial applicability. For example, the electronic device (200) can ensure the quality of the dataset by determining whether the answer includes, without omission, key information that satisfies the intent of the question, using specialized terminology specific to the chemical domain context rather than being limited to a general explanation. The electronic device (200) can update the weights of the sLLM by incorporating the collected expert feedback into the learning of the 'Reward Model' and using it again as input to the 'Policy Optimization' stage. Through this process, the electronic device (200) can control the model so that the expert's expertise and judgment criteria are internalized into the model's response generation logic. For example, by setting the answer to receive a higher reward than an answer that explains the chemical mechanism in detail and consistently rather than an answer with a contradictory logical flow of changes in physical properties according to a specific mixing ratio, the electronic device (200) can advance the sLLM to provide progressively more reliable research guidance.The electronic device (200) can perform automatic updates of the model when one or more of the preset trigger conditions are met. Specifically, the electronic device (200) can perform updates by determining that the data accumulation condition is met when 100 or more newly accumulated feedback data are accumulated. Additionally, the electronic device (200) can activate the model update process when performance degradation is detected, such that negative opinions (Bad) are tallied at 50% or more in the real-time user satisfaction evaluation results. Furthermore, the electronic device (200) can maintain the information reliability of the entire system by performing selective updates to maintain performance balance in the relevant domain when a situation is detected where response quality degradation is observed only for a specific chemical domain according to the domain imbalance detection condition. The collection device (e.g., user terminal, electronic device) according to the embodiments disclosed in this document may be of various forms. The collection device may include, for example, a portable communication device (e.g., smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The collection device according to the embodiments of this document is not limited to the devices described above. The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In relation to the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise.In this document, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationly,” it means that said component may be connected to said other component directly (e.g., by wire), wirelessly, or through a third component. As used in some embodiments of this document, the term “module” may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC). One embodiment of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., internal memory or external memory) that can be read by a machine (e.g., an electronic device).For example, a processor of a device (e.g., electronic device) may call at least one of one or more instructions stored from a storage medium and execute it. This enables the device to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium readable by a device may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium. According to one embodiment, the method according to the embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. A computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. According to one embodiment, each component (e.g., a module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components.According to one embodiment, one or more of the aforementioned components or operations may be omitted, or one or more other components or operations may be added. Generally or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components in the same or similar manner as those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

Claim 1 A method of operation for an electronic device that provides information specialized in a chemical domain comprises: receiving a user input query and user profile information; generating an extended input query by combining the user profile information and the input query; obtaining a weight vector for each of a plurality of search models (Retrievers) based on the extended input query; extracting at least one document chunk from a multimodal vector database using the plurality of search models to which the weight vector is applied; and generating a response including formulation, predicted physical property values, and recommended process conditions along with the basis, based on the document chunk and the input query, using a chemical-specialized sLLM (Smaller Large Language Model). The step of generating the extended input query comprises: converting industry group information included in the user profile information into an industry group embedding vector; converting a sentence describing the purpose of use included in the user profile information into a purpose semantic vector; and generating a single keyword embedding vector by applying one of average pooling, importance-weighted average, or attention-based integration to a plurality of interest keywords selected by the user. A method comprising the step of integrating the industry group embedding vector, the purpose semantic vector, and the single keyword embedding vector into a user profile vector based on the above industry group embedding vector. Claim 2 A method according to claim 1, further comprising, as a preliminary preparation step, a step of constructing the multimodal vector database including text, table summary, and chart summary data by collecting chemical substance-related literature; a step of storing keyword candidate sets by industry group by extracting and filtering entity names from the metadata of the multimodal vector database; a step of forming query clusters by applying clustering to a plurality of extended input queries; and a step of calculating and storing cluster-specific weight vectors by measuring the response consistency of each of a plurality of search models (Retrievers) for the query clusters. Claim 3 In paragraph 2, the step of storing the keyword candidate group in advance comprises: a step of extracting entity names corresponding to raw materials, physical properties, and processes by applying an entity recognition model to the metadata for text chunks, table summary text, and chart summary text; and a step of storing the entity names as the keyword candidate group by filtering them based on the frequency of occurrence, literature distribution, and domain fit within the metadata. Claim 4 delete Claim 5 A method according to claim 2, wherein the step of obtaining the weight vector comprises: a step of mapping the embedding of the extended input query to a specific cluster among the query clusters formed in the preliminary preparation step that is most similar in terms of industry group, query purpose, and keyword distribution characteristics; and a step of calling the weight vector for the mapped cluster. Claim 6 A method according to claim 2, wherein the step of calculating and storing the weight vector for each cluster comprises: a step of defining a reference response distribution by calculating the arithmetic mean of response tokens for each of the plurality of search models; a step of quantifying response consistency by measuring the KL-divergence between the response token distribution of each search model and the reference response distribution; a step of adjusting the weight sensitivity according to the KL-divergence value by applying a temperature parameter; and a step of determining the weight vector based on a final objective function including an entropy regularization term to prevent excessive concentration of weights on a specific search model. Claim 7 A method according to paragraph 2, wherein the multimodal vector database stores text, table, and chart data within a document in an integrated manner so that they can be searched in the same semantic space, embeds only summary data of the text, table, and chart in a vector space for search efficiency, and stores the raw data of the table and chart separately in individual fields of metadata. Claim 8 In claim 2, the step of constructing the multimodal vector database comprises: recognizing a table area within a document using a document layout recognition model and converting the table area into a markdown format; designating an area containing a predetermined tag within the markdown format as a table candidate and extracting a caption of the table within a predetermined line before or after the table candidate; generating a table-caption pair by matching the caption with the table, wherein a table without a caption is classified as administrative or bibliographic data and excluded; separating a header area and a data row from the table-caption pair and reconstructing them into semantic-based serialized text; generating a summary by extracting core scientific information from the serialized text through a first LLM specialized for content summarization; and analyzing the summary through a second LLM specialized for text semantics to determine whether the table corresponds to research data including experimental results, physical property values, or composition ratios, and finally storing the result. Claim 9 In claim 2, the step of constructing the multimodal vector database comprises: identifying images within a document through a document layout recognition model and sequentially assigning unique identifiers according to the order of appearance within the document and storing them in a directory; generating image caption candidates by extracting sentences containing predetermined keywords from a document converted into Markdown format; generating image-caption pairs by sequentially matching the unique identifiers and the image caption candidate group; analyzing the content of the image captions through LLM to determine whether the images are related to a chart of experimental results and filtering them; applying a Chart-to-Text generation model to image-caption pairs determined to be related to the chart to convert the visual information of the chart into summary text; and embedding the summary text and storing it in a vector space, and storing the original data of the chart corresponding to the summary text in individual fields of the metadata. Claim 10 A method according to claim 1, further comprising the step of updating a model based on expert feedback to maintain the answer quality of the sLLM, wherein the update is performed when at least one of the following conditions is satisfied: when new feedback data has accumulated more than a preset number, when the negative response rate in the user satisfaction evaluation is greater than a preset percentage, or when a domain imbalance is detected by detecting a quality degradation in a specific chemical domain.

Images