Multimodal search and response generation device for design evaluation

Abstract

The present invention relates to a multimodal search and response generation device for design evaluation that can improve design decision-making and evaluation efficiency by receiving design data of various formats, querying similar references, and automatically generating explanation, analysis, and comparison results corresponding to queries through a design search and response generation device that fuses artificial intelligence (AI), vector-based search (Vector Retrieval), large-scale multimodal model (LMM), and RAG (Retrieval-Augmented Generation) technologies. The present invention comprises: a design evaluation AI processing device (210) that embeds LMM-based multimodal input data to perform a search on a vector database (100) where multimodal design data is stored, and generates a semantically interpreted natural language response based on the search results; and a visualization processing device (220) that visualizes the data processed by the design evaluation AI processing device (210) and outputs it to a dashboard (400), which is a device that visualizes and displays the response results generated by the design evaluation AI processing device (210). A multimodal search and response generation device for design evaluation is provided, characterized by comprising: a system operating device (230) for handling log storage, authorization management, learning data update, and distribution management for a user terminal device (300); an internal database (240) for storing structured and unstructured data; and a data management device (250) which is a central data hub device for storing, searching, linking, and transmitting various types of design data.

Description

Technology Field

[0001] The present invention relates to a system for analyzing design elements and quantitatively evaluating them, and more specifically, to a multimodal search and response generation device for design evaluation that searches for multimodal design data of different forms, such as images, text, and 3D objects, and automatically generates responses to queries and design comparison and analysis results based on a large-scale multimodal language model (LMM). Background Technology

[0003] Currently, design evaluations are predominantly subjective, and there is a lack of evaluation systems based on objective data. Furthermore, it is necessary to establish an automated process that quantitatively analyzes and evaluates design elements using AI-based evaluation technology. Additionally, cloud-based SaaS (Software as a Service) solutions should be implemented to enable companies and designers to utilize evaluation functions anytime and anywhere.

[0004] Recently, leading global AI companies such as OpenAI and Google DeepMind have been developing AI models applicable to design evaluation; however, due to insufficient development of AI technology for design evaluation in Korea, it is difficult to secure global competitiveness.

[0005] The use of AI within the design industry is increasing. As AI technology is utilized in various sectors such as healthcare, manufacturing, and finance, innovation through AI is also required in the design industry. Therefore, it is necessary to strengthen the competitiveness of the design industry and contribute to productivity improvement through the development of AI-based design evaluation solutions.

[0006] Meanwhile, the government is promoting the digital transformation of the design industry and actively supporting the development of design evaluation systems utilizing AI technology; however, existing design evaluation methods have faced difficulties in consistent evaluation due to a lack of quantitative data.

[0007] Therefore, it is necessary to improve the reliability and accuracy of design evaluation by developing quantitative interpretation and evaluation technologies utilizing AI-based learning engines.

[0008] In addition, there is a need for a method to improve the efficiency of design decision-making and evaluation by receiving design data in various formats, querying similar references, and automatically generating explanation, analysis, and comparison results in response to queries. Prior art literature

[0010] Patent Document 1: Korean Published Patent No. 10-2009-0090613 (Published August 26, 2009) - Multimodal Interactive Image Management System and Method Patent Document 2: Korean Published Patent No. 10-2014-0093957 (Published July 29, 2014) - Interactive Multi-modal Image Search Technique The problem to be solved

[0011] Accordingly, the present invention aims to solve the problems of the prior art as described above by providing a multimodal search and response generation device for design evaluation that can improve design decision-making and evaluation efficiency by receiving design data of various formats, querying similar references, and automatically generating explanation, analysis, and comparison results corresponding to queries through a design search and response generation device that integrates artificial intelligence (AI), vector-based search (Vector Retrieval), large-scale multimodal model (LMM), and Retrieval-Augmented Generation (RAG) technologies. means of solving the problem

[0013] To achieve the above objective, the present invention provides a multimodal search and response generation device for design evaluation, characterized by comprising: a design evaluation AI processing device (210) that embeds LMM-based multimodal input data to perform a search on a vector database (100) where multimodal design data is stored and generates a semantically interpreted natural language response based on the search results; a visualization processing device (220) that visualizes and displays the response results generated by the design evaluation AI processing device (210) and outputs the data processed by the design evaluation AI processing device (210) to a dashboard (400); a system operating device (230) that handles log storage, authorization management, training data update, and distribution management for a user terminal device (300); an internal database (240) where structured data and unstructured data are stored; and a data management device (250) which is a central data hub device that stores, searches, links, and transmits various types of design data.

[0014] Here, the design evaluation AI processing unit (210) is characterized by including an embedding model that converts different design data, including images, text, 2D and 3D models, and UI captures, into a single embedding space.

[0015] And the design evaluation AI processing unit (210) is characterized by generating a quantitative evaluation result for the search result using a Score Head-based weighted sum score, a Feature Similarity-based similarity score, a Regression-based prediction score, or a structured / unstructured Score Fusion method. Effects of the invention

[0017] The present invention has the following effects.

[0018] First, by mapping heterogeneous design data such as images, 3D, text, and UI screens into a single embedding space, similar designs can be searched more precisely compared to existing keyword / tag-based searches.

[0019] Second, based on a Vector DB, it can quickly provide design candidates that match the intent.

[0020] Third, since the LMM automatically generates summarized, analyzed, compared, and evaluated responses from search results rather than a simple list, it is possible to identify design differences and strengths and weaknesses.

[0021] Fourth, search response speed and quality are improved, which can significantly reduce the time required for initial decision-making in planning and design. Brief explanation of the drawing

[0023] FIG. 1 is a block diagram illustrating a generative AI design evaluation system including a multimodal search and response generation device for design evaluation according to the present invention. FIG. 2 is a drawing for explaining an embodiment of a multimodal search and response generation device for design evaluation according to the present invention, Figure 3 is an example UI of Shape Grammar Interpreter, a design element interpretation and automatic design generation program based on Shape Grammar, which is one of the design element interpretation and evaluation methodologies. Figure 4 is an example of a Sortal GI-based Shape Grammar Interpreter UI. FIG. 5 is a diagram showing the result of the Sortal GI-based Shape Grammar Interpreter shown in FIG. 4. FIG. 6 is an embodiment showing an LMM-based design analysis and evaluation workflow used in a multimodal search and response generation device for design evaluation according to the present invention. FIG. 7 is a diagram showing the cloud-based evaluation environment and real-time analysis technology used in the present invention. Specific details for implementing the invention

[0024] A preferred embodiment of the present invention will be described in detail with reference to the attached drawings as follows.

[0025] Furthermore, while the terms used in this invention have been selected to be as widely used as possible, there are also terms arbitrarily chosen by the applicant in specific cases. Since the meanings of these terms are described in detail in the relevant description of the invention, it should be noted that the invention should be understood based on the meaning of the terms rather than their mere names. Additionally, in describing the embodiments, explanations regarding technical content that is well known in the technical field to which this invention belongs and is not directly related to this invention are omitted. This is intended to convey the essence of the invention more clearly without obscuring it by omitting unnecessary explanations.

[0027] FIG. 1 is a diagram illustrating a generative AI design evaluation system including a multimodal search and response generation device for design evaluation according to the present invention.

[0028] A generative AI design evaluation system including a multimodal search and response generation device for design evaluation according to the present invention collects seven major design evaluation elements and data as shown in FIG. 1.

[0029] These seven major design evaluation elements and data are product design, visual design, digital / multimedia design, spatial design, fashion / textile design, service / experience design, and industrial craft design.

[0030] These seven major design evaluation elements are collected as evaluation datasets for each design element, and are classified into structured data and unstructured data.

[0031] Meanwhile, the collected evaluation datasets by design element are stored in the vector DB and evaluation data DB, and processed into DataOps (Data Operations).

[0032] In addition, through the 7 major design evaluation LMMs, an expandable model capable of interpreting language, visuals, structure, and behavior is generated through LMMs (Large Multimodal Models) for product design, visual design, digital / multimedia design, spatial design, fashion / textile design, service / experience design, and industrial craft design. By including an LMM-based Design RAG (Retrieval-Augmented Generation for Design) structure to search for references similar to the input design and generate evaluation scores, similar characteristics, and improvement points in natural language, a generative AI system for design evaluation and improvement is implemented. Furthermore, by including an LMMOps (LMM Operations)-based operating system, a multimodal operating environment is provided in which the model is continuously improved by connecting the results of design data collection, embedding generation, similarity search (Vector Search), and the 7 major evaluation Score Heads to an automatic learning loop.

[0033] The generative AI design evaluation system used in the LMM-based multimodal design element analysis and quantitative evaluation system according to the present invention designs an analysis and evaluation model for input design information (images, data, etc.) using an LMM-based design analysis and evaluation model, and designs the entire design evaluation information storage and analysis system and the design element analysis and evaluation platform through a method of linking SGI (Shape Grammar Interpreter) technology.

[0034] Through this, design information is visually expressed, and a responsive interface is designed for modifying and utilizing it. It is designed to provide automatic feedback on design evaluations using generative AI through an interactive evaluation environment based on expert evaluation methods and an evaluation environment that reflects expert evaluation methods, based on UX / UI design and responsive interface concepts.

[0035] In addition, a method and model for processing design source-work result learning data were designed, and research on constructing and preprocessing the design source-work result dataset was conducted, along with continuous data learning and update DataOps regarding database linkage and internal data utilization methods for Sub-detail 1.

[0037] FIG. 2 is a drawing for explaining an embodiment of a multimodal search and response generation device for design evaluation according to the present invention.

[0038] An embodiment of the multimodal search and response generation device for design evaluation according to the present invention is composed of a vector database (100), a multimodal search and response generation device for design evaluation (200), a user terminal device (300), and a dashboard (400), as shown in FIG. 2.

[0039] Here, the vector database (100) may consist of 3D images, 2D images, text documents, and image training datasets. In addition, multimodal design data is stored that can simultaneously process and integrate different forms of input data, such as images, 2D graphics, UI screenshots, 3D models, text descriptions, videos, design-related attribute metadata, and user interaction records.

[0040] Here, the dataset includes definitions of source-result relationships by design variation type (color changes, layout adjustments, etc.), source-result data collection (design tool change history, CAD data, etc.), and design of standard data formats and storage structures (including text, 2D, and 3D).

[0041] For reference, a vector database stores “meanings and features” such as images, text, 3D models, and voice in the form of numeric vectors and quickly finds the data most similar to the input. While existing SQL DBs search by word = same / exclude, Vector DBs have the advantage of being able to search by calculating even “how similar” they are.

[0042] The multimodal search and response generation device (200) for design evaluation consists of a design evaluation AI processing device (210), a visualization processing device (220), a system operation device (230), an internal database (240), and a data management device (250).

[0043] The design evaluation AI processing unit (210) provides a core processing engine for design search, analysis, and response generation that embeds multimodal input data to perform similarity search based on Vector DB, combines the searched references with LMM to generate a response to a query, and returns a quantitative evaluation score and comparative analysis results in the form of natural language. At this time, the classification of the industry sector of the interpretation and evaluation results from the query of the user terminal device (300), the ratio of the accuracy of the evaluation response according to the evaluation element, and generally, the accuracy can be calculated as the ratio of cases where the model correctly predicted. For example, as a measurement method, the accuracy is verified by the result after inputting each item into the user input area of ​​the test case (a question in the data that has a correct answer). At this time, the design evaluation AI processing unit (210) is a core processing module for design search, analysis, and response generation that collects multimodal input data (image, text, 2D / 3D model, UI capture, etc.), performs LMM-based embedding conversion and vector search, and generates a semantic-based response sentence by comparing with the searched reference design. At this stage, a vector database-based search is performed, and RAG-based response generation (Retrieval-Augmented Generation) is executed. Additionally, quantitative scores and analysis results can be output together in an integrated manner during response generation.

[0044] In addition, by designing a data pipeline that automatically collects and cleans source-to-job result datasets, it automates the data collection, transformation, validation, and storage processes (applying DataOps), and performs data versioning and continuous updates.

[0045] The design evaluation AI processing device (210) of the present invention is configured to automatically verify the design ethics of the generated AI using a generative AI. The AI-generated design ethical verification automatically analyzes and filters whether the design results generated by the AI ​​contain unethical or inappropriate elements. At this time, it is desirable to reflect at least two ethical evaluation items, and selectively perform Copyright & Plagiarism Detection, Bias & Discrimination Detection, Cultural Sensitivity Analysis, Accessibility & Readability Compliance, and Brand Ethics & Consistency.

[0046] Meanwhile, LMM-based design analysis and evaluation models can apply the highest-performing LMM models based on comparison of multimodal LLM architectures and instruction-tuning methods, such as LLaVA - multimodal LLM model, Stable Diffusion + ControlNet - design generation and editing, YOLO / Detectron2 / Segment Anything (SAM) - image analysis, and LangChain + RAG - automatic feedback generation of evaluation criteria.

[0047] In addition, LoRA-based fine-tuning was enabled to reduce hallucinations in the LMM model and improve accuracy; quantitative evaluation algorithms for design evaluation were defined and designed; evaluation indicators and criteria were linked based on expert evaluation guidelines; and a RAG (Retrieval Augmented Generation) system was designed.

[0048] Furthermore, to improve the response performance of the LMM-based design element analysis and evaluation model, we designed a RAG performance comparison to incorporate field-specific design expert evaluations and the latest data; selected an embedding model suitable for the design data; and selected an optimal vector database and implemented vector data for target data management.

[0049] In addition, it was possible to derive effective implementation and management methods for vector data and metadata (internal database (240)) for interpreting design elements and searching for evaluation data.

[0050] For reference, quantitative evaluation methods that can be used include the Score Head-based weighted sum evaluation, Feature Similarity-based vector similarity evaluation, supervised learning regression-based score prediction, and the Score Fusion method, which combines unstructured evaluation and structured analysis.

[0051] The visualization processing unit (220) is a device (UI / rendering layer) that visualizes and displays results, and visualizes the data processed by the design evaluation AI processing unit (210) and outputs it to the dashboard (400).

[0052] The system operating device (230) is a device that manages system operation (control, authority, version control) and handles log storage, authority management, learning data updates, distribution management, etc.

[0053] The internal database (240) stores structured data and unstructured data.

[0054] Here, structured data includes design evaluation scores, number of UX elements, coordinates, summary of model analysis results, user behavior logs, tag / classification attributes, time and version information, etc., and unstructured data includes image files, text documents, 3D model files, video resources, etc.

[0055] The data management device (250) is a central data hub device that stores, searches, links, and transmits various types of design data, and is composed of a video linkage module (251), a Pub / Sub message module (252), a message queue module (253), a DB linkage module (254), and an external connection module (255).

[0056] The video connection module (251) is an input device that transmits / loads video / image files to the system and performs outlier detection and automatic filtering (detection of modified error data), original-result data matching and alignment techniques (matching based on pattern recognition), data consistency verification and automatic error detection.

[0057] The Pub / Sub message module (252) handles asynchronous message transmission and reception between systems and event-based trigger processing.

[0058] The message queue module (253) handles throughput distribution / scheduling / concurrency guarantee → load equalization of the multimodal search and response generator (200) for design evaluation, etc.

[0059] The DB linkage module (254) performs linkage with the Vector DB and the internal database (240).

[0060] The external connection module (255) is an external API gateway device and is an interface between the multimodal search and response generation device (200) for design evaluation, the vector database (100), the user terminal device (300), and the dashboard (400).

[0061] The user terminal device (300) includes a PC, mobile, or browser-based terminal and is a user interface device that performs user search input, design evaluation requests, UI-based result verification, and analysis data dashboard visualization.

[0062] The dashboard (400) is a monitoring / result analysis device (UI Layer) that visually displays structured / unstructured results produced by the design evaluation AI and allows the user to compare, verify, and request re-evaluation. It visualizes quantitative evaluation results for multiple design candidates in the form of a radar chart, bar chart, and time trend graph, and provides a comparison and analysis interface that performs comparison of scores by evaluation element, comparison of overall quantitative indices, and comparison of ethics and accessibility indices by selecting the same design before and after improvement or multiple options.

[0064] Figure 3 is an example UI of Shape Grammar Interpreter, a program for interpreting design elements and automatically generating designs based on Shape Grammar, which is one of the design element interpretation and evaluation methodologies. It derives key variables to be considered when analyzing design interpretation rules used in existing SGI and applying LMM, and reflects them in the design of an intelligent engine for LMM-based design element interpretation and evaluation.

[0066] Figure 4 is an example of a Sortal GI-based Shape Grammar Interpreter UI, and Figure 5 is a diagram showing the result of the Sortal GI-based Shape Grammar Interpreter shown in Figure 4.

[0067] In Fig. 4, which shows an example UI of a Sortal GI-based Shape Grammar Interpreter, real-time design element (shape, point, line, surface) detection, shape grammar for design elements, rule summaries, and shape grammar and rule-based design generation functions are provided. Accordingly, results such as those shown in Fig. 5 can be produced.

[0068] To this end, the present invention incorporates UX / UI design and responsive interface concepts, reflects expert evaluation indicators and guideline LMM models, designs an expert evaluation method for designing a data analysis technique that compares expert feedback and AI evaluation results, designs an automatic design evaluation correction and feedback model using LMM, and designs an automatic design evaluation feedback using generative AI through research on reliability evaluation methods for AI automatic feedback, and establishes a development environment for a design element interpretation and evaluation platform and conducts data research and analysis for building a SaaS design element interpretation and evaluation platform.

[0070] FIG. 6 is an example showing an LMM-based design analysis and evaluation workflow used in a multimodal search and response generation device for design evaluation according to the present invention.

[0071] An example of an LMM-based design analysis and evaluation workflow used in a multimodal search and response generation device for design evaluation according to the present invention is shown in FIG. 6. In the present invention, the design evaluation AI processing device (210) shown in FIG. 2 is programmed to perform LMM-based design analysis and evaluation, and to generate a response for the analyzed and evaluated design.

[0072] To this end, the design evaluation AI processing unit (210) is composed of a data processing analysis module (211), an artificial intelligence model and algorithm (212), and a design generation and editing engine (213).

[0073] The data processing analysis module (211) collects data, refines data, and analyzes trends, and the artificial intelligence model and algorithm (212) performs deep learning optimization, a recommendation system, and a GAN algorithm, and for the recommended images, the design generation and editing engine (213) performs automatic layout placement, color and font adjustment, and object recognition, and responds to the user terminal device (300) and the dash terminal (400).

[0074] In this invention, design evaluation tests utilizing a RAG-based LMM model, verification of model accuracy and processing performance by evaluation indicator, and comparative analysis of LMM-based design analysis and evaluation results with expert evaluation data are performed, and Advanced-RAG (Retrieval Augmented Generation) is used for this purpose.

[0075] In other words, Advanced-RAG was developed for LMM-based design element analysis and evaluation services, and the performance of priority search and internal data search within tables and images was improved based on the latest materials, and the search result list was optimized.

[0077] FIG. 7 is a diagram showing the cloud-based evaluation environment and real-time analysis technology used in the present invention.

[0078] The cloud-based evaluation environment and real-time analysis technology used in the present invention optimized real-time data storage and search functions through the automation of RAG-based design evaluation result updates as shown in Fig. 7, and designed an AI learning model that integrates expert feedback and user evaluation data to build a vector database (100).

[0080] Although the present invention has been described with the examples above, the present invention is not necessarily limited to these examples and can be implemented with various modifications within the scope of the technical concept of the present invention. Accordingly, the examples disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by these examples. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0082] 100 : Vector Database 200: Multimodal search and response generator for design evaluation 210: Design Evaluation AI Processor 220: Visualization Processor 230 : System operating unit 240 : Database 250: Data management device 251: Video connection module 252 : Pub / Sub message module 253 : Message queue module 254 : DB Integration Module 255 : External Connection Module 300 : User terminal device 400 : Dashboard

Claims

Claim 1 A multimodal search and response generation device for design evaluation comprises: an embedding model that converts various different types of design data, including images, text, 2D, 3D models, and UI captures, into a single embedding space; a design evaluation AI processing device (210) that performs a vector-based similarity search on multimodal design data stored in a vector database (100) by embedding multimodal input data, generates semantic-based natural language responses and design evaluation results by combining the searched reference designs with a large-scale multimodal model (LMM), and generates quantitative evaluation results using a Score Head-based weighted sum score, a Feature Similarity-based similarity score, a Regression-based prediction score, or a structured / unstructured Score Fusion method for the performed similarity search results; a visualization processing device (220) that visualizes the natural language responses and design evaluation results generated from the design evaluation AI processing device (210) and displays them on a dashboard (400); a system operating device (230) that handles log storage, authorization management, training data updates, and distribution management for the user terminal device (300); and the design evaluation AI An internal database (240) that stores structured data including design evaluation scores, number of UX elements, coordinates, and model analysis result summary data generated by a processing unit (210), and unstructured data including image files, text documents, 3D model files, and video resources;and includes a data management device (250) that stores, searches, links, and transmits design data between the vector database (100), internal database (240), design evaluation AI processing device (210), visualization processing device (220), user terminal device (300), and dashboard (400). The data management device (250) comprises: a video linkage module (251) that collects and inputs design data through outlier detection, matching and alignment of original-result data, data consistency verification, and error detection; a Pub / Sub message module (252) that performs asynchronous message transmission and reception and event-based trigger processing according to system events between the devices; a message queue module (253) that controls the processing load of the devices through throughput distribution, job scheduling, and concurrency control; a DB linkage module (254) that performs linkage with the vector database (100) and internal database (240); and final interface data between the user terminal device (300), vector database (100), and dashboard (400). A multimodal search and response generation device for design evaluation, comprising an external connection module (255) that performs transmission and reception, wherein the video connection module (251) collects and inputs design data, the DB linkage module (254) stores and searches the design data in the vector database (100) and internal database (240), the Pub / Sub message module (252) and message queue module (253) perform asynchronous transmission and throughput control of the design data, and the external connection module (255) performs data linkage and transmission and reception with the user terminal device (300), the vector database (100), and the dashboard (400). Claim 2 delete Claim 3 delete

Images