Method and system for classifying cultural heritage based on large-scale language model

Abstract

A large-scale language model-based method and system for classifying cultural heritage are disclosed. A method for training a cultural heritage classification model may include the steps of: translating metadata described in a first language to generate alternative data described in a second language; generating image data using the alternative data; and training a cultural heritage classification model using the image data.

Description

Technology Field

[0001] The present invention relates to a method and system for classifying cultural heritage, and more specifically, to a method and system for classifying cultural heritage information composed of classical Korean words using a large-scale language model. Background Technology

[0002] Cultural heritage is currently in the process of digitizing materials accumulated over long periods, and due to the vast volume of data, a systematic and efficient classification and management system is essential. The amount of information in cultural heritage is highly diverse, and customized management is required based on the characteristics and importance of each heritage site.

[0003] Currently, the systematization of cultural heritage is carried out primarily manually, requiring significant costs and time. Furthermore, as the digitization of cultural heritage accelerates over time, the amount of data on new cultural heritage will also continuously increase.

[0004] Conventional cultural heritage data faces difficulties in integrated management across various institutions due to inconsistent structures and a lack of information.

[0005] Therefore, there is a demand for a method to classify cultural heritage data more effectively. The problem to be solved

[0006] The present invention can provide a method and system for classifying cultural heritage information by translating cultural heritage information written in classical Korean text into English, and then using the translated English sentences as input to a text-image generation model to generate more accurate and consistent classification images. means of solving the problem

[0007] A method for training a cultural heritage classification model according to an embodiment of the present invention may include the steps of: translating metadata described in a first language to generate alternative data described in a second language; generating image data using the alternative data; and training a cultural heritage classification model using the image data.

[0008] The metadata of the cultural heritage classification model learning method according to one embodiment of the present invention may be text data including at least one of the name, size, era, material, lifestyle, and artifact number of the cultural heritage.

[0009] The step of generating image data in the method for learning a cultural heritage classification model according to one embodiment of the present invention may generate image data including the attributes of the alternative data and the external features of the cultural heritage corresponding to the first language by inputting the alternative data and the external features of the cultural heritage into a large language model (LLM).

[0010] The step of learning the cultural heritage classification model of the method for learning the cultural heritage classification model according to one embodiment of the present invention may include: a step of assigning a label of a class corresponding to the image data to the image data based on the metadata or the alternative data; a step of generating a learning and evaluation dataset with the image data to which the label has been assigned; and a step of inputting the learning and evaluation dataset to the cultural heritage classification model and learning the cultural heritage classification model according to the output of the cultural heritage classification model.

[0011] The cultural heritage classification model of the method for learning a cultural heritage classification model according to one embodiment of the present invention may include: a spatial stream that extracts features of cultural heritage from image data; a semantic stream that extracts features of cultural heritage from labels of classes corresponding to the image data; and a convolution layer that classifies the class of cultural heritage by merging the features of cultural heritage extracted from the spatial stream and the semantic stream.

[0012] The class of cultural heritage in the cultural heritage classification model learning method according to one embodiment of the present invention may include at least one of an era class indicating the time when the cultural heritage was created; a material class indicating the material characteristics of the cultural heritage; and a lifestyle class indicating the social or industrial background reflected by the cultural heritage.

[0013] A cultural heritage classification method according to one embodiment of the present invention may include the step of receiving cultural heritage data; and the step of classifying the class of cultural heritage included in the cultural heritage data by inputting the cultural heritage data into a cultural heritage classification model learned using text data and image data.

[0014] The cultural heritage classification model of the cultural heritage classification method according to one embodiment of the present invention can be trained by translating metadata written in a first language to generate alternative data written in a second language, and then using image data generated using the alternative data.

[0015] The cultural heritage classification model of the cultural heritage classification method according to one embodiment of the present invention may include: a spatial stream for extracting features of cultural heritage from image data; a semantic stream for extracting features of cultural heritage from a label of a class corresponding to the image data; and a convolution layer for classifying the class of cultural heritage by merging the features of cultural heritage extracted from the spatial stream and the semantic stream. Effects of the invention

[0016] According to one embodiment of the present invention, cultural heritage information written in classical Korean text is translated into English, and then the translated English sentence is used as input to a text-image generation model to generate more accurate and consistent classification images.

[0017] In addition, according to one embodiment of the present invention, by analyzing the association between cultural heritages based on images generated based on sentences translated into English, the entire process of preprocessing, classification, and searching of large-scale cultural heritage data can be automated to maximize work efficiency.

[0018] In addition, according to one embodiment of the present invention, fast and accurate classification and searching are possible at low cost through an automated system, and the efficiency of cultural heritage management using visual data can be increased. Brief explanation of the drawing

[0019] FIG. 1 is a drawing showing a cultural heritage classification system according to an embodiment of the present invention. FIG. 2 is a diagram showing the operation of a translation unit of a cultural heritage classification system according to an embodiment of the present invention. FIG. 3 is an example of the operation result of the translation unit of a cultural heritage classification system according to an embodiment of the present invention. FIG. 4 is a diagram showing the operation of an image generation unit of a cultural heritage classification system according to an embodiment of the present invention. FIG. 5 is an example of the operation result of an image generation unit of a cultural heritage classification system according to an embodiment of the present invention. FIG. 6 is an example of a classification of classes of cultural heritage according to an embodiment of the present invention. Figure 7 is the structure of a cultural heritage classification model according to an embodiment of the present invention. FIG. 8 is a flowchart illustrating a method for classifying cultural heritage according to an embodiment of the present invention. Specific details for implementing the invention

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. A cultural heritage classification method according to one embodiment of the present invention can be performed by a cultural heritage classification system.

[0021] FIG. 1 is a drawing showing a cultural heritage classification system according to an embodiment of the present invention.

[0022] A cultural heritage classification control system (100) may include a translation unit (110), an image generation unit (120), a classification model learning unit (130), and a class classification unit (140) as shown in FIG. 1. At this time, the translation unit (110), the image generation unit (120), the classification model learning unit (130), and the class classification unit (140) may be different processors as shown in FIG. 1, or they may be respective modules included in a program executed on a single processor.

[0023] Additionally, the cultural heritage classification control system (100) may be composed of a cultural heritage classification model learning device, which is a terminal or server including a translation unit (110), an image generation unit (120), and a classification model learning unit (130), and a cultural heritage classification device, which is a terminal or server including a class classification unit (140).

[0024] The translation unit (110) can translate metadata written in a first language to generate alternative data written in a second language. At this time, the metadata may be text data including at least one of the name, size, era, material, lifestyle, and artifact number of the cultural heritage. For example, the first language may be Korean, and the second language may be at least one of English, Japanese, Chinese, French, and German.

[0025] The image generation unit (120) can generate image data using the alternative data generated by the translation unit (110). The image generation unit (120) can generate image data containing the attributes of the alternative data and the external characteristics of the cultural heritage corresponding to the first language by inputting the alternative data and the external characteristics of the cultural heritage corresponding to the first language into a large language model (LLM). For example, if the first language is Hangul, the image generation unit (120) can generate an image that includes the external characteristics of Korean cultural heritage (characteristics that can be defined as Korean style) and corresponds to the attributes of the metadata.

[0026] The classification model learning unit (130) can learn a cultural heritage classification model using image data generated by the image generation unit (120). The classification model learning unit (130) can assign a label of a class corresponding to the image data to the image data based on metadata or alternative data. At this time, the class of cultural heritage may include at least one of an era class indicating the time when the cultural heritage was created; a material class indicating the material characteristics of the cultural heritage; and a lifestyle class indicating the social or industrial background reflected by the cultural heritage.

[0027] And, the classification model learning unit (130) can generate a learning and evaluation data set with image data to which labels have been assigned.

[0028] Next, the classification model learning unit (130) can input a learning and evaluation data set into the cultural heritage classification model and learn the cultural heritage classification model by comparing the output of the cultural heritage classification model with the correct answer data corresponding to the metadata.

[0029] A cultural heritage classification model may include a spatial stream that extracts features of cultural heritage from image data; a semantic stream that extracts features of cultural heritage from labels of classes corresponding to the image data; and a convolution layer that classifies the class of cultural heritage by merging the features of cultural heritage extracted from the spatial stream and the semantic stream.

[0030] Additionally, the classification model learning unit (130) can evaluate the performance of the cultural heritage classification model by inputting a learning and evaluation dataset into the learned cultural heritage classification model. If the performance of the cultural heritage classification model is above a threshold, the classification model learning unit (130) can provide the cultural heritage classification model to the class classification unit (140). If the performance of the cultural heritage classification model is below a threshold, the classification model learning unit (130) can repeat the learning of the cultural heritage classification model until the performance becomes above the threshold.

[0031] Additionally, the classification model learning unit (130) can analyze the association between image data and the relationship between labels to provide the user with a visual understanding of the relationship between cultural heritage information. Furthermore, the classification model learning unit (130) can apply the association between image data and the relationship between labels to the learning of the cultural heritage classification model. Thus, the cultural heritage classification model can be trained to classify the class of cultural heritage included in the cultural heritage data by considering the association between image data and the relationship between labels.

[0032] For cultural heritage with limited information or insufficient records, text-based classification is difficult, so utilizing visual data through generative AI can be more effective.

[0033] Therefore, image data can be utilized to supplement visual materials in cultural heritage classification models when it is difficult to classify cultural heritage using only text data. Furthermore, by classifying cultural heritage using image data rather than text data such as metadata or alternative data, cultural heritage classification models can consequently apply text information with varying amounts of information to cultural heritage systems through consistent results.

[0034] The class classification unit (140) can receive cultural heritage data. The class classification unit (140) can classify the class of cultural heritage included in the cultural heritage data by inputting the cultural heritage data into a cultural heritage classification model trained using text data and image data. At this time, the cultural heritage data may be text data in which a description of the cultural heritage is written in a first language.

[0035] A cultural heritage classification system according to one embodiment of the present invention can generate more accurate and consistent classification images by translating cultural heritage information written in classical Korean text into English and using the translated English sentences as input to a text-image generation model.

[0036] A cultural heritage classification system according to one embodiment of the present invention can analyze the association between cultural heritages based on images generated from sentences translated into English. Accordingly, the cultural heritage classification system according to one embodiment of the present invention can maximize work efficiency by automating the entire process, including preprocessing, classification, and searching of large-scale cultural heritage data.

[0037] In addition, the cultural heritage classification system according to one embodiment of the present invention enables fast and accurate classification and searching at low cost through an automated system, and can increase the efficiency of cultural heritage management by utilizing visual data.

[0038] FIG. 2 is a diagram showing the operation of a translation unit of a cultural heritage classification system according to an embodiment of the present invention.

[0039] The translation unit (110) can input metadata (210) into a large-scale language model (LLM) (220) to generate alternative data (230) written in a second language. At this time, the metadata (210) may be text data including at least one of the name, size, era, material, lifestyle, and artifact number of the cultural heritage. For example, the metadata (210) may include the name, material, era, lifestyle, etc. (X1, X2, X3, X4) of the cultural heritage. And, the alternative data (230) may include a prompt to be input into the image generation unit (120).

[0040] Since each of the cultural heritages included in the metadata (210) has common elements such as name, material, era, and lifestyle, the large-scale language model (LLM) (220) can construct prompts based on the common elements in each of the cultural heritages included in the metadata (210).

[0041] FIG. 3 is an example of alternative data (320) generated by the translation unit of a cultural heritage classification system according to an embodiment of the present invention, which translates Korean words of metadata (310) into English sentences using a large-scale language model (GPT-3.5).

[0042] The metadata (310) used as a search query in a large-scale language model included words such as "lid piece," "reorganization," "early iron age," "earthenware," "industry / livelihood," and "prehistoric life," as shown in FIG. 3.

[0043] At this time, each of the Korean words in the metadata (310) can be translated as "Lid fragment", "Lid piece", "Early Iron Age Korea", "Earthenware pottery", "Prehistoric livelihoods and industries", etc., as shown in the alternative data (320).

[0044] FIG. 4 is a diagram showing the operation of an image generation unit of a cultural heritage classification system according to an embodiment of the present invention.

[0045] The image generation unit (120) can generate image data using the alternative data (230) generated by the translation unit (110). The image generation unit (120) can input the alternative data (230) into a text-based image generation model (AI) (420). The text-based image generation model (AI) (420) can generate image data (430) by visually converting the alternative data (230) using a large-scale language model (LLM). For example, since the image data (430) is an intermediate image for association and classification, the text-based image generation model (AI) (420) may be an SDXL 1.0 model.

[0046] Image data (43) can be used for learning and evaluation. Accordingly, the image generation unit (120) can generate image data (430) containing the attributes of alternative data (230) and the external features of the cultural heritage corresponding to the first language by inputting the external features of the cultural heritage corresponding to the first language into a text-based image generation model (AI) (420).

[0047] FIG. 5 is an example of the operation result of an image generation unit of a cultural heritage classification system according to an embodiment of the present invention.

[0048] The image (510) is an image generated by the image generation unit (120) by adding the condition of the external characteristics of Korean cultural heritage to the attribute of being a wooden statue in the shape of a duck and the attribute representing the era of the cultural heritage, and the image (515) is the original (actual) image of the cultural heritage.

[0049] The image (520) is an image generated by the image generation unit (120) by adding the condition of the external characteristics of Korean cultural heritage to the attribute of a tile fragment and the attribute representing the era of the cultural heritage, and the image (525) is the original (actual) image of the cultural heritage.

[0050] The image (530) is an image generated by the image generation unit (120) by adding the condition of the external characteristics of Korean cultural heritage to the attribute of being jewelry and the attribute indicating the era of cultural heritage, and the image (535) is the original (actual) image of the cultural heritage.

[0051] As shown in FIG. 5, the image generated by the image generation unit (120) may be an image with a shape similar to the original (actual) image of the cultural heritage.

[0052] FIG. 6 is an example of a classification of classes of cultural heritage according to an embodiment of the present invention.

[0053] The class label corresponding to the image data assigned by the classification model learning unit (130) can be classified according to a 'super category' as shown in FIG. 6.

[0054] Specifically, the classes of cultural heritage corresponding to the image data can be assigned to each of the three major categories: era, material, and lifestyle. Furthermore, each major category may include various sub-classes in detail.

[0055] The 'Era' category includes classes that distinguish historical periods of Korea. For example, the Era class can be assigned to image data as one of a total of 11 eras: Bronze Age, Early Iron Age, Proto-Three Kingdoms, Baekje, Silla, Three Kingdoms, Unified Silla, Goryeo, Late Goryeo and Early Western Goryeo, Joseon, and Japanese Colonial Period.

[0056] The 'Material' category represents the material characteristics of each cultural heritage and is classified based on surface texture or material. For example, the material class can be assigned to image data as one of 14 subclasses: metal, earthenware, pottery, stone, glass / jewelry, herbaceous materials, wood, bone / horn / shell, paper, leather / fur, fiber, seed, mineral, rubber, and lacquerware.

[0057] The 'Lifestyle' category is a class that describes the social or industrial background reflected by the cultural heritage. The lifestyle class can be assigned to image data as one of seven subclasses: transportation / communication, culture and arts, social life, industry / livelihood, food life, housing life, and clothing life.

[0058] The classification model learning unit (130) can distinguish the visual, material, and lifestyle characteristics of cultural heritage by assigning classes based on different criteria corresponding to the image data, thereby enabling the derivation of various results according to the classification criteria.

[0059] Figure 7 is the structure of a cultural heritage classification model according to an embodiment of the present invention.

[0060] The cultural heritage classification model may be a multi-label classification model that processes image and text data simultaneously by using a Vision Transformer and a Large Language Model (LLM) together. Specifically, the cultural heritage classification model (700) may consist of a spatial stream (710), a semantic stream (720); and a convolution layer (730).

[0061] A spatial stream (710) can extract features of cultural heritage from image data (430). Specifically, the spatial stream (710) can encode visual features of image data (430) using a vision transformer.

[0062] The semantic stream (720) can extract features of cultural heritage from the labels (721) of the class corresponding to the image data assigned by the classification model learning unit (130). Specifically, the semantic stream (720) can use a large-scale language model to understand the text context for each of the labels (721) of the class corresponding to the image data. Then, the semantic stream (720) can extract the understood results as text features of cultural heritage.

[0063] The convolution layer (730) can classify the class of cultural heritage by merging the features of cultural heritage extracted from the spatial stream (710) and the semantic stream (720), respectively, and performing a final label prediction.

[0064] FIG. 8 is a flowchart illustrating a method for classifying cultural heritage according to an embodiment of the present invention.

[0065] In step (810), the translation unit (110) can translate metadata written in a first language to generate alternative data written in a second language. At this time, the metadata may be text data including at least one of the name, size, era, material, lifestyle, and artifact number of the cultural heritage. For example, the first language may be Korean, and the second language may be at least one of English, Japanese, Chinese, French, and German.

[0066] In step (820), the image generation unit (120) can generate image data using the alternative data generated in step (810). The image generation unit (120) can generate image data containing the attributes of the alternative data and the external features of the cultural heritage corresponding to the first language by inputting the alternative data and the external features of the cultural heritage corresponding to the first language into a large-scale language model (LLM).

[0067] In step (830), the classification model learning unit (130) can learn a cultural heritage classification model using the image data generated in step (820). The classification model learning unit (130) can assign a label of a class corresponding to the image data to the image data based on metadata or alternative data. Then, the classification model learning unit (130) can generate a learning and evaluation data set using the image data to which the labels have been assigned. Next, the classification model learning unit (130) can input the learning and evaluation data set into the cultural heritage classification model and learn the cultural heritage classification model by comparing the output of the cultural heritage classification model with the correct answer data corresponding to the metadata.

[0068] Additionally, the classification model learning unit (130) can evaluate the performance of the cultural heritage classification model by inputting a learning and evaluation dataset into the learned cultural heritage classification model. If the performance of the cultural heritage classification model is above a threshold, the classification model learning unit (130) can provide the cultural heritage classification model to the class classification unit (140). If the performance of the cultural heritage classification model is below a threshold, the classification model learning unit (130) can repeat the learning of the cultural heritage classification model until the performance becomes above the threshold.

[0069] In addition, the classification model learning unit (130) can analyze the association between image data and the relationship between labels to provide the user with a visual understanding of the association between cultural heritage information.

[0070] In step (840), the class classification unit (140) can receive cultural heritage data. Then, the class classification unit (140) can classify the class of cultural heritage included in the cultural heritage data by inputting the cultural heritage data into a cultural heritage classification model trained using text data and image data. At this time, the cultural heritage data may be text data in which a description of the cultural heritage is written in a first language.

[0071] The present invention can generate more accurate and consistent classification images by translating cultural heritage information written in classical Korean text into English and using the translated English sentences as input to a text-image generation model.

[0072] The present invention can analyze the correlation between cultural heritage based on images generated from sentences translated into English. Therefore, the present invention can maximize work efficiency by automating the entire process, including preprocessing, classification, and searching of large-scale cultural heritage data.

[0073] In addition, the present invention enables fast and accurate classification and searching at low cost through an automated system, and can increase the efficiency of cultural heritage management by utilizing visual data.

[0075] Meanwhile, the method according to the present invention is written as a program executable on a computer and can be implemented on various recording media such as magnetic storage media, optical reading media, and digital storage media.

[0076] Implementations of the various technologies described herein may be implemented as digital electronic circuits, or as computer hardware, firmware, software, or combinations thereof. Implementations may be implemented as computer program products, i.e., computer programs tangibly embodied in information carriers, such as machine-readable storage devices (computer-readable media) or radio signals, for processing by the operation of data processing devices, e.g., programmable processors, computers, or multiple computers, or for controlling such operation. Computer programs such as the computer program(s) described above may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. Computer programs may be deployed to be processed on one computer or multiple computers at one site, or distributed across multiple sites and interconnected by a communication network.

[0077] Processors suitable for processing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Generally, the processor will receive instructions and data from read-only memory or random access memory, or both. The elements of the computer may include at least one processor that executes instructions and one or more memory devices that store instructions and data. Generally, the computer may include one or more mass storage devices that store data, for example, magnetic, magneto-optical disks, or optical disks, or may be combined to receive data from these, transmit data to these, or both. Information carriers suitable for embodying computer program instructions and data include, for example, semiconductor memory devices, magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disks); magneto-optical media such as floptical disks; ROMs (Read Only Memory); RAMs (Random Access Memory); flash memory; EPROMs (Erasable Programmable ROM); EEPROMs (Electrically Erasable Programmable ROM); etc. Processors and memory may be supplemented by or included in special-purpose logic circuit organizations.

[0078] Additionally, a computer-readable medium may be any available medium accessible by a computer and may include both computer storage media and transmission media.

[0079] Although this specification contains details of a number of specific embodiments, they should not be understood as limiting the scope of any invention or claimables, but rather as descriptions of features that may be characteristic of a specific embodiment of a specific invention. Specific features described in this specification in the context of individual embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any appropriate sub-combination. Furthermore, while features may operate in a specific combination and be described as initially claimed, one or more features from the claimed combination may be excluded from the combination in some cases, and the claimed combination may be changed to a sub-combination or a variation of the sub-combination.

[0080] Likewise, although operations are depicted in the drawings in a specific order, this should not be understood as requiring that such operations be performed in that specific or sequential order depicted to obtain a desirable result, or that all depicted operations must be performed. In certain cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various device components of the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and devices can generally be integrated together into a single software product or packaged into multiple software products.

[0081] Meanwhile, the embodiments of the present invention disclosed in this specification and drawings are merely specific examples provided to aid understanding and are not intended to limit the scope of the present invention. It is obvious to those skilled in the art that other variations based on the technical concept of the present invention are possible in addition to the embodiments disclosed herein. Explanation of the symbols

[0082] 100: Cultural Heritage Classification System 110: Translation Department 120: Image generation section 130: Classification model training unit 140: Class classification section

Claims

Claim 1 A method for learning a cultural heritage classification model, comprising: a step of translating metadata written in a first language to generate alternative data written in a second language; a step of generating image data using the alternative data; and a step of learning a cultural heritage classification model using the image data.

Images