Summarization device, summarization method, and summarization program

The summarization device effectively generates structured summaries by linking keywords and visual information to create a clear depiction of concept relationships, addressing the limitations of existing technologies in summarizing content with both text and visual elements.

JP2026182534APending Publication Date: 2026-11-19PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2025078843
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-09
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing summarization technologies using Large Language Models (LLM) struggle to generate clear summaries that adequately explain both the concepts and relationships between concepts when dealing with content containing both text and visual information, often resulting in redundant or unclear summaries.

Method used

A summarization device and method that utilizes an element extraction module to generate metadata linking keywords, detailed content text, and visual supplementary information, followed by a structured summarization module to construct relationships between concepts, generating a structured summary based on these elements.

Benefits of technology

The solution enables the generation of clear summaries that accurately depict the structure and relationships between concepts, providing a concise and structured overview of complex content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2026182534000001_ABST
    Figure 2026182534000001_ABST
Patent Text Reader

Abstract

Generates a clear summary that highlights the structure of concepts and the relationships between them within the content. [Solution] The summarization device comprises: an acquisition unit that acquires a content source including text information and visual information; an element extraction module that inputs the content source into a predetermined language model to extract keywords related to each of the multiple concepts contained in the content source, and generates detailed content text that explains each of the multiple concepts, thereby generating metadata in which keywords and detailed content text are linked for each of the multiple concepts; and a structured summarization module that inputs the metadata into a language model to structure the concepts for each of the multiple concepts based on keywords and detailed content text, build relationships by linking the multiple related concepts together, and generate a structured summary of the content source based on the structured multiple concepts and the built relationships.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a summarization device, a summarization method, and a summarization program.

Background Art

[0002] Patent Document 1 discloses a content generation system that generates content using LLM (Large Language Models). The system selects visualization software capable of generating visual information to be included in the content, based on linguistic information input via a user interface. Further, the system operates the selected visualization software with text information obtained based on the output of the LLM according to the linguistic information input via the user interface. Then, the system acquires visual information to be included in the content, and generates content that includes at least part of the acquired visual information and at least part of the linguistic information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, the technology of summarizing content using LLM has become widespread. Content input to LLM may include both text and visual information, and summarization by LLM based on each type of information presents the following challenges. First, when LLM summarizes content using only text information, the generated summary may, for example, adequately explain the concepts contained in the content, but the relationships between the concepts may be unclear. On the other hand, when LLM summarizes content using only visual information, the generated summary may, for example, clearly explain the relationships between the concepts contained in the content, but the explanation of the concepts may be insufficient. Furthermore, even when LLM summarizes content using both text and visual information, a redundant summary may be generated, where the content generated based on the text and visual information is simply linearly combined. In this redundant summary, the relationships between concepts contained in the content, or the structure of the concepts, may be unclear.

[0005] This disclosure was devised in light of the aforementioned conventional circumstances and aims to generate a clear summary of the structure of concepts and the relationships between concepts contained in the content. [Means for solving the problem]

[0006] This disclosure provides a summarization device comprising: an acquisition unit that acquires a content source including text information and visual information; an element extraction module that inputs the content source into a predetermined language model to extract keywords related to each of a plurality of concepts contained in the content source, and generates detailed content text that explains each of the plurality of concepts, thereby generating metadata for each of the plurality of concepts in which the keywords and the detailed content text are linked; and a structured summarization module that inputs the metadata into the language model to structure each of the plurality of concepts based on the keywords and the detailed content text, constructs relationships by linking the plurality of related concepts together, and generates a structured summary of the content source based on the structured plurality of concepts and the constructed relationships.

[0007] Furthermore, this disclosure provides a summarization method that acquires a content source including text information and visual information, inputs the content source into a predetermined language model to extract keywords related to each of a plurality of concepts contained in the content source, generates detailed content text explaining each of the plurality of concepts, generates metadata for each of the plurality of concepts linked to the keywords and the detailed content text, inputs the metadata into the language model to structure each of the plurality of concepts based on the keywords and the detailed content text, constructs relationships by linking the related plurality of concepts together, and generates a structured summary of the content source based on the structured plurality of concepts and the constructed relationships.

[0008] Furthermore, this disclosure provides a summarization program that causes a computer to perform the following actions: acquire a content source including text information and visual information; input the content source into a predetermined language model to extract keywords related to each of a plurality of concepts contained in the content source and generate detailed content text explaining each of the plurality of concepts; generate metadata for each of the plurality of concepts linking the keywords and the detailed content text; input the metadata into the language model to structure each of the plurality of concepts based on the keywords and the detailed content text; build relationships by linking the related plurality of concepts together; and generate a structured summary of the content source based on the structured plurality of concepts and the built relationships.

[0009] Furthermore, any combination of the above components, as well as any conversion of the expressions of this disclosure between methods, apparatus, systems, storage media, computer programs, etc., are also valid as aspects of this disclosure. [Effects of the Invention]

[0010] According to this disclosure, it is possible to generate a clear summary of the structure of concepts included in the content and the relationships between those concepts. [Brief explanation of the drawing]

[0011] [Figure 1] Block diagram showing an example of the configuration of the functions of the summarization device according to Embodiment 1. [Figure 2] A schematic diagram showing an example of text information related to Embodiment 1. [Figure 3] Table diagram showing an example of information included in metadata related to Embodiment 1 [Figure 4] A schematic diagram showing an example of concept structuring by the concept structuring unit according to Embodiment 1. [Figure 5] A schematic diagram showing an example of the construction of relationships between concepts by the knowledge structure construction unit according to Embodiment 1. [Figure 6] A schematic diagram showing an example of a structured summary related to Embodiment 1. [Figure 7] Table diagram showing an example of the display format of a structured summary according to Embodiment 1 [Figure 8] Schematic diagram illustrating prompts for structured summaries according to Embodiment 1 [Figure 9] Schematic diagram illustrating the storage of structured summaries according to Embodiment 1 in a data database. [Figure 10] Block diagram showing an example of the hardware configuration of the summarization device related to this disclosure. [Figure 11] Table diagram illustrating the differences between summaries generated by conventional technology and summaries generated by Embodiment 1. [Modes for carrying out the invention]

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings. However, descriptions that are unnecessarily detailed may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims. The functions of one configuration shown in the embodiments of the present disclosure may be implemented by two or more physical configurations, or the functions of two or more configurations may be implemented by, for example, a single physical configuration, which is permissible.

[0013] (Embodiment 1) Figure 1 is a block diagram showing an example of a functional configuration of a summarization device 10 according to Embodiment 1.

[0014] The summarization device 10 is a device capable of outputting the content of a content source 30 as a structured summary 60 in response to an input of the content source 30 from a user. In the present embodiment, the content of the content source 30 is assumed to include a plurality of concepts. Further, the "structured summary" means a summary that includes a plurality of concepts contained in the content source 30 and organizes them in a hierarchical manner.

[0015] The summarization device 10 includes, as functions, an acquisition unit 20, an element extraction module 40, a structured summary module 50, an output unit 70, and a material database 80. Further, the structured summary module 50 includes, as functions, a concept structuring unit 51 and a knowledge structure construction unit 52. These functions may be implemented by a processor 1001 (see Figure 10) included in the summarization device 10 executing a program in cooperation with a memory 1002 (see Figure 10).

[0016] The content source 30 includes text information 31 and visual information 32 as information. The text information 31 is information represented by a character string included in the content source 30. Figure 2 is a schematic diagram illustrating an example of the text information 31 according to the first embodiment.

[0017] The visual information 32 is information represented by visual elements such as images, moving images, diagrams, illustrations, and photographs included in the content source 30. Note that the visual information 32 may include the text information 31. This is because, for example, images, moving images, diagrams, illustrations, photographs or the like may contain character strings.

[0018] Examples of the content source 30 include PDF files, web pages, moving image files, and the like.

[0019] A user inputs the content source 30 including the text information 31 as exemplified in Figure 2 and the visual information 32 such as a moving image via an input device 1004 (see Figure 10) included in the summarization apparatus 10. Then, the acquisition unit 20 acquires the input content source 30.

[0020] The element extraction module 40 extracts text information 31 and visual information 32 from the content source 30 acquired by the acquisition unit 20 using a predetermined language model. Here, the language models used by the element extraction module and the structured summarization module 50 described later may include LLM (Large Language Models) and VLM (Vision Language Model). LLM primarily processes text information 31 and has functions for extraction and organization. On the other hand, VLM processes visual information 32, recognizing visual elements such as charts or illustrations, and generating supplementary explanatory text from these visual elements. The element extraction module 40 and the structured summarization module 50 described later may use multiple language models as needed, rather than using only one language model. Furthermore, the element extraction module 40 and the structured summarization module 50 described later may use a general-purpose language model that has the functions of LLM and VLM, for example. The element extraction module 40 may also use OCR (Optical Character Reader) technology when extracting text information 31 from the content source 30.

[0021] The element extraction module 40 uses a language model to further extract important words or phrases related to multiple concepts contained in the content source 30 as keywords from the extracted text information 31. The element extraction module 40 also uses a language model to generate supplementary explanatory text about the multiple concepts contained in the content source 30 based on the visual elements recognized from the extracted visual information 32. Furthermore, the element extraction module 40 uses a language model to organize and generate detailed information about each concept as detailed content text, based on the extracted text information 31 and the generated supplementary explanatory text. Finally, the element extraction module 40 uses a language model to extract information related to each concept (images, still images from videos, diagrams, illustrations, or photographs, etc.) from the visual information 32 as visual supplementary information, and also extracts the location information of the visual supplementary information within the content source 30. The location information within the content source 30 may be, for example, a page number if the content source 30 is a PDF, or the playback time if the content source 30 is a video.

[0022] The element extraction module 40 generates metadata 41 by using a language model to link extracted keywords, generated detailed content text, visual supplementary information, and the location information of the visual supplementary information for each concept.

[0023] In summary, metadata 41 is generated in a format that includes keywords, detailed content text, visual supplementary information, and location information of the visual supplementary information for each of the multiple concepts contained in the content source 30.

[0024] Keywords are important words or phrases that define the concepts contained in content source 30. Whether a word or phrase is important or not may be determined by a language model.

[0025] The detailed content text is a detailed explanation of the concepts contained in the content source 30. If the content source 30 includes visual information 32 about a certain concept, the detailed content text describing that concept is generated based not only on the text information 31 but also on the visual information 32. In this case, the detailed content text can describe the concept more accurately than if it were generated based solely on the text information 31.

[0026] Visual supplementary information is information from the visual information 32 included in the content source 30 that provides supplementary explanations about the concepts included in the content source 30. Note that visual information 32 may contain information about a particular concept included in the content source 30, but may not contain information about a different concept. Therefore, the metadata 41 does not necessarily have to include visual supplementary information for each of multiple concepts.

[0027] Location information indicates the location of visually supplementary information within the content source 30, and may be, for example, a page number or the playback time of a video. Note that if visually supplementary information does not exist for a given concept, then the location of the corresponding visually supplementary information does not exist either. Therefore, similar to visually supplementary information, location information for each of multiple concepts does not necessarily have to be included in the metadata 41.

[0028] In this way, the element extraction module 40 can use a predetermined language model to identify multiple concepts based on the text information 31 and visual information 32 contained in the content source 30, and generate metadata 41 that summarizes various pieces of information related to each concept.

[0029] Figure 3 is a table diagram showing an example of the information contained in the metadata 41 according to Embodiment 1. For the sake of explanation, the content source 30 in this embodiment is assumed to be a video introducing the GDPR (General Data Protection Regulation). In this case, the element extraction module 40 generates metadata 41 as illustrated in Figure 3.

[0030] In the example in Figure 3, for example, "data protection," "EU," and "personal information" are grouped together as keywords relating to a certain concept and included in metadata 41. Furthermore, metadata 41 also includes detailed content text explaining that concept in detail, a GDPR overview diagram as visual supplementary information, and the video playback time as location information for the GDPR overview diagram. These keywords, detailed content text, visual supplementary information, and location information relate to one of several concepts included in content source 30 and are linked to each other. In the example in Figure 3, metadata 41 further includes "access rights" as information related to another concept, and further includes detailed content text, visual supplementary information, and location information linked to "access rights."

[0031] The structured summarization module 50 analyzes the metadata 41 generated by the element extraction module using a predetermined language model and generates a structured summary 60 by performing a two-stage structuring process on the content source 30. The first stage of structuring is performed by the conceptual structuring unit 51. The language model used by the structured summarization module 50 may be the same language model used by the element extraction module 40.

[0032] Figure 4 is a schematic diagram showing an example of concept structuring by the concept structuring unit 51 according to Embodiment 1. The concept structuring unit 51 structures each of the multiple concepts included in the content source 30 based on keywords and detailed content text linked to each other in the metadata 41 by inputting metadata 41 into a predetermined language model. As illustrated in Figure 4, the concept structuring unit 51 generates a structured summary 53 for the concept C1 "access rights" (see Figure 5) and a structured summary 54 for the concept C2 "consent management" (see Figure 5) among the multiple concepts included in the content source 30. For example, the concept structuring unit 51 may generate the structured summary 53 using a language model based on the keyword "access rights" included in the metadata 41 and the detailed content text linked to that keyword. In this way, the concept structuring unit 51 generates structured summaries for each of the multiple concepts included in the content source 30 by structuring each of the multiple concepts.

[0033] Furthermore, the conceptual structuring unit 51 may use a language model to link a structured concept with visual supplementary information related to that concept and the location information of that visual supplementary information. In the example in Figure 4, the conceptual structuring unit 51 links the structured concept of "access rights" (i.e., structured summary 53) with visual supplementary information 32-1, visual supplementary information 32-2, and visual supplementary information 32-3, respectively. Also in the example in Figure 4, the conceptual structuring unit 51 links the structured concept of "consent management" (i.e., structured summary 54) with visual supplementary information 32-4.

[0034] The conceptual structuring unit 51 may link each visual supplement to the corresponding portion of the structured summary. For example, the conceptual structuring unit 51 links the visual supplement 32-1 to the summary portion of "data subject identification information" in the structured summary 53. When the conceptual structuring unit 51 links the visual supplement to the structured summary, it also links the positional information of the visual supplement.

[0035] The second stage of structuring by the structured summarization module 50 is performed by the knowledge structure construction unit 52. Figure 5 is a schematic diagram showing an example of the construction of relationships between concepts by the knowledge structure construction unit 52 according to Embodiment 1.

[0036] The knowledge structure construction unit 52 analyzes the relationships between concepts by inputting the structured summaries for each of the multiple concepts generated by the concept structuring unit 51 into the language model. For example, the knowledge structure construction unit 52 analyzes whether the relationships between multiple concepts are superior, equivalent, or subordinate. Then, the knowledge structure construction unit 52 constructs the relationships between concepts by linking them together. For the sake of explanation, in the example in Figure 5, if concepts are connected by double-headed arrows, those concepts are considered equivalent. Also, if concepts are connected by arrows, the concept connected to the starting end of the arrow is considered a subordinate concept of the concept connected to the ending end of the arrow.

[0037] As illustrated in Figure 5, the knowledge structure construction unit 52 analyzes, using a language model, that concepts C1, C2, and C3 are all equivalent concepts. Concept C1 relates to "access rights," concept C2 to "consent management," and concept C3 to "data retention." If there are multiple equivalent concepts, the knowledge structure construction unit 52 may generate a superordinate concept for the multiple equivalent concepts. Here, the generated superordinate concept may be a concept obtained by the language model (e.g., LLM) abstracting the multiple equivalent concepts based on the knowledge of the language model. In this case, the generated superordinate concept may be a concept that does not explicitly appear or is not shown in the content source 30. If a superordinate concept explicitly appears in the content source 30, the knowledge structure construction unit 52 may adopt that superordinate concept as the superordinate concept to generate. Furthermore, in the content source 30, a superordinate concept and multiple equivalent concepts may explicitly appear, but the relationship between the superordinate concept and the multiple equivalent concepts may not be explicitly shown. In this case, based on the knowledge held by the language model, the knowledge structure construction unit 52 may link the higher-level concept to the concept that encompasses the multiple equivalent concepts. The knowledge structure construction unit 52 may generate concept C4 related to "data management policy" as a higher-level concept of concepts C1, C2, and C3.

[0038] Furthermore, as illustrated in Figure 5, concept C5, which relates to "data quality management," is at the same level as concept C4. Also, concept C5 is a higher-level concept than concepts C6, C7, and C8, and concept C7 is a higher-level concept than concept C6. Concept C6 relates to "verification of accuracy," concept C7 to "maintaining integrity," and concept C8 to "limiting redundancy." In Figure 5, the relationship between concepts C1 to C4 is represented as a hierarchical structure where equivalent concepts C1 to C3 exist below the higher-level concept C4. However, when fully representing the knowledge structure, the hierarchical relationships between concepts do not necessarily follow a hierarchical structure. For example, in the case of concepts C5 to C7, concept C6 is a subordinate concept to concepts C5 and C7, but concept C7 is also a higher-level concept than concept C6. Therefore, from the perspective of concept C5, concept C6 is a subordinate concept one level below, and at the same time, a subordinate concept two levels below, with concept C7 in between. Furthermore, in relation to concepts C6 and C7, concept C8 is neither superior, equivalent, nor inferior, but like concepts C6 and C7, it is a subordinate concept of concept C5.

[0039] Thus, the relationships between concepts do not necessarily form a hierarchical structure, and therefore, higher-level concepts are not necessarily generated based on multiple equivalent concepts. For example, concepts C6 to C8 are not equivalent concepts, but a higher-level concept C5 may be generated based on these concepts. In other words, a higher-level concept can be generated based on multiple concepts that have any relationship. However, if a higher-level concept is generated from multiple mutually independent concepts, there is a risk that the generated higher-level concept will become too abstract. Therefore, generating a higher-level concept based on multiple equivalent concepts is more likely to generate a higher-level concept with an appropriate level of abstraction than generating one from multiple mutually independent concepts.

[0040] Note that the concepts shown in Figure 5, along with the arrows and double-headed arrows indicating the relationships between concepts, are merely illustrative examples. Therefore, the number of concepts included in the content source 30, and the method of linking concepts by the knowledge structure construction unit 52, are not limited to the example in Figure 5.

[0041] The knowledge structure construction unit 52 reconstructs the structured summaries (e.g., structured summary 53, structured summary 54) of the multiple concepts contained in the content source 30, which were generated by the concept structuring unit 51, so as to maintain the relationships between the constructed concepts as shown in the example in Figure 5. This allows the knowledge structure construction unit 52 to generate a structured summary 60 of the content source 30.

[0042] Figure 6 is a schematic diagram showing an example of a structured summary 60 according to Embodiment 1. As shown in Figure 6, in the structured summary 60, the descriptions of concepts C1, C2, and C3 are written at a level one level below concept C4. In other words, the relationships between the concepts explained with reference to Figure 5 are maintained in the structured summary 60. Furthermore, the descriptions of concepts C1 and C2 are based on the structured summaries 53 and 54 generated by the concept structuring unit 51, respectively. In this way, the knowledge structure construction unit 52 can generate a structured summary of the content source 30 by reconstructing the structured summaries for each concept, such as the structured summaries 53 and 54 generated by the concept structuring unit 51. This structured summary is the overall structured summary 60 of the multiple concepts included in the content source 30.

[0043] The structured summary 60 generated by the knowledge structure construction unit 52 is output from the output unit 70 to the display device 1005 (see Figure 10) of the summarization device 10 and displayed on the display device 1005. The output unit 70 may display the structured summary 60 on the display device 1005 in, for example, an expandable hierarchical text format. By displaying the structured summary 60 in an expandable hierarchical text format, the summarization device 10 can display the content of the content source 30 concisely, even if the content is complex. The user can then expand the explanation of the parts of the content source 30 they want to know more about, or quickly grasp the overall overview.

[0044] The output unit 70 may, for example, switch between expanded and collapsed displays of the structured summary 60 in response to user operation via the input device 1004. In the expanded display of the structured summary 60, the collapsed expansion section 60-1 may be expanded and displayed, as illustrated in Figure 6. In the collapsed display, the expansion section 60-1 is displayed in a collapsed state. In other words, in the collapsed display, the expansion section 60-1 is not displayed. When the structured summary 60 is expanded, the detailed content text or visual supplementary information of one or more concepts contained in the content source 30 is further displayed. In the example in Figure 6, the expansion section 60-1 is expanded to display further detailed content text about the identification information and visual supplementary information 32-1. User operation for switching between expanded and collapsed displays may be, for example, clicking the expand button 65 or hovering the pointer over the expand button 65. An example where the user operation is a click will be described below.

[0045] Furthermore, the input device 1004 may accept predetermined user operations on the text in the structured summary 60 relating to concepts to which visual supplementary information is linked. For example, the input device 1004 may accept clicks on the text "access rights" in the structured summary 60. The output unit 70 may then display the portion of the visual information 32 of the content source 30 corresponding to the visual supplementary information on the display device 1005 in a format visible to the user, based on the positional information of the visual supplementary information. For example, if the user clicks on the text "access rights" in the structured summary 60, the summarization device 10 may adjust the playback time of the video introducing GDPR to match the time when the explanation of "access rights" is given, and then display it on the display device 1005. Alternatively, for example, if the content source 30 is a PDF, the summarization device 10 may display the page of the PDF corresponding to the text in the structured summary on the display device in response to the user's click on the text in the structured summary.

[0046] Furthermore, the output unit 70 may display the structured summary 60 of the content source 30 on the display device 1005 in a format other than a text format with an expandable hierarchical structure. For example, the output unit 70 may display the structured summary 60 of the content source 30 in a diagrammatic format with an expandable hierarchical structure. Examples of diagrammatic formats with an expandable hierarchical structure include mind maps and flowcharts. By displaying the structured summary 60 of the content source 30 as a mind map, the summarization device 10 can display each concept in a way that expands based on the core concept among the multiple concepts included in the content source 30. By displaying the structured summary 60 of the content source 30 as a flowchart, the summarization device 10 can display the sequential order of each concept in an emphasis.

[0047] Alternatively, the output unit 70 may display the structured summary 60 of the content source 30 in a knowledge organization format that clearly identifies each of the multiple concepts contained in the content source 30. The knowledge organization format can be rephrased as a learning support format and is a format that can support the user's learning. Examples of knowledge organization formats that clearly identify each of the multiple concepts include outlines, Cornell notes, or comparison tables. By displaying the structured summary 60 of the content source 30 in outline format, the summarization device 10 can organize and display each concept in a hierarchical bulleted list. By displaying the structured summary 60 of the content source 30 in the format of Cornell note-taking, the summarization device 10 can clearly distinguish and display the main points and details of each concept. By displaying the structured summary 60 of the content source 30 in a comparison table as illustrated in Figure 7, the summarization device 10 can visually organize and display, for example, the commonalities and differences between concepts. Figure 7 is a table diagram showing an example of the display format of the structured summary 60 according to Embodiment 1.

[0048] The summarization device 10 may accept the user's desired display format for the structured summary 60 via the input device 1004.

[0049] Furthermore, as will be explained with reference to Figure 8, the structured summary module 50 may, after generating the structured summary 60, further generate structured summaries related to specific content specified by the user based on user instructions (prompts). Figure 8 is a schematic diagram illustrating prompts for the structured summary 60 according to Embodiment 1.

[0050] For example, a user can specify a portion of the structured summary 60 for output. For instance, if a user wants to extract only the summary regarding the data subjects of access rights from the structured summary 60 of the content source 30, they can input a prompt into the input device 1004 specifying the data subjects of access rights as the output. The structured summary module 50 can input this prompt into the language model and generate a structured summary 61 regarding the data subjects of access rights based on the structured summary 60.

[0051] Furthermore, for example, a user can have the system output a response based on the structured summary 60 to the query they entered. For instance, after reviewing the structured summary 60 of the content source 30, if the user wants to learn more about the relationship between processing purposes and types of personal data, they can input a query into the input device 1004 specifying the response regarding the relationship between processing purposes and types of personal data. The structured summary module 50 can input this query into the language model to generate a structured summary 62 that explains the relationship between processing purposes and types of personal data.

[0052] Furthermore, as will be explained with reference to Figure 9, the output unit 70 may extract the portion of the structured summary 60 selected by the user and record (store) it in the data database 80. Figure 9 is a schematic diagram illustrating the storage of the structured summary 60 in the data database 80 according to Embodiment 1.

[0053] The user can select the portion of the structured summary 60 they need via the input device 1004. In the example in Figure 9, the user has selected the structured summary portion 100 of the structured summary 60. The user may also select not only a portion of the structured summary 60, but the entire structured summary 60. Furthermore, the user may select not only the structured summary 60, but also the portion or all of the content source 30 from which the structured summary 60 originated. In the example in Figure 9, the user has selected the video portion 200 and the video portion 210 of the content source 30.

[0054] The output unit 70 may display a user interface on the display device 1005 for the user to select a structured summary 60 or a content source 30.

[0055] The output unit 70 records the structured summary portion 100, video portion 200, and video portion 210, selected by the user, into the data database 80. This allows the summarization device 10 to record the content the user needs, and the recorded content can be used, for example, for learning, reuse in summary generation, etc. The data database 80 may be analyzable by a language model. This enables various analyses and text generation tailored to the individual user's needs using a language model that has analyzed the data database 80.

[0056] Figure 10 is a block diagram showing an example of the hardware configuration of the summarization device 10 according to this disclosure.

[0057] The summarization device 10 comprises, as hardware, a processor 1001, memory 1002, storage 1003, input device 1004, display device 1005, and communication device 1006. The summarization device 10 may be interpreted as a computer or server device.

[0058] The processor 1001 realizes the functions of the summarization device 10 described above by reading a predetermined program from the memory 1002 and executing it. For example, the processing of the acquisition unit 20, element extraction module 40, structured summarization module 50, and output unit 70 described above may be performed by the processor 1001. The processor 1001 may also be interpreted as a Central Processing Unit (CPU), controller, control device, Large Scale Integration (LSI), etc. Furthermore, the processor 1001 may include a Graphics Processing Unit (GPU) and / or a Neural Network Processing Unit (NPU).

[0059] Memory 1002 stores programs and data, etc., for realizing the functions of the summarization device 10 described above. Memory 1002 may be composed of a volatile storage medium and / or a non-volatile storage medium.

[0060] The storage device 1003 stores programs and data, etc., for realizing the functions of the summarization device 10 described above. The storage device 1003 may be composed of a non-volatile storage medium. Examples of the storage device 1003 include a hard disk drive (HDD), a solid state drive (SSD), and flash memory.

[0061] For example, the content source 30, text information 31, visual information 32, metadata 41, structured summary 60, and data database 80 described above may be stored in memory 1002 and / or storage 1003.

[0062] The input device 1004 is a device that receives input from the user. Examples of the input device 1004 include a keyboard, mouse, touchpad, touch panel, and microphone.

[0063] The display device 1005 is a device that displays characters, images, etc. Examples of the display device 1005 include liquid crystal displays and organic EL displays.

[0064] The communication device 1006 is a device that connects the summarization device 10 to a predetermined communication network. Examples of communication networks include wired LAN (e.g., Ethernet®), wireless LAN (e.g., Wi-Fi®), the Internet, mobile communication networks (4G, 5G), and Bluetooth®.

[0065] The user input and display of information described above may also be performed from other terminals (e.g., smartphones, tablet devices, PCs, etc.) via the communication device 1006 and the communication network.

[0066] (Comparative example of output between the conventional technology and this embodiment) The differences between summaries generated by the prior art and summaries generated by this embodiment will be explained below with reference to Figure 11. Figure 11 is a table diagram illustrating the differences between summaries generated by the prior art and summaries generated by Embodiment 1. In the explanation of Figure 11, both the prior art and this embodiment are assumed to generate summaries of content (e.g., content source 30) that includes text information (e.g., text information 31) and visual information (e.g., visual information 32). Furthermore, the text information mainly includes information such as detailed explanations of concepts contained in the content, and the visual information mainly includes information such as relationships between concepts contained in the content.

[0067] The summary S1 generated by conventional technology, which uses only text information, is output in a format that lists the concepts contained in the content and their explanations. In this case, although the summary S1 includes detailed explanations of the concepts contained in the text information, it does not reflect the relationships between concepts shown in the visual information. Therefore, it is difficult to understand the relationships between concepts from the summary S1. Furthermore, when using only text information, conventional technology cannot reflect in the summary S1 information that can only be obtained from visual information, such as the importance of concepts that can be inferred from the presence or absence of emphasis in the visual information.

[0068] The summary S2 generated by conventional technology that uses only visual information outputs information about the concepts contained in the content in a format that reflects the relationships between concepts. However, because the information that can be extracted from visual information is limited, the summary S2 does not include detailed explanations of each concept.

[0069] The summary S3 generated by this embodiment can include information (e.g., metadata 41) that integrates information obtained from text information and information obtained from visual information. Therefore, this embodiment can output a summary S3 that reflects both detailed information for each concept obtained from text information and the interrelationships between concepts obtained from visual information.

[0070] In this embodiment, the generation of metadata (e.g., metadata 41) may be omitted. Even without generating metadata, it is possible to generate a summary S4 that reflects a detailed explanation of the concepts and the interrelationships between them. The summary S4 generated in this way reflects the interrelationships of the concepts contained in the content more than a summary S1 generated by conventional technology that uses only text information, and reflects a detailed explanation of the concepts contained in the content more than a summary S2 generated by conventional technology that uses only visual information. However, if metadata is not used to generate the summary S4, the correspondence between the information extracted from the text information and the information extracted from the visual information may not be accurately reflected in the summary S4. For example, even if the concept extracted from the text information and the concept extracted from the visual information are the same concept, a summary may be generated that makes them appear as if they are different concepts with the same name by chance.

[0071] (Other variations) In the embodiment described above, the element extraction module 40 included visual supplementary information and location information in the metadata 41 and linked the visual supplementary information and location information to the structured concept. However, since location information is used to display the portion of the visual information 32 of the content source 30 that corresponds to the visual supplementary information, if it is not necessary to display such information (i.e., the portion of the visual information 32 that corresponds to the visual supplementary information), the element extraction module 40 may exclude location information from the information to be included in the metadata 41, and the concept structuring unit 51 may omit the linking of location information to the structured concept.

[0072] (Summary of the embodiments) The following technologies are disclosed based on the above description of the embodiments. Note that the components etc. in the above embodiments are examples, but are not limited to these.

[0073] <Technology 1> The summarization device (e.g., summarization device 10) includes: an acquisition unit (e.g., acquisition unit 20) that acquires a content source (e.g., content source 30) that includes text information (e.g., text information 31) and visual information (e.g., visual information 32); an element extraction module (e.g., element extraction module 40) that inputs the content source into a predetermined language model to extract keywords related to each of a plurality of concepts (e.g., concept C1, concept C2, etc.) contained in the content source, and generates detailed content text that explains each of the plurality of concepts, thereby generating metadata (e.g., metadata 41) in which the keywords and detailed content text are linked for each of the plurality of concepts; and a structured summarization module (e.g., structured summarization module 50) that inputs the metadata into a language model to structure the concepts based on the keywords and detailed content text for each of the plurality of concepts, build relationships by linking the related plurality of concepts together, and generate a structured summary of the content source (e.g., structured summary 60) based on the structured plurality of concepts and the built relationships.

[0074] This allows the summarizing device to analyze content sources (content) containing both text and visual information using a language model, generating metadata that links keywords and detailed content text for each of the multiple concepts contained in the content. The summarizing device then analyzes this metadata using the language model, structuring each of the multiple concepts in the content and constructing relationships between them. Based on the structured concepts and established relationships, the summarizing device can then generate a structured summary of the content. This enables the summarizing device to generate a summary that clearly demonstrates the structure of the concepts and the relationships between them.

[0075] <Technology 2> In the summarization device described in Technology 1, the element extraction module extracts information about each of the multiple concepts contained in the content source from the visual information as visual supplementary information, and generates metadata for each of the multiple concepts, with the visual supplementary information further linked to it.

[0076] This allows the summarization device to have a language model analyze the content and generate metadata that includes visually supplementary information contained within the content.

[0077] <Technology 3> In the summarization device described in Technology 2, the structured summarization module associates a structured concept with visually supplementary information related to that concept.

[0078] This allows the summarizing device to link visually supplementary information about a concept to its explanation in a structured summary. This enables the summarizing device to, for example, visually support the user's understanding of that concept.

[0079] <Technology 4> In the summarization device described in Technology 3, the element extraction module generates metadata for each of several concepts, which is linked to visual supplementary information and location information indicating the position of the visual supplementary information within the visual information. The structured summarization module links the visual supplementary information and location information related to the structured concept.

[0080] This allows the summarization device to include location information of visually supplementary information in the metadata. Furthermore, the summarization device can link the visually supplementary information and its location information to the conceptual explanation in the structured summary.

[0081] <Technology 5> In the summarization device described in any one of technologies 1 to 4, the structured summarization module generates a higher-level concept for multiple concepts and links the generated higher-level concept with the multiple concepts.

[0082] This allows the summarization device to generate higher-level concepts for multiple concepts contained in the content. This enables the summarization device to generate structured summaries that more clearly show the relationships between concepts.

[0083] <Technology 6> In the summarization device described in Technology 5, the structured summarization module analyzes whether the relationships between multiple concepts are hierarchical, equivalent, or hierarchical. If there are multiple equivalent concepts for multiple concepts, it generates a higher-level concept for the multiple equivalent concepts and links the generated higher-level concept with the multiple equivalent concepts.

[0084] This allows the summarization device to construct relationships between concepts by analyzing whether the interrelationships of multiple concepts contained in the content are hierarchical, equivalent, or subordinate. Furthermore, if there are multiple equivalent concepts, the summarization device can generate a higher-level concept for those multiple equivalent concepts. This enables the summarization device to generate a structured summary that more clearly shows the relationships between concepts.

[0085] <Technology 7> In the summarization device described in any one of technologies 1 to 6, the structured summarization module generates a structured summary of multiple concepts by structuring multiple concepts, and generates a structured summary of the content source by reconfiguring the structured summaries of multiple concepts so as to maintain the relationships established by the multiple concepts.

[0086] As a result, the summarization device can generate a structured summary of the content by performing a two-stage structuring process: a process that structures multiple concepts contained in the content, and a process that reconstructs the structured multiple concepts based on the relationships between those concepts.

[0087] <Technology 8> The summarization device described in Technology 3 or 4 further comprises an input device for receiving user operations and an output unit for displaying a structured summary of a content source on a display device, wherein the output unit displays the structured summary of the content source on the display device in a hierarchical text format that can be expanded, switches between expanded and collapsed displays of the structured summary of the content source in response to a predetermined user operation, and further displays at least one of one or more concept detail texts and visual supplementary information in the expanded display of the structured summary of the content source.

[0088] This allows the summarization device to display a structured summary of the content in a hierarchical text format that can be expanded. Furthermore, the summarization device can expand or collapse the structured summary in response to user interaction. Expanding the structured summary also allows the summarization device to display further detailed text or visual supplementary information related to the concepts contained within the content.

[0089] <Technology 9> The summarization device described in any one of the technologies 1 to 8 further comprises an output unit that causes a display device to display a structured summary of a content source, the output unit causing the display device to display the structured summary of the content source in one of the following formats: a text format with an expandable hierarchical structure, a diagrammatic format with an expandable hierarchical structure, or a knowledge organization format that clearly shows multiple concepts.

[0090] This allows the summarization device to display the structured summary in the format desired by the user.

[0091] <Technology 10> The summarization device described in any one of the technologies 1 to 9 further comprises an input device that accepts user input, and the structured summarization module, when the input device accepts a prompt or query input by user input, inputs the prompt or query to a language model to generate a structured summary related to specific content specified by the user, based on the structured summarization of the content source.

[0092] This allows the summarization device to accept user prompts or queries and, based on the generated structured summary, further generate and output a structured summary of content specified by the user.

[0093] <Technology 11> The summarization device described in Technology 4 further comprises an input device for receiving user operations and an output unit for displaying a structured summary of a content source on a display device, wherein when the input device receives a predetermined user operation on a concept associated with visual supplementary information in the structured summary of a content source, the output unit displays visual information corresponding to the visual supplementary information in a viewable format on the display device based on the positional information of the visual supplementary information.

[0094] This allows the summarization device to display content with the associated visual supplementary information, for example, when a user clicks on a summary portion of a structured summary that relates to a concept associated with that information.

[0095] <Technology 12> The summarization device described in any one of the technologies 1 to 11 further comprises an input device for receiving user operations and an output unit for displaying a structured summary of a content source on a display device, wherein the output unit records the selected portion in a predetermined database of the user (e.g., a data database 80) when the input device receives a user selection of at least a portion of the structured summary of the content source from the content source and the structured summary of the content source.

[0096] This allows the summarization device to record the portion of the structured summary selected by the user in a database. Furthermore, the summarization device can record the portion of the content selected by the user in the same database. By having a language model analyze this database, various processes (e.g., analysis, text generation, etc.) tailored to the user's needs can be performed using the language model.

[0097] <Technology 13> The summarization method involves acquiring a content source containing text and visual information, inputting the content source into a predetermined language model, extracting keywords for each of the multiple concepts contained in the content source, generating detailed content text that explains each of the multiple concepts, generating metadata that links the keywords and detailed content text for each of the multiple concepts, inputting the metadata into the language model, structuring the concepts based on the keywords and detailed content text for each of the multiple concepts, building relationships by linking related multiple concepts, and generating a structured summary of the content source based on the structured multiple concepts and the built relationships.

[0098] This allows the summarization method to achieve the same effect as technique 1.

[0099] <Technology 14> The summarization program obtains a content source containing text and visual information, inputs the content source into a predetermined language model, extracts keywords for each of the multiple concepts contained in the content source, generates detailed content text that explains each of the multiple concepts, generates metadata linking the keywords and detailed content text for each of the multiple concepts, inputs the metadata into the language model, structures the concepts based on the keywords and detailed content text for each of the multiple concepts, builds relationships by linking the related multiple concepts together, and generates a structured summary of the content source based on the structured multiple concepts and the built relationships, all of which are performed by a computer (e.g., summarization device 10).

[0100] This allows the summarization program to achieve the same effect as Technique 1.

[0101] The functions of the above-described embodiment can also be realized by supplying programs and applications for realizing the functions of the above-described embodiment to a system or device using a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the programs.

[0102] Furthermore, the functions of the above-described embodiment may be implemented by circuits that perform one or more functions (for example, an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)).

[0103] While embodiments have been described above with reference to the attached drawings, this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these are also understood to fall within the technical scope of this disclosure. Furthermore, the components of the embodiments described above can be combined in any way without departing from the spirit of the invention. [Industrial applicability]

[0104] The technology disclosed herein is useful as a summarization device, a summarization method, and a summarization program. [Explanation of Symbols]

[0105] 10. Summarization device 20 Acquisition Department 30 Content Sources 31 Text Information 32 Visual Information 40-element extraction page 41 Metadata 50 Structured Summarization Modules 51 Concept structuring part 52 Knowledge Structure Construction Department 60 Structured summaries 70 Output section 80. Database of materials 1001 Processor 1002 memory 1003 Storage 1004 Input device 1005 Display device 1006 Communication device

Claims

1. A content acquisition unit that acquires content sources including text information and visual information, An element extraction module that, by inputting the content source into a predetermined language model, extracts keywords related to each of the multiple concepts contained in the content source, generates detailed content text that explains each of the multiple concepts, and generates metadata for each of the multiple concepts that links the keywords and the detailed content text; The system includes a structured summarization module that, by inputting the metadata into the language model, structures each of the multiple concepts based on the keywords and the detailed content text, builds relationships by linking the related multiple concepts together, and generates a structured summary of the content source based on the structured multiple concepts and the built relationships. Summarization device.

2. The aforementioned element extraction module is From the aforementioned visual information, information relating to each of the multiple concepts included in the content source is extracted as visual supplementary information. For each of the aforementioned multiple concepts, the metadata is generated, to which the visual supplementary information is further linked. The summarization device according to claim 1.

3. The aforementioned structured summary module is: The structured concept is linked to the visual supplementary information related to that concept. The summarization device according to claim 2.

4. The aforementioned element extraction module is For each of the aforementioned multiple concepts, metadata is generated in which the visual supplementary information and location information indicating the position of the visual supplementary information in the visual information are linked. The aforementioned structured summary module is: The structured concept is linked to the visual supplementary information and location information related to that concept. The summarization device according to claim 3.

5. The aforementioned structured summary module is: Generate a higher-level concept for the aforementioned multiple concepts, and link the generated higher-level concept with the aforementioned multiple concepts. The summarization device according to claim 1.

6. The aforementioned structured summary module is: We analyze whether the relationships between the aforementioned multiple concepts are superior, equal, or subordinate. When there are multiple equivalent concepts in the aforementioned multiple concepts, a superordinate concept is generated for the aforementioned multiple equivalent concepts. The above-generated higher-level concept is linked to the above-mentioned multiple equivalent concepts. The summarization device according to claim 5.

7. The aforementioned structured summary module is: By structuring the aforementioned multiple concepts, a structured summary of the aforementioned multiple concepts is generated. The structured summary of the content source is generated by reconstructing the structured summary of the multiple concepts so that the multiple concepts maintain the relationships established above. The summarization device according to claim 1.

8. The system further comprises an input device for receiving user input and an output unit for displaying a structured summary of the content source on a display device, The output unit outputs to the display device, The structured summary of the aforementioned content source is displayed in a hierarchical text format that can be expanded. In response to a predetermined user operation, the system switches between displaying the structured summary of the content source in an expanded and collapsed state. In the expanded display of the structured summary of the content source, at least one of the content detail text and the visual supplementary information of one or more concepts is further displayed. The summarization device according to claim 3.

9. The device further includes an output unit that causes the display device to display a structured summary of the content source, The output unit is, The structured summary of the content source is displayed on the display device in one of the following formats: a text format with an expandable hierarchical structure, a diagrammatic format with an expandable hierarchical structure, or a knowledge organization format that clearly identifies the multiple concepts. The summarization device according to claim 1.

10. It is further equipped with an input device that accepts user input, The aforementioned structured summary module is: When the input device receives a prompt or query input from the user, it inputs the prompt or query into the language model to generate a structured summary related to specific content specified by the user, based on the structured summary of the content source. The summarization device according to claim 1.

11. The system further comprises an input device for receiving user input and an output unit for displaying a structured summary of the content source on a display device, The output unit is, When the input device receives a predetermined user operation on the concept to which the visual supplementary information in the structured summary of the content source is associated, it displays the visual information corresponding to the visual supplementary information in a viewable format on the display device, based on the positional information of the visual supplementary information. The summarization device according to claim 4.

12. The system further comprises an input device for receiving user input and an output unit for displaying a structured summary of the content source on a display device, The output unit is, When the input device receives a user selection of at least a portion of the structured summary of the content source from the content source and the structured summary of the content source, it records the selected portion in the user's predetermined database. The summarization device according to claim 1.

13. Retrieve content sources that include text and visual information, By inputting the content source into a predetermined language model, keywords related to each of the multiple concepts contained in the content source are extracted, and detailed content text explaining each of the multiple concepts is generated, and metadata is generated for each of the multiple concepts, linking the keywords and the detailed content text. By inputting the metadata into the language model, the system structures each of the multiple concepts based on the keywords and the detailed content text, builds relationships by linking related concepts together, and generates a structured summary of the content source based on the structured multiple concepts and the built relationships. How to summarize.

14. Retrieve content sources that include text and visual information, By inputting the content source into a predetermined language model, keywords related to each of the multiple concepts contained in the content source are extracted, and detailed content text explaining each of the multiple concepts is generated, and metadata is generated for each of the multiple concepts, linking the keywords and the detailed content text. By inputting the metadata into the language model, the system structures each of the multiple concepts based on the keywords and the detailed content text, builds relationships by linking related concepts together, and generates a structured summary of the content source based on the structured multiple concepts and the built relationships. A summary program that instructs a computer to perform a task.

Citation Information

Patent Citations

  • Content generation system

    WO2024218839A1