Information extraction method, program, and information extraction device

Abstract

This information extraction method, which is executed by a computer and extracts information contained in an image, comprises: an element extraction step for extracting a plurality of elements constituting an image; a localization step for determining a plurality of clusters on the basis of the plurality of elements extracted in the element extraction step; a splitting step for splitting the image into a plurality of split images on the basis of the plurality of clusters determined in the localization step; and an information extraction step for using a language model to extract, for each one of the plurality of split images obtained from splitting the image in the splitting step, information contained in the split image.

Description

Information Extraction Method, Program, and Information Extraction Device

[0001] The present disclosure relates to an information extraction method and the like.

[0002] Conventionally, techniques for extracting information contained in images have been disclosed (see, for example, Non-Patent Document 1). In Non-Patent Document 1, a technique for extracting the types and numbers of objects contained in construction drawings using a machine learning model is disclosed.

[0003] Laura Jamieson, et al., "Towards fully automated processing and analysis of construction diagrams: AI-powered symbol detection", [online], July 25, 2024, International Journal on Document Analysis and Recognition, Internet (https: / / link.springer.com / article / 10.1007 / s10032 - 024 - 00492 - 9)

[0004] When extracting information from an indefinite image as in the technique disclosed in Non-Patent Document 1, there is a problem that accurate information extraction cannot be performed when the image size is large or the image is complex.

[0005] Therefore, the present disclosure provides an information extraction method and the like that can more accurately extract information contained in an image.

[0006] One aspect of the information extraction method according to the present disclosure is an information extraction method for extracting information contained in an image, which is executed by a computer, and includes an element extraction step of extracting a plurality of elements constituting the image, a localization step of determining a plurality of clusters based on the plurality of elements extracted in the element extraction step, a division step of dividing the image into a plurality of divided images based on the plurality of clusters determined in the localization step, and an information extraction step of extracting information contained in each of the plurality of divided images of the image divided in the division step using a language model.

[0007] Furthermore, one aspect of the program relating to this disclosure is a program that causes a computer to execute the above-mentioned information extraction method.

[0008] Furthermore, one embodiment of the information extraction device according to the present disclosure is an information extraction device for extracting information contained in an image, comprising: an element extraction unit for extracting a plurality of elements constituting the image; a localization unit for determining a plurality of clusters based on the plurality of elements extracted by the element extraction unit; a division unit for dividing the image into a plurality of divided images based on the plurality of clusters determined by the localization unit; and an information extraction unit for extracting information contained in each of the plurality of divided images of the image divided by the division unit using a language model.

[0009] The information extraction methods described herein can extract information contained in images more accurately.

[0010] Figure 1 is a block diagram showing the configuration of the information extraction system according to the embodiment. Figure 2 is a sequence diagram illustrating the operation of the information extraction system according to the embodiment. Figure 3 is a diagram showing an example of a data database according to the embodiment. Figure 4 is a diagram showing an example of a partitioning criterion table according to the embodiment. Figure 5 is a diagram showing an example of an asset list according to the embodiment. Figure 6 is a diagram showing an example of an image after clustering according to the embodiment. Figure 7 is a flowchart illustrating a detailed example of the operation of step S103 in Figure 2. Figure 8 is a flowchart illustrating a detailed example of the operation of step S204 in Figure 7. Figure 9 is a flowchart illustrating another detailed example of the operation of step S204 in Figure 7. Figure 10 is a flowchart illustrating yet another detailed example of the operation of step S204 in Figure 7. Figure 11 is a flowchart illustrating yet another detailed example of the operation of step S204 in Figure 7.

[0011] (Knowledge forming the basis of this disclosure) Conventionally, techniques for extracting information contained in images have been disclosed (see, for example, Non-Patent Document 1). Non-Patent Document 1 discloses a technique for extracting the types and number of objects contained in construction drawings using a machine learning model.

[0012] When extracting information from an undefined image, as in the technology disclosed in Non-Patent Document 1, there is a problem in that it is not possible to accurately extract information when the image size is large or the image is complex.

[0013] Therefore, the inventors devised a technique to extract information contained in an image more accurately by extracting the elements that make up the image, localizing the elements using clustering, dividing the image so that the localized parts fit inside, and then providing the divided images as input images to a VLLLM (Vision Large Language Model) to extract information.

[0014] More specifically, the information extraction method according to the first embodiment is an information extraction method for extracting information contained in an image, which is performed by a computer and includes: an element extraction step for extracting a plurality of elements constituting the image; a localization step for determining a plurality of clusters based on the plurality of elements extracted in the element extraction step; a division step for dividing the image into a plurality of divided images based on the plurality of clusters determined in the localization step; and an information extraction step for extracting information contained in each of the plurality of divided images of the image divided in the division step, using a language model.

[0015] This allows for the clustering of the elements that make up the image, and by extracting information from each of the divided images based on the clustering results, it is possible to suppress information omissions. Therefore, this allows for more accurate extraction of information contained in the image.

[0016] Furthermore, the information extraction method according to the second embodiment is an information extraction method according to the first embodiment, further including an output step of outputting information contained in the image based on the information extracted from each of the plurality of divided images in the information extraction step.

[0017] This allows information contained in the entire image to be extracted by integrating information extracted from multiple segmented images.

[0018] Furthermore, the information extraction method according to the third embodiment is an information extraction method according to the first or second embodiment, wherein the plurality of elements constituting the image include information about objects included in the image, and in the element extraction step, the position of the objects included in the image is extracted.

[0019] This allows for the proper determination of clusters by extracting the locations of objects contained in the image.

[0020] Furthermore, the information extraction method according to the fourth embodiment is an information extraction method according to any of the first to third embodiments, wherein the plurality of elements constituting the image include character information contained in the image, and in the element extraction step, the character information contained in the image is extracted.

[0021] This allows for the proper determination of clusters by extracting textual information contained in the image.

[0022] Furthermore, the information extraction method according to the fifth embodiment is an information extraction method according to any of the first to fourth embodiments, wherein in the element extraction step, the positions of each of the plurality of elements are extracted, and in the localization step, the plurality of clusters are determined based on the positions of each of the plurality of elements.

[0023] This allows for the appropriate determination of clusters based on the positions of the elements that make up the image. These elements include, for example, objects or textual information contained within the image.

[0024] Furthermore, the information extraction method according to the sixth embodiment is an information extraction method according to the fifth embodiment, wherein in the localization step, the plurality of clusters are determined such that one or more elements located in a region in the image where the density of the elements is higher than a first threshold are included in one of the plurality of clusters.

[0025] This allows for the determination of clusters so that areas with a high concentration of information in an image are included in a single cluster, thereby suppressing the occurrence of information omissions.

[0026] Furthermore, the information extraction method according to the seventh embodiment is an information extraction method according to any of the first to sixth embodiments, wherein the element includes information indicating a predetermined region in the image, and in the localization step, the plurality of clusters are determined such that one or more elements included in the region indicated by the information indicating the predetermined region are included in one of the plurality of clusters.

[0027] This allows for more efficient information extraction from images by determining clusters such that regions indicated by information representing a predetermined area in the image are included in a single cluster.

[0028] Furthermore, the information extraction method according to the eighth embodiment is an information extraction method according to the fourth embodiment, wherein in the localization step, if the character information extracted in the element extraction step includes character information indicating an important object, a cluster centered on the coordinates of the important object is determined.

[0029] This allows for the identification of clusters that focus on textual information indicating important objects, thereby reducing the likelihood of missing important objects.

[0030] Furthermore, the information extraction method according to the ninth embodiment is an information extraction method according to the eighth embodiment, wherein the important object is an object that includes external connection information.

[0031] This allows us to determine clusters that focus on objects containing external connectivity information.

[0032] Furthermore, the information extraction method according to the tenth embodiment is an information extraction method according to the eighth embodiment, wherein the important object is an object related to network equipment or network information.

[0033] This makes it possible to determine clusters that focus on objects related to network devices or network information.

[0034] Furthermore, the information extraction method according to the 11th embodiment is an information extraction method according to the 8th embodiment, wherein the important object is an object that is inferred to be the center of control.

[0035] This allows us to determine clusters that focus on objects that are inferred to be the center of control.

[0036] Furthermore, the information extraction method according to the 12th embodiment is an information extraction method according to the 4th embodiment, wherein in the localization step, if the character information extracted in the element extraction step includes character information indicating a plurality of related objects, a cluster is determined such that the plurality of related objects are included in a single cluster.

[0037] This allows for more efficient information extraction from images by determining clusters so that multiple related objects are included in the same cluster.

[0038] Furthermore, the information extraction method according to the 13th embodiment is an information extraction method according to the 12th embodiment, wherein the character information indicating the related plurality of objects is a plurality of IP addresses in which characters within a predetermined range match.

[0039] This allows for determining which clusters belong to which objects with multiple IP addresses whose characters within a given range match are located.

[0040] Moreover, the information extraction method according to the 14th aspect is the information extraction method according to the 12th aspect, wherein the character information indicating the plurality of related objects is an identification number for identifying objects having similarity.

[0041] Thereby, it is possible to determine the cluster so that the objects having the identification numbers for identifying the objects having similarity are included in the same cluster.

[0042] Moreover, the information extraction method according to the 15th aspect is the information extraction method according to the 12th aspect, wherein the character information indicating the plurality of related objects is information indicating communication using the same communication protocol.

[0043] Thereby, it is possible to determine the cluster so that the objects communicating using the same communication protocol are included in the same cluster.

[0044] Moreover, the information extraction method according to the 16th aspect is the information extraction method according to any one of the 1st to 15th aspects, wherein in the dividing step, the image is divided into a plurality of divided images such that all of the one or more elements included in one of the plurality of clusters are included in the same divided image.

[0045] Thereby, since the image can be divided so that the elements included in the same cluster are included in the same divided image, it is possible to suppress the occurrence of omission of information extraction from the image.

[0046] Moreover, the information extraction method according to the 17th aspect is the information extraction method according to the 2nd aspect, wherein in the output step, the information included in the image is output in a list format.

[0047] Thereby, it is possible to assist the user in confirming the information extracted from the image.

[0048] Further, the information extraction method according to the 18th aspect is the information extraction method according to the 2nd aspect or the 17th aspect, and in the output step, information indicating the position occupied by the information included in the image in the image is output in association with the information included in the image. This is an information extraction method.

[0049] This can assist the user in confirming whether information is accurately extracted from the image.

[0050] Further, the program according to the 19th aspect is a program for causing a computer to execute the information extraction method according to any one of the 1st to 18th aspects.

[0051] This can assist in more accurately extracting the information included in the image.

[0052] Further, the information extraction device according to the 20th aspect is an information extraction device that extracts information included in an image, and includes an element extraction unit that extracts a plurality of elements constituting the image, and based on the plurality of elements extracted by the element extraction unit, a localization unit that determines a plurality of clusters, a division unit that divides the image into a plurality of divided images based on the plurality of clusters determined by the localization unit, and for each of the plurality of divided images of the image divided by the division unit, an information extraction unit that extracts the information included in the divided image using a language model. This is an information extraction device.

[0053] This can suppress information extraction omission by clustering the elements constituting the image and extracting the information included in each of the divided images divided based on the clustering result. Therefore, this can more accurately extract the information included in the image.

[0054] These general or specific aspects may be implemented by a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or may be implemented by any combination of a system, method, integrated circuit, computer program, and recording medium. The recording medium may also be a non-temporary recording medium.

[0055] (Embodiments) Hereinafter, embodiments of the information extraction method, program, and information extraction device relating to this disclosure will be described in detail with reference to the drawings. The embodiments described below are all specific examples of this disclosure. The numerical values, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. Furthermore, among the components in the following embodiments, components not described in an independent claim will be described as optional components.

[0056] Furthermore, each figure is a schematic diagram and not necessarily a strictly accurate representation. In each figure, substantially identical components are given the same reference numerals, and redundant explanations may be omitted or simplified.

[0057] [Configuration] First, the configuration of the information extraction system according to this embodiment will be described. Figure 1 is a block diagram showing the configuration of the information extraction system 10 according to this embodiment.

[0058] As shown in Figure 1, the information extraction system 10 comprises an information extraction device 20, an information terminal 30, and a VLLLM 40. The information extraction device 20 and the information terminal 30 are connected to each other in a way that allows them to communicate with one another. The information extraction device 20 and the VLLLM 40 are also connected to each other in a way that allows them to communicate with one another. The information extraction device 20 and the information terminal 30 may be the same single device. The information terminal 30 may also be a device that executes an application program that implements the functions of the information extraction device 20.

[0059] The information extraction system 10 is a system that extracts information contained in an image using a large-scale language model. The image is, for example, a network configuration diagram of a communication system (not shown). The information contained in the image is, for example, information for identifying the devices included in the network configuration diagram.

[0060] The user of the information extraction system 10 is, for example, a Security Operation Center (SOC) analyst. When a security incident or abnormal activity is detected in the monitored communication system, the SOC analyst takes action to address the security incident or abnormal activity. The user of the information extraction system 10 monitors the communication system using, for example, a list of devices that can identify the devices included in the monitored communication system.

[0061] The information extraction device 20 is a device that extracts information contained in an image using the VLLLM 40. More specifically, the information extraction device 20 creates input information for the VLLLM 40. This input information is generated so that the information contained in the image can be extracted more accurately when input to the VLLLM 40. The information extraction device 20 is implemented, for example, by a server device. Specifically, the information extraction device 20 comprises a communication unit 21, an information processing unit 22, and a storage unit 23.

[0062] The communication unit 21 is a communication circuit (communication module) for the information extraction device 20 to communicate with the information terminal 30 and the VLLLM 40. The communication standard used for communication by the communication unit 21 is not particularly limited.

[0063] The information processing unit 22 performs information processing and other operations necessary for the information extraction device 20 to operate. The information processing unit 22 is implemented, for example, by a microcomputer, but may also be implemented by a processor or dedicated circuit. The functions of the information processing unit 22 are realized by the hardware, such as a microcomputer or processor that constitutes the information processing unit 22, executing a computer program (software) stored in the storage unit 23. Specifically, the information processing unit 22 includes an acquisition unit 221, an element extraction unit 222, a localization unit 223, a division unit 224, and an information extraction unit 225.

[0064] The acquisition unit 221 acquires an image. The image is, for example, a network configuration diagram of the communication system being monitored by the user.

[0065] The element extraction unit 222 extracts multiple elements that constitute the image acquired by the acquisition unit 221. The elements that constitute the image are objects contained in the image or text information contained in the image. The element extraction unit 222 extracts multiple elements that constitute the image, as well as information related to those multiple elements.

[0066] Specifically, the element extraction unit 222 extracts the positions of objects contained in the image. For example, the element extraction unit 222 extracts the positions and outlines of objects contained in the image by performing segmentation on the image.

[0067] Furthermore, the element extraction unit 222 extracts character information contained in the image. For example, the element extraction unit 222 extracts character information contained in the image and the position coordinates of said character information by performing optical character recognition (OCR) on the image.

[0068] The localization unit 223 determines multiple clusters based on the multiple elements extracted by the element extraction unit 222. A cluster is a group of elements that classify the extracted elements based on the content they represent and their positions in the image.

[0069] The division unit 224 divides the image into multiple divided images based on the multiple clusters determined by the localization unit 223. Specifically, the division unit 224 divides the image into multiple divided images such that all elements included in one of the determined clusters are included in the same divided image.

[0070] Furthermore, there may be overlapping areas between multiple divided images. In other words, the original image may contain areas that are included in two or more of the divided images. This helps to prevent the loss of elements that make up the image when it is divided into divided images.

[0071] The information extraction unit 225 extracts information contained in each of the multiple divided images obtained by the division unit 224, using a language model. Furthermore, the information extraction unit 225 outputs information contained in the image based on the information extracted from the multiple divided images.

[0072] The storage unit 23 is a memory device that stores various programs necessary for the operation of the information extraction device 20. The storage unit 23 is implemented, for example, by a semiconductor memory.

[0073] The information terminal 30 is an information terminal used by a user. The information terminal 30 is implemented, for example, by a personal computer. The information terminal 30 may also be a tablet terminal or a smartphone. The information terminal 30 is equipped with a user interface (not shown) for receiving user input and presenting various information to the user.

[0074] VLLM40 is a large-scale language model that generates responses to image information and prompt inputs. VLLM40 is an example of a language model. VLLM40 is a large-scale language model that uses RAG (Retrievable-Augmented Generation) which refers to various materials stored in the memory unit 23.

[0075] In this embodiment, the language model was implemented using VLLM40, but the language model is not limited to VLLM; it may also be VLM (Vision-Language Model) or MLLM (Multimodal Large Language Model).

[0076] [Overall Operation] Next, the overall operation of the information extraction system 10 according to this embodiment will be described. The information extraction system 10 extracts information contained in an image using the VLLLM 40. Furthermore, the information extraction system 10 extracts the elements that make up the image, performs clustering to localize the elements, divides the image so that the localized parts fit inside, and then inputs it to the VLLLM 40, thereby enabling more accurate extraction of information contained in the image. Figure 2 is a sequence diagram illustrating the operation of the information extraction system 10 according to this embodiment.

[0077] First, the information terminal 30 acquires a network configuration diagram related to the monitored object (S101). For example, the information terminal 30 receives input from the user for a network configuration diagram of the communication system to be monitored and acquires the network configuration diagram. The network configuration diagram is an image containing information about multiple devices included in the communication system. The network configuration diagram may also be an image containing information showing the communication connections between devices and the arrangement of devices in the communication system. Furthermore, the network configuration diagram may consist of multiple images. Figure 3 shows an example of a data database D1 according to an embodiment.

[0078] As shown in Figure 3, in the data database D1, each network configuration diagram is managed by associating it with a "monitoring target tag" and a "file name." The "monitoring target tag" is information used to identify the monitoring target. The "file name" is information indicating the name of the network configuration diagram.

[0079] For example, in the data database D1 shown in Figure 3, a network configuration diagram whose "file name" stored in the storage unit 23 is "NW configuration diagram 1st floor_A1.jpeg" indicates that it is a network configuration diagram that monitors "Building A".

[0080] In the example shown in Figure 3, when generating a list of devices included in the communication system installed in "Building A" from a network configuration diagram related to Building A, the information terminal 30 obtains two network configuration diagrams: one with the file name "NW Configuration Diagram 1st Floor_A1.jpeg" and another with the file name "Entire Network Configuration Diagram.jpeg".

[0081] The information terminal 30 transmits the network configuration diagram to the information extraction device 20 (S102). The network configuration diagram transmitted by the information terminal 30 is received by the communication unit 21 of the information extraction device 20 and acquired by the acquisition unit 221.

[0082] Once the network configuration diagram is acquired, the information processing unit 22 of the information extraction device 20 performs preprocessing of the network configuration diagram (S103). Preprocessing of the network configuration diagram is performed on the network configuration diagram before input to the VLLLM 40 in order to enable the VLLLM 40 to extract the information contained in the network configuration diagram more accurately.

[0083] For example, the preprocessing of a network diagram involves extracting the elements that make up the network diagram, clustering the extracted elements, and then dividing the network diagram based on the clustering results. Clustering is the process of determining clusters based on multiple elements.

[0084] Here, for example, the following segmentation criterion table shows the clustering criteria used for segmenting images. Figure 4 is a diagram showing an example of segmentation criterion table D2 according to an embodiment. In segmentation criterion table D2, the type of image segmentation, the perspective of image segmentation, and the information source used for said image segmentation are associated.

[0085] For example, in the segmentation criterion table D2 shown in Figure 4, if it is determined that the network configuration diagram contains an object that includes external connection information based on the string information obtained by the image's OCR, the image is segmented in order to identify important terminals in SOC analysis.

[0086] The information extraction device according to this embodiment performs preprocessing of the network configuration diagram according to the division criteria shown in the division criteria table D2 in Figure 4. The processing in step S103 in Figure 2 will be described in detail later.

[0087] Next, the information extraction unit 225 creates an information extraction prompt (S104). The information extraction prompt is, for example, a prompt to cause the VLLLM 40 to generate an asset list related to the devices included in the network configuration diagram.

[0088] Next, the information extraction unit 225 transmits the pre-processed network configuration diagram and the information extraction prompt to the VLLLM 40 using the communication unit 21 (S105). In other words, the information extraction unit 225 provides the pre-processed network configuration diagram and the information extraction prompt as input to the VLLLM 40.

[0089] When a pre-processed network diagram and an information extraction prompt are input, the VLLLM 40 extracts information based on the prompt and generates multiple asset lists (S106). Specifically, the VLLLM 40 extracts information from each of the network diagrams input based on the prompt and generates an asset list for each network diagram. An asset list is a list of devices included in the network diagram. If there is only one network diagram, one asset list will be generated.

[0090] When multiple asset lists are generated, the VLLLM 40 transmits the multiple asset lists to the information extraction device 20 (S107). In other words, the VLLLM 40 outputs the generated multiple asset lists to the information extraction device 20.

[0091] When multiple asset lists are received, the information extraction device 20 combines the obtained asset lists into one (S108). For example, if there are duplicate devices in the acquired asset lists, the information extraction unit 225 of the information extraction device 20 combines the multiple asset lists into one asset list by resolving the duplicate devices. This creates an asset list related to the monitored items. Since the extracted information in the asset list includes information indicating the location where the devices were placed in the network configuration diagram, the information extraction unit 225 can identify duplicate devices. Furthermore, since the extracted information in the asset list includes unique device information, the information extraction unit 225 can identify duplicate devices.

[0092] Next, the information extraction unit 225 transmits an asset list related to the monitored object to the information terminal 30 using the communication unit 21 (S109). The asset list related to the monitored object transmitted by the information extraction device 20 is received by the information terminal 30.

[0093] When an asset list relating to the monitored items is received, the information terminal 30 presents the asset list to the user (S110). The user confirms the asset list displayed on the information terminal 30. Figure 5 shows an example of an asset list D3 according to the embodiment.

[0094] As shown in Figure 5, Asset List D3 is a list of devices included in the network configuration diagram. In Asset List D3, the "number," "IP address," "device name," "location," "management vendor," and "protocol" are associated. The "number" is an ID used to identify the device. The "IP address" is the IP address that the device uses to communicate in the communication system. The "device name" is the name used to identify the device in the network configuration diagram. The "location" is where the device is installed. The "management vendor" is the company that supplied the device. The "protocol" is the communication protocol that the device uses for communication.

[0095] In this way, the information extraction unit 225 outputs the information contained in the image (network diagram) in a list format. This helps the user to confirm the information extracted from the image.

[0096] Furthermore, the information extraction unit 225 may highlight and display devices that are presumed to play an important role in the monitored communication system from among the multiple devices shown in asset list D3. Highlighting means, for example, displaying them in a manner that changes the font size, font color, and background color.

[0097] Furthermore, the information extraction unit 225 can output information indicating the position of the information contained in the image, in association with the information contained in the image. For example, the information extraction unit 225 outputs an asset list related to the monitored object in association with the clustered image. Figure 6 is a diagram showing an example of the clustered image D4 according to the embodiment.

[0098] In the clustered image D4 shown in Figure 6, the elements included in each of the determined clusters are displayed with different hatching. In other words, elements with the same hatching indicate that they belong to the same cluster. In the clustered image D4 shown in Figure 6, five clusters have been determined.

[0099] For example, if a user selects a device from the asset list D3, the clustered image D4 will show which cluster the device belongs to.

[0100] In this way, by outputting information indicating the location of the information contained in an image, and associating it with the information contained in the image, it is possible to help the user verify whether or not the information has been accurately extracted from the image.

[0101] [Pre-processing operation] The process of step S103 in Figure 2 will be explained in detail below using Figure 7. Figure 7 is a flowchart illustrating an example of the detailed operation of step S103 in Figure 2.

[0102] First, the acquisition unit 221 acquires image information related to the monitored object (S201). The image information related to the monitored object is, for example, a network configuration diagram of the monitored object.

[0103] Next, the information processing unit 22 determines whether the image size (number of pixels) of the acquired image information is less than 1000 x 1000 (S202).

[0104] If it is determined that the image size is less than 1000 x 1000 (Yes in S202), the information processing unit 22 does not perform preprocessing on the image information relating to the monitored object (S203). In other words, if the image size is small enough that information can be accurately extracted from the image information without performing preprocessing, the image is used as the input image for the VLLLM 40 without preprocessing.

[0105] If it is determined that the image size is 1000 x 1000 or larger (No in S202), the information processing unit 22 divides the image information related to the monitored object (S204). This reduces the image size to a degree that allows the VLLLM 40 to accurately extract information from the image information. The processing in step S204 of Figure 7 will be described in more detail later.

[0106] By performing this preprocessing, the VLLLM40 can extract information contained in the image information more accurately.

[0107] [Clustering Operation Example 1] Next, the process of step S204 in Figure 7 will be explained in detail using Figure 8. Figure 8 is a flowchart illustrating an example of the detailed operation of step S204 in Figure 7.

[0108] First, the element extraction unit 222 performs segmentation on the image and generates an image in which objects in the image are identified (S301). This generates an image in which the outlines and coordinates of the objects contained in the image are extracted.

[0109] Next, the element extraction unit 222 identifies a standard number of clusters from the image size (S302). For example, the image size and the standard number of clusters are associated with a pre-created table, and the element extraction unit 222 identifies the standard number of clusters by referring to this table.

[0110] Next, the element extraction unit 222 extracts character information and character coordinates from the image using OCR (S303).

[0111] Subsequently, the localization unit 223 determines whether or not character information related to external connections is included in the image (S304). Character information related to external connections is, for example, information indicating a public network such as the Internet. The localization unit 223 determines that character information related to external connections is included in the image if the character information extracted in the processing of step S303 includes character information related to external connections.

[0112] If it is determined that the image does not contain character information related to external connections (No in S304), the localization unit 223 performs clustering on the image with a standard number of clusters, identifying the objects in the image (S305).

[0113] Furthermore, at this time, multiple clusters may be determined based on the positions of each of the multiple elements extracted in the processing of step S301. Specifically, multiple clusters may be determined based on the density of multiple elements in the image calculated from the positions of multiple elements in the image. More specifically, multiple clusters may be determined such that one or more elements located in a region where the density of elements in the image is higher than a first threshold are included in one of the multiple clusters. The first threshold is, for example, 10 elements per 100 pixels. The first threshold is not particularly limited. This makes it possible to determine clusters such that regions where information is densely concentrated in the image are included in one of the clusters, thereby suppressing the occurrence of information omissions.

[0114] If it is determined that the image contains character information related to an external connection (Yes in S304), the localization unit 223 acquires the coordinate information of the character information related to the external connection (S306).

[0115] Next, the number of coordinate pieces is added to the number of clusters (S307). In other words, the number of clusters is increased by the number of character pieces related to external connections to the number of clusters identified in step S302.

[0116] Next, the localization unit 223 designates the coordinate information as the central coordinate and performs clustering on the image from which the object has been identified (S308). The localization unit 223 determines a cluster centered on the coordinates of the object containing the external connection information. In other words, the localization unit 223 determines a cluster focused on the object containing the external connection information. To put it another way, if the image contains textual information indicating an important object, the localization unit 223 determines a cluster centered on the coordinates of the important object.

[0117] When clustering is performed on an image in which objects have been identified, the splitting unit 224 divides the image based on the clustering results so that the objects are contained within it (S309). In other words, the splitting unit 224 divides the image into multiple split images such that all one or more elements contained in one of the clusters are included in the same split image. This allows the image to be divided so that elements contained in the same cluster are included in the same split image, thereby suppressing the occurrence of information being missed during extraction from the image.

[0118] In this way, by performing clustering, it is possible to determine clusters that focus on character information indicating important objects, thereby suppressing the occurrence of missing important objects.

[0119] In step S304 of Figure 8, it was determined whether or not character information related to external connections is included in the image, but other determinations may be made.

[0120] For example, it may be determined whether or not the image contains textual information indicating objects related to network devices or network information. This makes it possible to determine clusters that focus on objects related to network devices or network information.

[0121] Alternatively, for example, it may be determined whether or not the image contains textual information indicating an object that is presumed to be the control center. This makes it possible to determine a cluster that focuses on the object that is presumed to be the control center. An object that is presumed to be the control center is an object that indicates the device that is presumed to be the control center of the monitored communication system, for example, an object that indicates a large number of devices that communicate with that device.

[0122] [Clustering Operation Example 2] Next, we will explain another example of the detailed operation of step S204 in Figure 7. Figure 9 is a flowchart illustrating another example of the detailed operation of step S204 in Figure 7.

[0123] First, the element extraction unit 222 performs segmentation on the image and generates an image in which objects in the image have been identified (S401).

[0124] Next, the element extraction unit 222 identifies a standard number of clusters from the image size (S402).

[0125] Next, the element extraction unit 222 extracts character information and character coordinates from the image using OCR (S403).

[0126] Subsequently, the localization unit 223 determines whether or not the image contains an IP address (S404).

[0127] If it is determined that the image does not contain an IP address (No in S404), the localization unit 223 performs clustering on the image with a standard number of clusters, identifying the objects in the image (S405).

[0128] If it is determined that the image contains an IP address (Yes in S404), a set of IP addresses that match up to the third octet is extracted (S406). If the IP addresses match up to the third octet, these devices are presumed to be related to each other.

[0129] The localization unit 223 calculates the center coordinates for each extracted set of IP addresses (S407).

[0130] Next, the number of central pieces of information is added to the number of clusters (S408). In other words, the number of clusters is increased by the number of sets of IP addresses that match up to the third octet, relative to the number of clusters identified in step S402.

[0131] Next, the localization unit 223 specifies the center coordinates of the IP addresses and performs clustering on the images in which objects have been identified (S409). In other words, it determines a cluster centered on the center coordinates of multiple objects whose IP addresses match within a predetermined range of characters. To put it another way, when the image contains character information indicating multiple objects related to the image, the localization unit 223 determines a cluster such that the multiple related objects are included in a single cluster.

[0132] When clustering is performed on an image in which objects have been identified, the splitting unit 224 divides the image based on the clustering results so that the objects are contained within it (S410).

[0133] By performing clustering in this way, information can be extracted from images more efficiently by determining clusters so that multiple related objects are included in the same cluster.

[0134] In step S404 of Figure 9, it was determined whether or not the image contains an IP address, but other determinations may also be made.

[0135] For example, it may be determined whether an image contains an identification number for identifying similar objects. The identification number is, for instance, information set by the user when registering the device with the system, used to identify the device within the system. This allows for the determination of clusters such that objects with identification numbers for identifying similar objects are included in the same cluster.

[0136] Alternatively, for example, it may be determined whether or not the image contains text information indicating that communication is being conducted using the same communication protocol. This makes it possible to determine clusters so that objects communicating using the same communication protocol are included in the same cluster.

[0137] [Clustering Operation Example 3] Next, we will explain yet another example of the detailed operation of step S204 in Figure 7. Figure 10 is a flowchart illustrating yet another example of the detailed operation of step S204 in Figure 7.

[0138] First, the element extraction unit 222 performs segmentation on the image and generates an image in which objects in the image have been identified (S501).

[0139] Next, the element extraction unit 222 identifies a standard number of clusters from the image size (S502).

[0140] Next, the element extraction unit 222 extracts character information and character coordinates from the image using OCR (S503).

[0141] Subsequently, the localization unit 223 determines whether or not the image is a color image (S504).

[0142] If the image is determined not to be a color image (No in S504), the localization unit 223 performs clustering on the image with a standard number of clusters, identifying the objects in the image (S505).

[0143] If the image is determined to be a color image (Yes in S504), the color information surrounding the object is acquired (S506). The color information surrounding the object is an example of information that indicates a predetermined area.

[0144] The localization unit 223 groups the objects according to the acquired color information and calculates the center coordinates for each group (S507).

[0145] Next, the number of central pieces of information is added to the number of clusters (S508). The localization unit 223 specifies the central coordinates of the object group and performs clustering on the image in which the objects of the image have been identified (S509). In other words, multiple clusters are determined such that one or more elements included in the region indicated by the information indicating a predetermined region are included in one of the multiple clusters.

[0146] When clustering is performed on an image in which objects have been identified, the splitting unit 224 divides the image based on the clustering results so that the objects are contained within it (S510).

[0147] By performing clustering in this way, it is possible to determine clusters in an image such that regions indicated by information representing a given area are included in one cluster, thereby allowing for more efficient extraction of information from the image.

[0148] In step S504 of Figure 10, it was determined whether or not the image is a color image, but other determinations may also be made.

[0149] For example, it may be determined whether or not a line (e.g., a dashed rectangle) that specifies a particular area in the image is included in the image.

[0150] [Clustering Operation Example 4] The processes described in Operation Examples 1 to 3 above can be combined as appropriate and arbitrarily. Below, we will describe such a clustering operation example 4. Figure 11 is a flowchart illustrating yet another example of the detailed operation of step S204 in Figure 7.

[0151] First, the element extraction unit 222 performs segmentation on the image and generates an image in which objects in the image have been identified (S601).

[0152] Next, the element extraction unit 222 identifies a standard number of clusters from the image size (S602).

[0153] Next, the element extraction unit 222 extracts character information and character coordinates from the image using OCR (S603).

[0154] Subsequently, the localization unit 223 determines whether or not character information related to external connections is included in the image (S604).

[0155] If it is determined that the image does not contain any character information related to external connections (No in S604), the process proceeds to step S607, and the following determination is made.

[0156] If it is determined that the image contains character information related to an external connection (Yes in S604), the localization unit 223 acquires the coordinate information of the character information related to the external connection (S605).

[0157] Next, the number of coordinate pieces is added to the number of clusters (S606).

[0158] Subsequently, the localization unit 223 determines whether or not the image is a color image (S607).

[0159] If the image is determined not to be a color image (No in S607), the process proceeds to step S611.

[0160] If the image is determined to be a color image (Yes in S607), the color information of the surrounding objects is obtained (S608).

[0161] The localization unit 223 groups objects according to the acquired color information and calculates the center coordinates for each group (S609). Next, it adds the number of these center coordinates to the number of clusters (S610).

[0162] The localization unit 223 performs clustering on the image, which has identified objects in the image using the determined number of clusters (S611).

[0163] When clustering is performed on an image in which objects have been identified, the splitting unit 224 divides the image based on the clustering results so that the objects are contained within it (S612).

[0164] In this way, by performing clustering, it is possible to determine clusters that focus on textual information indicating important objects, and clusters that include areas indicated by information that represents a predetermined region. This suppresses the possibility of missing important objects and allows for efficient information extraction.

[0165] (Other Embodiments) The information extraction system relating to this disclosure has been described above based on embodiments, but this disclosure is not limited to these embodiments. As long as it does not depart from the spirit of this disclosure, various modifications that a person skilled in the art can conceive of will be applied to these embodiments and modifications, as well as other forms constructed by combining some of the components of the embodiments, are also included within the scope of this disclosure.

[0166] In the above embodiment, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0167] Furthermore, the communication method between devices in the above embodiment is not particularly limited. In addition, a relay device (not shown) may be interposed in the communication between devices.

[0168] For example, in the above embodiment, a process executed by a specific processing unit may be executed by another processing unit. Furthermore, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0169] For example, the order of processing described in the flowchart of the above embodiment is just one example. The order of multiple processing steps may be changed, and multiple processing steps may be executed in parallel.

[0170] Furthermore, some or all of the functions of the information extraction system according to the above embodiment may be realized by a processor such as a CPU executing a program.

[0171] Furthermore, each functional configuration of the information extraction system according to the above embodiment may be implemented using, for example, a machine learning model or generative AI (Artificial Intelligence) instead of LLM to realize various functions.

[0172] Furthermore, each component may be implemented by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or they may be separate circuits. Also, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0173] Some or all of the components constituting each of the above devices may consist of detachable IC cards or standalone modules attached to each device. The IC card or module is a computer system composed of a microprocessor, ROM, RAM, etc. The IC card or module may also include a highly functional LSI. The microprocessor operates according to a computer program, thereby enabling the IC card or module to achieve its function. The IC card or module may also be tamper-resistant.

[0174] Furthermore, comprehensive or specific embodiments of this disclosure may be implemented as systems, apparatus, methods, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs. They may also be implemented as any combination of systems, apparatus, methods, integrated circuits, computer programs, and recording media. The recording media may also be non-temporary recording media.

[0175] Furthermore, this disclosure may be implemented as a method to be executed by a computer system of an information extraction system, or as a program to cause a computer system to execute the method. Alternatively, this disclosure may be implemented as a computer-readable non-temporary recording medium on which such a program is recorded.

[0176] Furthermore, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art could conceive, or forms realized by arbitrarily combining the components and functions of each embodiment without departing from the spirit of this disclosure.

[0177] This disclosure is available for use in systems for extracting information contained in images.

[0178] 10 Information extraction system 20 Information extraction device 21 Communication unit 22 Information processing unit 221 Acquisition unit 222 Element extraction unit 223 Localization unit 224 Division unit 225 Information extraction unit 23 Storage unit 30 Information terminal 40 VLLM D1 Data database D2 Division criteria table D3 Asset list D4 Image after clustering

Claims

1. An information extraction method for extracting information contained in an image, which is performed by a computer, comprising: an element extraction step of extracting a plurality of elements constituting the image; a localization step of determining a plurality of clusters based on the plurality of elements extracted in the element extraction step; a division step of dividing the image into a plurality of divided images based on the plurality of clusters determined in the localization step; and an information extraction step of extracting information contained in each of the plurality of divided images of the image divided in the division step using a language model.

2. The information extraction method according to claim 1, further comprising an output step of outputting information contained in an image based on the information extracted from each of the plurality of divided images in the information extraction step.

3. The information extraction method according to claim 1 or 2, wherein the plurality of elements constituting the image include information about objects included in the image, and in the element extraction step, the position of the objects included in the image is extracted.

4. The information extraction method according to claim 1 or 2, wherein the plurality of elements constituting the image include character information contained in the image, and the element extraction step involves extracting the character information contained in the image.

5. The information extraction method according to claim 1 or 2, wherein in the element extraction step, the positions of each of the plurality of elements are extracted, and in the localization step, the plurality of clusters are determined based on the positions of each of the plurality of elements.

6. The information extraction method according to claim 5, wherein in the localization step, the plurality of clusters are determined such that one or more elements located in a region of the image where the density of the elements is higher than a first threshold are included in one of the plurality of clusters.

7. The information extraction method according to claim 1 or 2, wherein the element includes information indicating a predetermined region in the image, and in the localization step, the plurality of clusters are determined such that one or more elements included in the region indicated by the information indicating the predetermined region are included in one of the plurality of clusters.

8. The information extraction method according to claim 4, wherein, in the localization step, if the character information extracted in the element extraction step includes character information indicating an important object, a cluster centered on the coordinates of the important object is determined.

9. The information extraction method according to claim 8, wherein the important object is an object that includes external connection information.

10. The information extraction method according to claim 8, wherein the important object is an object related to network equipment or network information.

11. The information extraction method according to claim 8, wherein the important object is an object that is inferred to be the center of control.

12. The information extraction method according to claim 4, wherein, in the localization step, if the character information extracted in the element extraction step includes character information indicating a plurality of related objects, a cluster is determined such that the plurality of related objects are included in a single cluster.

13. The information extraction method according to claim 12, wherein the character information indicating the plurality of related objects is a plurality of IP addresses in which characters within a predetermined range match.

14. The information extraction method according to claim 12, wherein the character information indicating the plurality of related objects is an identification number for identifying similar objects.

15. The information extraction method according to claim 12, wherein the character information indicating the multiple related objects is information indicating that they communicate using the same communication protocol.

16. The information extraction method according to claim 1 or 2, wherein in the division step, the image is divided into a plurality of divided images such that all of the one or more elements included in one of the plurality of clusters are included in the same divided image.

17. The information extraction method according to claim 2, wherein in the output step, the information contained in the image is output in a list format.

18. The information extraction method according to claim 2, wherein in the output step, information indicating the position of the information contained in the image is output in association with the information contained in the image.

19. A program for causing a computer to execute the information extraction method described in claim 1 or 2.

20. An information extraction device for extracting information contained in an image, comprising: an element extraction unit for extracting a plurality of elements constituting the image; a localization unit for determining a plurality of clusters based on the plurality of elements extracted by the element extraction unit; a division unit for dividing the image into a plurality of divided images based on the plurality of clusters determined by the localization unit; and an information extraction unit for extracting information contained in each of the plurality of divided images of the image divided by the division unit, using a language model.

Images