Information extraction device, information extraction system, information extraction method, and program

Abstract

An information extraction device comprising an acquisition unit for acquiring an image obtained by imaging an extraction object, a control unit for inputting the image and indication information to a language model and acquiring information on the image from the language model, and an output unit for outputting the information.

Description

Information extraction device, information extraction system, information extraction method, and program

[0001] The present invention relates to a technique for extracting information from an extraction target, which is an object from which information is to be extracted.

[0002] A data center is equipped with many racks, each of which is equipped with devices such as servers. To operate the data center, it is important to monitor the devices installed in the racks. Both the racks and the devices installed in the racks are examples of the above-mentioned extraction targets.

[0003] A conventional technique for collecting information about devices mounted on a rack is disclosed in, for example, Non-Patent Document 1. This technique makes it possible to obtain physical information such as server temperature, fan speed, and power supply status. Furthermore, the technique disclosed in Non-Patent Document 2 makes it possible to centrally manage physical asset information about servers, storage, network devices, and the like.

[0004] Intel, Hewlett-Packard, NEC, Dell, "Intelligent Platform Management Specification", Version 2.0, Revision 1.1, 2013.Ferreira, Leylane, et al. "Standardization efforts for traditional data center infrastructure management: the big picture." IEEE Engineering Management Review 48.1 (2020): 92-103.

[0005] While the technology in Non-Patent Document 1 makes it possible to extract information from racks and their mounted devices, this technology cannot extract information such as the physical location of the devices or loose cables. The conventional technology requires manual labor to accurately grasp the latest information on the physical location of the devices, loose cables, etc. This increases labor costs and is inefficient.

[0006] In other words, the conventional technology has a problem in that it is not possible to efficiently extract information from the extraction target. This problem can occur not only when the extraction target is a rack and its mounted devices.

[0007] The present invention has been made in view of the above points, and has an object to provide a technique for efficiently extracting information from an extraction target.

[0008] According to the disclosed technology, there is provided an information extraction device including: an acquisition unit that acquires an image of an extraction target; a control unit that inputs the image and instruction information into a language model and acquires information related to the image from the language model; and an output unit that outputs the information.

[0009] The disclosed technology provides a technology for efficiently extracting information from an extraction target.

[0010] It is a block diagram of an information extraction system according to an embodiment of the present invention. It is a block diagram of a rack information extraction system according to an embodiment of the present invention. It is a flowchart for explaining the operation of the rack information extraction system according to an embodiment of the present invention. It is a diagram showing an example of a hardware configuration of a device.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0012] (System Configuration Example) FIG. 1 shows an example of the configuration of an information extraction system according to this embodiment. As shown in FIG. 1, the information extraction system according to this embodiment includes an information extraction device 1, an LLM (Large Language Model) 2, and a camera 3. FIG. 1 also shows an extraction target 4, which is an object from which information is to be extracted. The rack mentioned above is an example of the extraction target 4. The extraction target 4 is not limited to a specific object, and may be, for example, a person or an animal. When a person is the extraction target, it becomes possible to efficiently determine the presence or absence of a suspicious person and authenticate the person without using human labor.

[0013] 1, the LLM 2 is located outside the information extraction device 1, but the LLM 2 may be provided inside the information extraction device 1. Similarly, the camera 3 may be located outside the information extraction device 1, or the LLM 2 may be provided inside the information extraction device 1.

[0014] The camera 3 captures an image of the extraction target 4. The LLM2 is a neural network language model, and for example, GPT-4V can be used as the LLM2. Although the LLM2 is a "large-scale language model," a language model that is not limited to being "large-scale" can be used, so "LLM2" in FIG. 1 may be replaced with "language model." The "large-scale language model" is an example of a "language model."

[0015] The information extraction device 1 has a control unit 11, an acquisition unit 12, and an output unit 13. The acquisition unit 12 acquires an image of an extraction target 4 captured by a camera 3. The control unit 12 inputs the image and instruction information to the LLM 2, and acquires information extracted from the image from the LLM 2. The output unit 13 outputs the information.

[0016] In this embodiment, information is extracted from an image of the extraction target 4 using LLM2, so that information can be extracted efficiently.

[0017] Below, as a more specific example, an embodiment will be described in which racks in a data center and the devices installed thereon are used as the target of information extraction. By applying the technology according to the present invention to racks in a data center, it is possible to support efficient operation of the data center. Note that the various effects in the following embodiment in which racks and the devices installed thereon are used as an example of the target of extraction are effects obtained according to the embodiment when the present invention is applied to a specific example. Note that the extraction of information about racks is an example of the application of the technology according to this embodiment to monitoring work such as monitoring machinery in a factory.

[0018] (Outline of the embodiment) Data centers, as the foundation of information technology, are responding to the rapid increase in digital data in business and daily life. As data centers expand, the number of devices is also increasing significantly, and accurate and real-time understanding of information such as the operating status, settings, and physical location of each device is essential for improving operational efficiency and ensuring service continuity.

[0019] However, conventional methods for managing information on rack-mounted devices have limitations, and the management burden increases in proportion to the increase in the number of devices, while collecting and updating information requires a lot of time and effort. As the importance of device information management, which is at the core of data center operations, increases, there is a need to develop new solutions that can perform these tasks more efficiently and maintain high accuracy.

[0020] In this situation, along with the emergence of GPT-4, GPT-4V, which further improved its image recognition capabilities, was developed. In this embodiment, for example, these AI technologies are utilized to improve the operational efficiency of data centers.

[0021] In this embodiment, the image recognition and analysis functions of AI technology are applied to data center rack management, automating tasks that previously required manual labor and significantly improving their accuracy. By achieving more efficient management, data center operation costs can be reduced while providing the flexibility to quickly respond to changes in the operating status or layout of equipment. Furthermore, accurately understanding data center equipment information contributes to faster response times when failures occur and enhanced security.

[0022] (Differences between the Technology According to the Present Embodiment and the Prior Art) Here, the prior art described in Non-Patent Documents 1 and 2 will be explained, and the differences between the technology according to the present embodiment and the prior art will be explained.

[0023] Non-Patent Document 1 discloses a technology called IPMI (Intelligent Platform Management Interface). IPMI is an industry-standard interface for remotely monitoring hardware status and managing systems. It is possible to obtain physical information such as server temperature, fan speed, and power supply status, and to issue a warning if the system is in an abnormal state. Furthermore, even if the system is down, the machine's power can be turned on and off through an independent management network.

[0024] However, IPMI does not have the ability to identify the physical location of a device within a rack, and it cannot diagnose device errors that are caused by physical factors such as loose cables or improper racking. These limitations force data center managers to perform inefficient manual inspections, which can significantly impair operational efficiency.

[0025] In contrast, the technology according to this embodiment takes photographs of racks in a data center and analyzes the image data to collect information on the status of devices or their precise physical location. This approach also provides information on physical aspects that IPMI does not fully cover, making it possible to more quickly determine whether device errors are caused by physical factors. This allows administrators to take more accurate action based on physical location information, making it possible to immediately identify and resolve physical problems that would previously have been overlooked.

[0026] Non-Patent Document 2 discloses a technology called DCIM (Data Center Infrastructure Management).

[0027] DCIM stands for data center physical and virtual infrastructure management, and DCIM tools are a type of software designed for this purpose. The rack management function, in particular, allows for centralized management of physical asset information, such as servers, storage, and network devices. This includes detailed specifications such as the device model, serial number, unit location within the rack, power consumption, and network connection information. Furthermore, environmental data such as temperature and humidity can be recorded to support asset health and efficient operation.

[0028] However, this information is essentially static data at the time of installation, and because equipment is constantly being added, removed, and moved during data center operations, it must be manually updated to reflect actual conditions. This updating work, especially in large data centers, increases in cost and consumes a large amount of human resources as the number of racks increases. Furthermore, every time equipment is moved, all related data must be updated, requiring significant effort and time. This makes it difficult to keep the information up to date, sometimes forcing inefficient operations.

[0029] In contrast, the technology according to this embodiment collects information by photographing racks in a data center and analyzing the image data obtained. This significantly reduces the need for manual information collection. Even when equipment is moved or replaced, the database can be automatically updated by photographing and analyzing the images, reducing the burden of manual updating work and enabling the accuracy of information and efficient operation.

[0030] (Examples of Problems Solved by the Technology According to the Present Embodiment) Here, as examples of problems in the conventional technology that can be solved by the technology according to the present embodiment, the following Problem 1 and Problem 2 will be described. Also, a solution method in the present embodiment for each problem will be described.

[0031] The first issue is the difficulty of remotely identifying physical problems with equipment in a data center. Conventional remote management systems can remotely monitor the status of hardware, but it is difficult to identify problems with physical placement and connections. In particular, errors caused by physical factors, such as poor cable connections or incorrect placement of equipment, are difficult to detect with remote management systems such as IPMI.

[0032] In contrast, the technology according to this embodiment captures the status of the devices in the rack as an image and applies image analysis technology, making it possible to identify these physical problems even remotely. This allows the administrator to take appropriate action based on physical location information, enabling immediate resolution of physical problems that would previously have been overlooked.

[0033] The second issue is the high cost and effort required to update the status of equipment in a rack. In data center rack management, equipment is frequently added, replaced, and moved, so it is necessary to keep the information about equipment in the rack up to date. In infrastructure management systems such as DCIM, this information must be updated manually, which is a time-consuming and costly process, especially in large data centers.

[0034] In contrast, the technology according to the present embodiment automatically recognizes the status of the equipment from photographed images of the rack, making it possible to automate updating of the database. This process reduces the effort and cost of updating work that requires manual intervention, and makes it possible to maintain accurate and up-to-date equipment status information.

[0035] The system configuration and operation according to this embodiment will be described in detail below.

[0036] (System Configuration Example) A configuration example of a rack information extraction system according to this embodiment is shown in Fig. 2. The rack information extraction system is an example of the information extraction system shown in Fig. 1 .

[0037] 1, the rack information extraction system includes an LLM server 200, a control server 100, and a network camera 300. Below, an overview of the functions of each device will be explained.

[0038] <LLM Server 200> The LLM Server 200 has the function of analyzing the photographed image of the rack, extracting necessary information, and responding with the analysis results in text. Furthermore, based on the data read from the image, the LLM Server 200 determines which part of the rack should be photographed next or whether re-photography is necessary, and outputs specific instructions in text format.

[0039] The LLM server 200 has a large-scale language model such as GPT-4V, and realizes the above functions using the large-scale language model. The functions of the LLM server 200 described below correspond to the functions of the large-scale language model.

[0040] <Control Server 100> The control server 100 functions as the core of the rack information extraction system. As shown in FIG. 2, the control server 100 includes a prompt control engine 110 and a network camera control interface 120.

[0041] The control server 100 has a function to incorporate the captured image from the network camera 300 into the LLM input prompt and transmit it to the LLM server 200, and a function to receive instructions in text form from the LLM server 200, check whether or not re-shooting is required, and if re-shooting is required, convert the re-shooting instruction into an operation command for the network camera 300.

[0042] <Control Server 100: Prompt Control Engine 110> The prompt control engine 110 appropriately manages input prompts to the LLM server 200, generating queries for analyzing image information, queries for assessing the need for retakes, etc., thereby helping the LLM server 200 provide the most relevant instructions.

[0043] <Control Server 100: Network Camera Control Interface 120> Based on instructions from the control server 100, the network camera control interface 120 embeds shooting parameters such as the shooting angle, position, and zoom level of the network camera 300 into a specific control request and sends the control request to the network camera 300. This automates the shooting process and improves the accuracy of shooting.

[0044] <Network Camera 300 > The network camera 300 takes an image of the rack in accordance with a control request from the control interface 120 .

[0045] (System Operation) The overall operation of the rack information extraction system according to this embodiment will be described with reference to the flowchart in Fig. 3. Details of the operation of each step will be described later.

[0046] <S101: Initial Preparation> Determine the optimal shooting position, connect the network camera 300 to the network, set a schedule, and prepare the environment including adjusting the lighting. Furthermore, pre-learning and fine tuning are performed on the LLM to improve the accuracy of image recognition.

[0047] <S102: Execution of Photography> The network camera 300 is operated via the control server 100 to cause the network camera 300 to photograph the rack based on a schedule or in response to a request.

[0048] <S103: Sending Image Data and Instruction Prompt to LLM Server 200> The control server 100 sends the captured image data and instruction prompt to the LLM server 200. The LLM server 200 analyzes the image data and extracts information such as the manufacturer name, model number, and operating status of the device.

[0049] <S104: Image Analysis and Re-shooting Determination> The LLM server 200 evaluates the image analysis results and checks the completeness and accuracy of the information. If the analysis results are determined to be incomplete or inaccurate, the LLM server 200 issues an instruction to the control server 100 to take a re-shoot. Specifically, it issues an instruction to adjust the angle or location for re-shooting. In this case, the process returns to S102. If it is determined that re-shooting is not necessary, the process proceeds to the next step, S105.

[0050] <S105: Output of Information> If the LLM server 200 confirms that the extracted information is complete and accurate, the information is sent to the control server 100, and the control server 100 outputs the extracted information. Note that in the control server 100, the prompt control engine 110 or another functional unit may output the information. Note that the control server 100 may also confirm that the extracted information is complete and accurate.

[0051] The processing of each of the above steps will now be described in more detail.

[0052] (S101: Initial Preparation) In the initial preparation, the optimal shooting position of the network camera 300 is determined so that the rack can be captured as a whole or a specific device in the rack can be captured in detail. Next, the network camera 300 is connected to the data center network, and network settings are performed to ensure that it can be accessed and controlled from a remote location.

[0053] Furthermore, you can set up a schedule for the network camera 300 to take pictures automatically, either periodically or in response to specific events. To ensure an optimal environment for taking pictures, adjust the lighting in front and behind the rack and make sure there are no obstacles within the field of view of the network camera 300.

[0054] Additionally, preset rules regarding the purpose of analysis and the method of response are provided in advance to the LLM server 200. The preset rules may be provided to the LLM server 200 by the prompt control engine 110 of the control server 100, or may be provided to the LLM server 200 by a device other than the control server 100.

[0055] Specific examples of pre-defined rules are as follows:

[0056] <Examples of pre-selection rules> { "analysis_points": { "device_identification": "The LLM server must identify and confirm the visibility and legibility of the device's identifying markers such as labels, barcodes, and indicators.", "connectivity_status": "The LLM server must assess the connection status of cables and ports; any unplugged or damaged connectors should trigger a reshoot.", "operational_signs": "The LLM server must evaluate operational signs such as LED indicators; unlit or flashing warning lights should be noted and may necessitate a reshoot." }, "reshoot_criteria": { "legibility_issue": "If text or critical indicators on the device are not legible due to blurring or low resolution, a reshoot is required.", "information_omission": "If any required information is missing from the image, a reshoot is required focusing on the area lacking detail.}, "reshoot_instructions": { "focus_area": ​​"If the initial image lacks detail in a specific area, LLM will instruct to focus on that area in the reshoot.", "zoom_level": "If critical text or indicators are blurred, LLM will increase the zoom level to capture a clearer image.", "lighting": "If the initial image is poorly lit, affecting the visibility of crucial details, LLM will adjust the lighting parameters for a reshoot.", "angle": "If necessary information is obscured due to perspective, LLM will change the capture angle to reveal the obscured details."}, "analysis_priorities": [ "Ensure all devices within the rack are identifiable by their labels.", "Verify that all connections are secure and properly in place.", "Check for any visible signs of device malfunction or environmental risk factors." ] <Example of preconfigured rules: Written in Japanese> The language of the predefined rules is not limited to a specific language. For example, they may be written in Japanese, as shown below. The example below corresponds to the English sentence above translated into Japanese. In the following, parentheses are omitted and numbers such as (1) are used.

[0057] (1) Analysis Points Device Identification: The LLM server must identify and verify the visibility and readability of the device's identification markers, such as labels, barcodes, and indicators.

[0058] Connection status: The LLM server must evaluate the connection status of the cable and port. An unplug or a faulty connector must trigger a retake.

[0059] Operational indicators: The LLM server must evaluate operational indicators such as LED indicators and note any warning lights that are off or flashing, which may require a retake.

[0060] (2) Re-shooting Indicators Readability Issues: If text or important indicators on the device are unreadable due to blur or low resolution, a re-shoot is required.

[0061] Missing information: If an image is missing necessary information, it will need to be re-photographed, focusing on the area where the detail is missing.

[0062] (3) Retake Instructions Focus Area: If the initial image lacks detail in a particular area, the LLM will instruct you to focus on that area in the retake.

[0063] Zoom Level: If important text or indicators are blurred, LLM increases the zoom level to capture a clearer image.

[0064] Lighting: If the lighting is insufficient in the initial image, affecting the visibility of important details, LLM adjusts the lighting parameters for the retake.

[0065] Angle: If the required information is obscured due to an inappropriate viewpoint, LLM changes the capture angle to reveal the obscured details.

[0066] (4) Analysis Priority - Ensure that all devices in the rack are identifiable by their labels.

[0067] Verify that all connections are secure and properly in place.

[0068] Check for visible signs of equipment malfunction or environmental risk factors.

[0069] The above are examples of specific pre-set rules.

[0070] The above pre-set rules are rules that allow the LLM server 200 to determine whether re-imaging is necessary based on the analysis results, and if so, to issue clear instructions for re-imaging. These rules can be flexibly modified depending on the purpose or context of the analysis, and provide important guidelines when the rack information extraction system performs specific tasks, such as collecting rack information or checking the status of individual devices. Examples of items (1) to (4) that may be included in the pre-set rules are shown below. The above specific rules are examples of rules that have the following items:

[0071] (1) analysis_points The analysis_points are rules that indicate analysis points, and are rules for "checking the visibility of identification information, evaluating the connection status, and confirming operational indicators" of the equipment. The setting of these rules is necessary for the rack information extraction system to clearly identify important markers or indicators in the image and perform accurate information extraction based on them.

[0072] (2) reshoot_criteria The reshoot_criteria is a rule that defines the criteria for reshooting an image when the image is not suitable for reading information. Specifically, this rule indicates that if there is unclear text or omitted information, a reshoot is required to ensure that the information is readable.

[0073] (3) reshoot_instructions The reshoot_instructions are rules for generating instructions to adjust shooting parameters when it is determined that reshooting is necessary. These instructions include specifying the focus, adjusting the zoom level, optimizing the lighting, and changing the shooting angle. These instructions are used by the LLM server 200 to communicate specific shooting conditions to the control server 100 in order to clearly capture unclear information.

[0074] (4) analysis_priorities The analysis_priorities are the priorities for performing analysis, and include items such as identifying the device, checking the safety of the connection, and observing signs of functional failure or environmental risk. The priorities may be determined in the order in which the items are listed. The analysis_priorities are set to clarify points that the rack information extraction system should pay particular attention to when performing analysis and to correctly evaluate the need for re-imaging.

[0075] <About Fine Tuning> Fine tuning is an important process in the initial preparation stage to maximize the capabilities of the LLM. In particular, to improve the image recognition accuracy of data center server racks and the equipment installed therein, the LLM is trained in advance on specific equipment images. This enables the LLM to accurately identify the appearance, labels, and other important features of the equipment. Furthermore, by performing image recognition tasks using sample images and providing specific feedback based on the results, the LLM's learning process is adjusted and the task success rate is improved. This fine tuning enables the LLM to demonstrate greater accuracy and efficiency in image analysis in a real-world operational environment.

[0076] (S102: Execution of Photography) Next, the execution of photography in S102 will be described. In S102, the control server 100 plays a central role and operates the network camera 300 to take a photograph of the rack. In the control server 100, the network camera control interface 120 issues a photography instruction to the network camera 300. This photography instruction includes detailed photography parameters such as the position of the rack to be photographed, the photography angle, and the zoom magnification. These parameters are flexibly adjusted depending on the purpose of the photography.

[0077] When taking photos for routine monitoring, wide-angle shots are taken so that the entire rack is in the frame, providing a comprehensive view of what is happening inside the rack. In contrast, when taking photos in response to an alert from a specific device, images of the individual device are taken, focusing on the problematic device and zooming in to obtain more detailed information.

[0078] The images captured can be used for a variety of purposes, including equipment status monitoring, alert verification, and maintenance scheduling. Through this process, data center operators can understand the status of racks in real time and take prompt action if necessary.

[0079] The control server 100 immediately collects image data from the network camera 300 after the image capture is completed, and proceeds to the next processing step on the assumption that the image data will be transmitted to the LLM server 200 .

[0080] (S103: Transmission of Image Data and Instruction Prompt to LLM Server 200) Next, the transmission of image data and instruction prompt to the LLM server 200 in S103 will be described.

[0081] In S103, the prompt control engine 110 in the control server 100 transmits the captured image data and an instruction prompt according to the purpose of analysis to the LLM server 200. This process utilizes the advanced image analysis function provided by the LLM server 200 to accurately extract information about each device in the rack.

[0082] The instruction prompts sent to the LLM server 200 are prompts that specify a particular analysis task to the LLM server 200 and guide it in identifying the necessary information from the image data.

[0083] If the goal is to gather overall information about rack-mounted equipment, the instruction prompts include instructions to recognize the appearance of the equipment and the equipment label in the image and read data such as the equipment manufacturer name, model number, and operating status from there.

[0084] As an example of an instruction prompt for grasping the overall status of the rack, a specific instruction prompt example 1 is shown below.

[0085] <Example 1 of instruction prompt> { "task": "rack_overview_analysis", "instructions": [ "Capture the entire server rack ensuring all devices are visible.", "Identify each device within the rack based on its physical characteristics and visible labels.", "Extract manufacturer names, model numbers, and operational status indicators from the visible labels.", "Assess any visible signs of damage, overheating risk, or warning indicators.", "Highlight any areas where cables are disorganized or potentially obstructive." ], "parameters": { "focus": "wide", "zoom": "none", "lighting": "standard", "angle": "frontal"}, "desired_output": [ "Complete list of devices with corresponding manufacturer names, model numbers, and operational statuses.", "Any critical observations that require immediate attention." ]} <Example 1 of instruction prompt: written in Japanese> The language of the instruction prompt is not limited to a specific language. For example, it may be an instruction prompt in Japanese as shown below. The following example corresponds to the Japanese translation of the English sentence above. In the following, parentheses are omitted and numbers such as (1) are added.

[0086] (1) Task: Rack Overview Analysis Instructions: Capture the entire server rack to ensure all devices are visible.

[0087] • Identify each piece of equipment in the rack based on its physical characteristics and visible label.

[0088] Extracting manufacturer name, model number, and operational status indicator from the visible label.

[0089] - Assess any visible signs of damage, overheating risk, or warning indicators.

[0090] Highlight any areas where cables are cluttered or potentially in the way.

[0091] (2) Parameters: Focus: Wide; Zoom: None; Lighting: Normal; Angle: Front; (3) Desired Output: A complete list of devices with their corresponding manufacturer names, model numbers, and operational status; Any significant observations requiring immediate attention. This is Example 1 of a specific instruction prompt.

[0092] The above instruction prompt is a prompt that instructs the LLM server 200 to perform a comprehensive image analysis of the entire rack. As shown in the above specific example, the instruction prompt clearly defines the type of task, specific analysis instructions, parameters related to photography, and expected output results.

[0093] With the above prompts, the LLM server 200 extracts comprehensive information about all the equipment in the rack and provides insights to contribute to the operational management of the data center. Such analysis is primarily used for asset management or inventory updates.

[0094] On the other hand, in scenarios where more detailed status checks are required for a specific device, the instruction prompt instructs the LLM server 200 to use image analysis to detect more specific physical conditions, such as whether the device's wiring is preventing heat dissipation.

[0095] As an example of an instruction prompt for grasping the detailed status of a specific device, a specific instruction prompt example 2 is shown below.

[0096] <Example 2 of Instruction Prompt> { "task": "detailed_device_inspection", "instructions": [ "Isolate the targeted device within the server rack for detailed analysis.", "Zoom in to capture high-resolution images of the device's front panel and any visible ports or cables.", "Check for the proper connection of all required cables and ensure no cables are unplugged or showing signs of wear.", "Inspect the device's indicators, such as LEDs, for signs of malfunction or abnormal operation.", "Evaluate the surrounding area for any obstructions that may affect airflow and cooling efficiency." ], "parameters": { "focus": "narrow", "zoom": "high", "lighting": "enhanced", "angle": "variable"}, "desired_output": [ "Detailed report on the device's current connection status, including any potential issues.", "Assessment of operational indicators and any necessary follow-up actions.", "Evaluation of environmental conditions affecting the device's performance." ]} <Instruction prompt example 2: written in Japanese> The language of the instruction prompt is not limited to a specific language. For example, it may be written in Japanese as shown below. The example below corresponds to the above English sentence translated into Japanese. In the following, parentheses and other symbols are omitted and numbers such as (1) are added.

[0097] (1) Task: Detailed Device Inspection Instructions: Isolate the target device in the server rack for detailed analysis.

[0098] - Zoom in to capture high-resolution images of the device's front panel and any visible ports or cables.

[0099] Check all necessary cables for proper connections and ensure they are not loose or showing signs of wear.

[0100] Inspect device indicators, such as LEDs, for signs of malfunction or abnormal operation.

[0101] - Evaluate the surrounding area for any obstructions that may affect airflow and cooling efficiency.

[0102] (2) Parameters Focus: Narrow Zoom: High Lighting: Strong Angle: Variable (3) Desired Output A detailed report on the current connection status of the device, including any potential problems An evaluation of operational indicators and any necessary follow-up actions An evaluation of environmental conditions that may affect the performance of the device This is Example 2 of a specific instruction prompt.

[0103] The above instruction prompt instructs the LLM server 200 to perform a detailed image analysis focused on a specific device. As shown in the above specific example, the imaging parameters are set in detail to clearly identify the details of the device, with the aim of extracting detailed information, particularly about the connection status or operational indicators.

[0104] The prompts also include an assessment of environmental conditions that may affect equipment cooling and performance, a necessary approach to assess operational risk and enable rapid maintenance response.

[0105] By using instruction prompts for detailed status understanding, it is possible to improve the operational efficiency of data centers and the accuracy of equipment status monitoring.

[0106] (S104: Image Analysis and Determination of Re-Photographing) Next, the image analysis and determination of re-photographing in S104 will be described.

[0107] At S104, the LLM server 200 performs analysis on the image data received from the control server 100 to evaluate whether the requested information has been properly collected according to pre-set rules and instruction prompts, including verifying that each information point identified in the image has been properly read.

[0108] If the LLM server 200 determines through its analysis that the information identified from the image is incomplete or inaccurate, it determines that recapture is necessary and notifies the control server 100 of the specific capture parameters required for recapture. For example, if the model number of a device in the image is unclear, the LLM server 200 instructs the control server 100 to focus on the device and increase the zoom level to obtain a closer image. In this way, the LLM server 200 generates and specifies capture parameters to clearly capture the missing information.

[0109] If the LLM server 200 determines that re-imaging is not necessary, that is, if it determines that all necessary information has been accurately collected, it proceeds to the next step.

[0110] In addition, the confirmation determination of whether information collection has been successful may not only depend on a single LLM server 200, but may also be made more accurate by integrating the analysis results of multiple LLM servers 200.

[0111] For example, the control server 100 may communicate with multiple LLM servers 200, and the control server 100 may determine that information collection has been successful if these multiple LLM servers 200 return the same information to the control server 100. Alternatively, the control server 100 may make multiple inquiries to the same LLM server 200, and determine that information collection has been successful if the same information is obtained from the multiple inquiries. These methods can improve the accuracy of confirmation and determination, and achieve more reliable data collection.

[0112] (S105: Output of Information) Next, the output of information in S105 will be described. In S105, the control server 100 acquires information obtained by image analysis from the LLM server 200 and performs processing to output the information. The information collected and verified up to S104 becomes a source of information used for operation management of the data center and emergency response. Therefore, the control server 100 appropriately organizes the information and supplies it to related systems.

[0113] The information output method (output form) is not limited to a specific method, but examples thereof include the following (a) to (e).

[0114] (a) Updating the DCIM Database In this example, the rack information or device information collected by the control server 100 is output from the control server 100 and input into the database of a data center infrastructure management (DCIM) system, thereby enabling inventory management, capacity planning, and asset life cycle management to be performed based on the latest information.

[0115] (b) Supplementary Information for the Network Monitoring System In this example, the analysis results obtained by the control server 100 are used as additional reference information for alerts or events in the network monitoring system. The analysis results obtained by the control server 100 assist the relevant parties in troubleshooting or responding to incidents so that they can act more quickly and effectively.

[0116] (c) Maintenance Scheduling In this example, information indicating the condition of a particular piece of equipment or environmental risk is used to plan a schedule of preventative maintenance or maintenance activities, thereby minimizing the risk of downtime and maintaining system reliability.

[0117] (d) Report Generation In this example, periodic operational or audit reports incorporate the information collected by the control server 100. This increases operational transparency and allows for quick decision making when necessary.

[0118] (e) Incident Response In this example, when an emergency incident occurs, information collected by the control server 100 is provided to an incident response team, and specific actions for resolving the problem are guided.

[0119] The output processes shown in (a) to (e) may be performed automatically based on a predefined workflow or protocol. Automation can reduce human error and improve operational efficiency. These outputs may also be notified to stakeholders as needed so that necessary actions can be taken.

[0120] (Summary of the embodiment) The rack information extraction system according to the embodiment described above has, for example, the following features (1) to (4).

[0121] (1) Automated image recognition and information extraction This rack information extraction system automatically takes images of racks in data centers and uses image recognition technology to automatically extract information from the images. This technology solves the problems of manual information collection and improves efficiency and accuracy.

[0122] (2) Optimization of retakes through a feedback loop: This rack information extraction system determines the need for retakes based on the analysis results and dynamically adjusts specific imaging parameters. This approach ensures that retakes are performed under optimal conditions when information is insufficient or unclear, ensuring the collection of necessary information.

[0123] (3) Customizable analysis prompts This rack information extraction system allows you to set customizable analysis prompts according to various monitoring and analysis needs, giving it the flexibility to respond to specific information requests. This customizability makes it possible to accommodate a variety of operational scenarios in data centers.

[0124] (4) Integrated system design This rack information extraction system is designed to process image capture, analysis, and information output within a single integrated system. This integration allows each stage of the process to work seamlessly together, ensuring a smooth flow of information.

[0125] The features (1) to (4) above promote innovation in operation management in data centers, and make it possible to significantly improve reliability, security, and economy. Furthermore, the technology according to this embodiment also serves as a foundation for the evolution of data center technology in the future.

[0126] (Effects of the embodiment) Effects 1 to 7 are shown as examples of effects obtained by the rack information extraction system according to the embodiment. The technology according to the embodiment, which has the following effects, contributes to overall cost reduction, efficiency improvement, and quality improvement in data center operations, and enables the provision of more competitive services.

[0127] Effect 1: Efficient information collection Automatic collection of rack and equipment information significantly improves the efficiency of data center operations. Compared to conventional methods that require human intervention, it reduces time and costs and ensures the immediacy and accuracy of information.

[0128] Effect 2: Real-time monitoring and rapid response: Real-time monitoring using image analysis enables early detection of equipment abnormalities or failures, enabling rapid response. This minimizes system downtime and maintains operational stability.

[0129] Effect 3: Optimization of maintenance work: Automated information collection and analysis allows for effective scheduling of preventative maintenance and maintenance work, thereby extending the life of assets and reducing costs due to unexpected breakdowns or damage.

[0130] Effect 4: Improved accuracy of infrastructure management Automatic updates to the DCIM system keep infrastructure management up to date, enabling more accurate asset management.

[0131] Effect 5: Reduced operational burden Automated processes reduce human error and reduce operational burden, allowing workers to focus on higher-value tasks.

[0132] Effect 6: Management of environmental risks Environmental risks such as heat exhaust problems in equipment or wiring defects can be monitored, improving the effectiveness of risk management.

[0133] Effect 7: Improved scalability The technology according to this embodiment is highly scalable, and can accommodate information collection needs that arise when new racks are introduced or when a data center is expanded.

[0134] (Hardware Configuration Example) Any of the devices described in this embodiment (such as the information extraction device 1, the control server 100, and the LLM server 200) can be realized by, for example, causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0135] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0136] Fig. 4 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 4 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0137] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0138] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0139] The following additional notes are provided regarding the above-described embodiments.

[0140] <Additional Notes> (Additional Item 1) An information extraction device including a memory and at least one processor connected to the memory, wherein the processor acquires an image of an extraction target, inputs the image and instruction information into a language model, acquires information related to the image from the language model, and outputs the information. (Additional Item 2) The information extraction device according to Additional Item 1, wherein the instruction information is information that instructs the language model to extract overall information about the extraction target from the image. (Additional Item 3) The information extraction device according to Additional Item 1, wherein the instruction information is information that instructs the language model to extract detailed information related to a specific object in the extraction target from the image. (Additional Item 4) The information extraction device according to any one of Additional Items 1 to 3, wherein, if the information acquired from the language model is information to be used for re-imaging the extraction target, the processor uses the information to acquire a re-imaging image of the extraction target. (Supplementary Item 5) The information extraction device according to Supplementary Item 4, wherein the language model generates shooting conditions for re-shooting based on preset rules for re-shooting, and the acquisition unit acquires the shooting conditions. (Supplementary Item 6) An information extraction system comprising the information extraction device according to Supplementary Item 1 and a camera for photographing the extraction target. (Supplementary Item 7) An information extraction method executed by an information extraction device, comprising: acquiring an image of an extraction target; inputting the image and instruction information into a language model and acquiring information relating to the image from the language model; and outputting the information. (Supplementary Item 8) A non-transitory storage medium storing a program for causing a computer to function as each unit in the information extraction device according to any one of Supplementary Items 1 to 5.

[0141] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0142] REFERENCE SIGNS LIST 1 Information extraction device 2 LLM 3 Camera 4 Extraction target 11 Control unit 12 Acquisition unit 13 Output unit 100 Control server 110 Prompt control engine 120 Network camera control interface 200 LLM server 300 Network camera 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. An information extraction device comprising: an acquisition unit that acquires an image of an extraction target; a control unit that inputs the image and instruction information into a language model and acquires information about the image from the language model; and an output unit that outputs the information.

2. The information extraction device according to claim 1, wherein the instruction information is information that instructs the language model to extract overall information of the extraction target from the image.

3. The information extraction device according to claim 1, wherein the instruction information is information that instructs the language model to extract detailed information about a specific object in the extraction target from the image.

4. The information extraction device according to claim 1, wherein, when the information acquired from the language model is information to be used for re-photographing the extraction target, the acquisition unit acquires a re-photographed image of the extraction target using the information.

5. The information extraction device according to claim 4, wherein the language model generates shooting conditions for re-shooting based on preset rules for re-shooting, and the acquisition unit acquires the shooting conditions.

6. An information extraction system comprising the information extraction device according to claim 1 and a camera for photographing the extraction target.

7. An information extraction method executed by an information extraction device, comprising: a step of acquiring an image of an extraction target; a step of inputting the image and instruction information into a language model and acquiring information about the image from the language model; and a step of outputting the information.

8. A program for causing a computer to function as each unit in the information extraction device according to any one of claims 1 to 5.

Images