Multi-source data acquisition and fusion method for submarine optical cable network

Abstract

The invention provides a submarine optical cable information acquisition and fusion method based on a large model intelligent agent, and the method comprises the steps: collecting first submarine optical cable data from a plurality of submarine cable websites through a crawler technology, and converting the first submarine optical cable data into structural data through a preset large model; acquiring second submarine optical cable data from a submarine optical cable and an official website of an operator by using a multi-agent collaboration architecture, and converting the second submarine optical cable data into structured data; storing the submarine optical cable data of each data source in a graph database in a unified manner, and loading submarine optical cable nodes, login point nodes and related relational data from each data source to the graph database to obtain integrated data; and the integrated data diversity set is stored in a MongoDB database, and submarine optical cable information fusion is completed. Through a multi-agent cooperation mode, automatic processing of a complex website structure is realized, and the data acquisition efficiency is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of submarine optical cable, and particularly relates to a multi-source data acquisition and fusion method for submarine optical cable network. BACKGROUND

[0002] The existing submarine cable data acquisition scheme mainly relies on traditional network information acquisition methods, and its core technologies include webpage crawling and structured data extraction.

[0003] Webpage information crawling is usually divided into two types: rule-based crawling and machine learning-based crawling. The former uses predefined rules or regular expressions to crawl specific information, which is simple to operate but has poor adaptability to complex websites and dynamic pages. The latter uses traditional artificial intelligence algorithms such as natural language processing and image recognition to automatically learn webpage content and extract data, which has better adaptability and can extract useful information from a large amount of unstructured data.

[0004] Structured data extraction is the process of extracting meaningful data from webpages and converting it into a structured format. The key to this technology is how to accurately extract effective information from a large amount of unstructured or semi-structured data and convert it into a structured data format that can be further analyzed and processed, such as tables, database records, etc.

[0005] Currently, there are two main methods for structured data extraction: one is based on templates or rules, which uses predefined rules or templates to identify and extract data; the other is based on machine learning, which uses algorithms to automatically learn and extract structured data from webpage content.

[0006] Although webpage information crawling technology has made significant progress, existing technologies still cannot meet the special needs of submarine cable data acquisition and fusion. First, due to the diversity and dynamic changes of submarine cable and operator websites, existing solutions often cannot be effectively processed, resulting in low crawling efficiency or data loss. Second, website anti-information crawling technology is constantly improving, increasing the difficulty of the crawling process, especially for some high-security sites, the crawling process is often blocked or limited.

[0007] In terms of data extraction, rule-based methods are generally suitable for simple and fixed-mode webpages, but their applicability and robustness are poor when dealing with complex and dynamically changing webpages. Machine learning-based methods can better adapt to complex pages and variable formats, but they require a large amount of labeled data and training process, and have high requirements for model selection and tuning.

[0008] The current prior art applied to the field of submarine cable data collection has many defects. If the existing technology is used to extract submarine cable related data, and the natural language text in the collected data is structured, the following problems will exist: Data consistency is difficult to guarantee: there are problems such as inconsistent update time, inconsistent original source, and inconsistent data format between data sources, resulting in serious conflicts between data from different sources and different data dimensions.

[0009] Information extraction efficiency is difficult: for Submarine Networks and Wikipedia, etc. The submarine cable information in the form of natural language text, the existing template or rule-based method cannot be used due to the difficulty in constructing a template with sufficient generality, and the machine learning data extraction method has the defects of needing manual marking of data sets, slow processing speed, high cost, and difficulty in updating data.

[0010] Information exploration and acquisition are difficult: although the data of submarine cable and operator official websites is the most authoritative, due to the complex and diverse forms of websites, the distribution of submarine cable data on official websites has the following characteristics: submarine cable information is hidden in subpages, sometimes it needs to use the search function of the website to find; the information display form is various, sometimes in the form of pure text, sometimes embedded in pictures, or even in videos or web page JavaScript dynamic scripts; the default language of some websites is not English, and the English data can be normally collected after interacting with the website to modify the language; after exploring all subpages of some operator websites, it is determined that they do not provide submarine cable data.

[0011] Therefore, it is challenging to obtain the required data from submarine cable data websites and various forms of submarine cable official websites and fuse them through traditional network data acquisition methods. SUMMARY

[0012] The present application aims to at least partially solve the technical problems of low efficiency and difficulty in ensuring data quality in the related art of submarine cable data collection.

[0013] To this end, the first object of the present application is to propose a submarine cable information collection and fusion method based on a large model agent.

[0014] The second object of the present application is to propose a submarine cable information collection and fusion device based on a large model agent.

[0015] The third object of the present application is to propose an electronic device.

[0016] The fourth object of the present application is to propose a computer readable storage medium.

[0017] A fifth object of the present application is to propose a computer program product.

[0018] To achieve the above object, the first aspect of the present application proposes a submarine cable information collection and fusion method based on a large model agent, comprising the following steps: Collect first submarine cable data from multiple submarine cable websites through a crawler technology, and convert the first submarine cable data into structured data through a preset large model; Collect second submarine cable data from submarine cable and operator official websites using a multi-agent collaboration architecture, and convert the second submarine cable data into structured data; Store the submarine cable data from each data source in a graph database, and load submarine cable nodes, landing point nodes and related relationship data from each data source into the graph database to obtain integrated data; Store the integrated data into a MongoDB database to complete submarine cable information fusion.

[0019] Optionally, the first submarine cable data is collected from multiple submarine cable websites through a crawler technology, comprising: Call the API interface of TelegeoGraphy to collect data related to submarine cables; Call the API interface of Infrapedia to collect data related to submarine cables; Manually simulate access to SubmarineNetworks to collect data related to submarine cables; Call the API interface of WikiPedia to collect data related to submarine cables.

[0020] Optionally, the multi-agent collaboration architecture comprises: URL collection agent, responsible for obtaining valid web page URLs; Set the web page parsing agent, responsible for processing JavaScript dynamic loading content; Information extraction agent, responsible for extracting key information from HTML; Data standardization agent, responsible for converting the extracted information into a unified JSON format.

[0021] Optionally, the multi-agent collaboration architecture is used to collect second submarine cable data from submarine cable and operator official websites, further comprising: If the target page is inaccessible or the content does not meet the expectations, record the corresponding link and error information, and skip the crawling task.

[0022] Optionally, the loading process is implemented through a graph query language, wherein: For each optical cable node, query the supplier information and landing point information, For each landing point node, query the basic attributes of the optical cable to which it belongs.

[0023] To achieve the above purpose, the second aspect embodiment of the present application provides a submarine optical cable information collection and fusion device based on a large model agent, comprising: A first module is configured to collect first submarine optical cable data from multiple submarine cable websites through a crawler technology, and convert the first submarine optical cable data into structured data through a preset large model; A second module is configured to collect second submarine optical cable data from submarine optical cable and operator official websites using a multi-agent collaboration architecture, and convert the second submarine optical cable data into structured data; A third module is configured to store the submarine optical cable data from each data source in a graph database, and load submarine optical cable nodes, landing point nodes and related relationship data from each data source into the graph database to obtain integrated data; A second module is configured to store the integrated data in a MongoDB database to complete submarine optical cable information fusion.

[0024] To achieve the above purpose, the third aspect embodiment of the present application provides an electronic device, comprising a processor and a memory in communication with the processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the method of any one of the first aspect.

[0025] To achieve the above purpose, the fourth aspect embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the method of any one of the first aspect.

[0026] To achieve the above purpose, the fifth aspect embodiment of the present application provides a computer program product, which is executed by a processor to implement the method of any one of the first aspect.

[0027] The technical scheme provided by the embodiments of the present application at least brings the following beneficial effects: 1. Improve the data collection efficiency: through the multi-agent collaboration mode, the automatic processing of the complex website structure is realized, and the data collection efficiency is significantly improved; 2. Ensure data quality: using the non-deterministic knowledge graph technology, the conflict problem between multi-source data is effectively solved, and the reliability of data is improved; 3. Good expansibility: modular design is adopted, and new data sources and agents are expanded according to actual needs to realize new processing functions.

[0028] Additional aspects and advantages of the present application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description and drawings or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0029] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings, wherein: Figure 1 A flowchart of a submarine cable information acquisition and fusion method based on a large model agent provided by an embodiment of the present application; Figure 2 A framework diagram of a submarine cable information acquisition and fusion method based on a large model agent provided by an embodiment of the present application; Figure 3 A Telegeography data acquisition flowchart provided by an embodiment of the present application; Figure 4 An Infrapedia data acquisition flowchart provided by an embodiment of the present application; Figure 5 A SubmarineNetworks data acquisition flowchart provided by an embodiment of the present application; Figure 6 A WikiPedia data acquisition flowchart provided by an embodiment of the present application; Figure 7 A flowchart of a multi-agent cooperative acquisition of a submarine cable official website provided by an embodiment of the present application; Figure 8 A data fusion flowchart provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0031] The following problems of the existing method are solved: Data consistency problem: In the prior art, there are problems such as inconsistent update time, inconsistent original source, and inconsistent data format between different data sources, resulting in conflicts between data from different sources and data dimensions. The present application aims to use different submarine cable data to build a unified non-deterministic knowledge graph containing submarine cable-related entities through unified standards and data integration algorithms, improve the consistency between different data sources, and thus ensure the accuracy and reliability of the data.

[0032] Information extraction efficiency problem: Existing rule-based or template-based methods cannot effectively handle complex formats and diverse information, and machine learning-based methods require a large amount of manually annotated data sets for training. Traditional methods are generally inefficient, costly, and difficult to update information. To solve this problem, the present application uses the powerful basic capabilities of large models to automatically extract and structure submarine cable data from natural language text, reducing human intervention and improving extraction efficiency.

[0033] Information exploration and acquisition problem: Existing technologies are often limited by the complex structure of websites, scattered information, and diverse display methods when acquiring and exploring official websites and data sources. The present application proposes a data exploration and collection scheme using large model agents that can perform active interactive operations in a virtual environment, dynamically find and identify submarine cable data in web pages, and realize cross-page and cross-modal information exploration and acquisition.

[0034] The embodiment of the present application proposes a multi-source data acquisition and fusion method for submarine optical cable networks, which utilizes the powerful basic capabilities of existing large models, such as the text understanding and extraction capabilities of GPT-4o and the visual capabilities of GPT-4 Turbo Vision. On this basis, a scheme for collecting and fusing submarine cable data using large model agents is realized. These agents have higher autonomy and can perform active interactive operations in a virtual environment, such as web navigation, dynamic content processing, and multi-page information extraction. This agent scheme combines the generalization capabilities and multi-task processing advantages of large models, effectively dealing with data sources of different formats and structures, and realizing more efficient and automated submarine cable data extraction and fusion, thereby significantly improving the accuracy and completeness of data acquisition.

[0035] Reference Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 The method comprises the following steps: S1, collect first submarine optical cable data from multiple submarine cable websites through crawler technology, and convert the first submarine optical cable data into structured data through a pre-set large model.

[0036] Specifically, step S1 includes the following steps: S11, calling the API interface of TelegeoGraphy to collect data related to submarine cables.

[0037] In the embodiments of the present application, step S11 is configured to call the API interface of TelegeoGraphy to collect data related to submarine cables, so as to realize systematic acquisition and structured management of global submarine cable information. Through this step of the present application, data collection, classification and warehousing can be automatically completed without human intervention, thereby providing a reliable basis for subsequent data analysis and visualization.

[0038] In the embodiments of the present application, first, by calling a plurality of API interfaces covering cables, landing points, suppliers, countries, states and categories such as “in planning” and “in use” according to years, the system automatically collects original data related to submarine cables from the TelegeoGraphy platform. The specific API interfaces used include: https: / / www.submarinecablemap.com / api / v3 / cable / cable-geo.json, used to obtain geographic spatial data of global submarine cables; https: / / www.submarinecablemap.com / api / v3 / cable / all.json, used to obtain basic information of all submarine cables; https: / / www.submarinecablemap.com / api / v3 / landing-point / landing-point-geo.json, used to obtain geographic location information of global cable landing points; https: / / www.submarinecablemap.com / api / v3 / status / {status}.json, used to obtain cable status data, including “in use” and “in planning” and the like; https: / / www.submarinecablemap.com / api / v3 / country / {country}.json, used to obtain submarine cable data related to a specific country; https: / / www.submarinecablemap.com / api / v3 / supplier / {supplier}.json, used to obtain related information of cable suppliers.

[0039] By calling the aforementioned interfaces, the system can comprehensively cover the basic data elements of submarine optical cable systems, including cable routes, landing points, country distribution, supplier information, and construction status. The system acquires data through multi-threaded or asynchronous requests, thereby improving the efficiency and real-time performance of data collection.

[0040] In this embodiment, the data returned by the aforementioned API interface is further automated for storage and management. The system dynamically generates storage paths based on the current date and stores the response results of various data types in JSON format. For example, the system generates folders according to dates (e.g., data_2024-11-12), and further subdivides folder levels within these date directories based on data categories (e.g., cable, landing-point, supplier, etc.). Data files from each API response are categorized and stored according to their data source, ensuring a clear, traceable, and reusable data structure. This categorization method not only facilitates subsequent data updates and comparisons but also provides convenience for data verification and version management.

[0041] In this embodiment, the locally stored data is further structured and processed for database storage. The system traverses the date directory and data category directory, parses the content of each data file, and writes it to the MongoDB database. The MongoDB database is divided according to date, forming a dataset for each date, and multiple collections are created under this database, including submarine cable information, cable route, country, landing station, supplier, activation date, status, etc. This design ensures the traceability of the data in the time dimension and the queryability in the category dimension. Through this organization, the database can support rapid retrieval and analysis based on multiple conditions such as time, country, and status, providing solid data support for subsequent data visualization, submarine cable network analysis, and intelligent operation and maintenance.

[0042] In summary, the embodiments of this application realize the complete process of submarine optical cable data from acquisition to storage by calling TelegeoGraphy's API interface, automating data collection, classifying and storing data, and managing data in a database. It has the advantages of timely data updates, clear structure, and strong scalability.

[0043] S12 calls Infrapedia's API interface to collect data related to submarine optical cables.

[0044] In this embodiment of the application, step S12 is used to call and process Infrapedia’s submarine optical cable data resources. Through a complete process of automated download, parsing, standardization and storage, a multi-source data acquisition capability that complements step S11 is formed, thereby improving the coverage, accuracy and timeliness of submarine cable information.

[0045] In this embodiment, a main directory for storing the data capture results is automatically created based on the current system date. Each data capture task corresponds to a new data directory named after the date, enabling cross-batch data versioning management. Under this date directory, the system pre-creates multi-level subdirectories according to data categories to store different types of data files, such as cable information, landing point information, and service information, ensuring clear data organization, logical consistency, and ease of backtracking and comparison.

[0046] In this embodiment, the system downloads PBF format submarine cable map tile files from a specified URL. The download process is executed programmatically using the wget library. File naming follows a rule of using a triplet of map zoom level (z), x coordinate, and y coordinate as a unique identifier to ensure the uniqueness and manageability of each PBF file. The download link for the PBF file is dynamically generated using the x, y, and z parameters, and the URL template is as follows: url_template=https: / / www.infrapedia.com / map / cables / {} / {} / {}.pbf.

[0047] The system uses nested loops to traverse the z (zoom level) coordinates and corresponding x and y coordinate combinations within a defined range. For each coordinate, it generates a target URL and initiates download requests sequentially, achieving systematic coverage of PBF data within the target area. Before initiating the actual download, the system first checks if the named PBF file already exists in the local target path. If it already exists, it skips the download to avoid bandwidth waste and storage redundancy caused by repeated downloads. If it does not exist, it performs the download and saves the file to a predefined subdirectory path in the current date directory. Simultaneously, it records metadata such as filename, coordinates, download time, and file size to support subsequent data quality auditing and update strategy formulation.

[0048] In this embodiment, after downloading the PBF files, the system performs structured parsing on the batch of PBF files. The parsing process is completed by the StandardParser component: the system iterates through each PBF file, locates and extracts cable element data, focusing on capturing the unique identifier field "_id". After successfully identifying the cable ID, the system further uses regular expressions to retrieve the "eol" (submarine cable service expiration time) field value related to the cable within the same data context. The parsed "cable ID-eol" key-value pairs are incrementally written to the index file of this batch in CSV format, providing a stable index foundation for subsequent interface retrieval and expiration analysis. For missing or abnormal PBF slices (such as those that cannot be parsed, have incomplete fields, or have abnormal encoding), the system records the corresponding file name and coordinate index for subsequent review and re-capture.

[0049] In this embodiment, the system initiates ID-based retrieval requests to multiple Infrapedia-related API interfaces based on the cable IDs extracted from the PBF, retrieving structured attribute data of submarine cables one by one. For each interface response, the system parses and cleans according to field-level rules, extracting and verifying key attributes such as cable name, status (e.g., "under construction," "planned," "in use"), design capacity, fiber pairs, and in-service time, and performs association matching with the "id-eol" index to achieve the combined labeling of expiration fields and main attributes. If the interface returns an empty string or the data object is missing key fields, the system records the corresponding cable ID, request URL, response code, and timestamp to an anomaly list to support subsequent recovery strategies and quality tracking.

[0050] In this embodiment, the system performs standardization and structured storage on successfully parsed cable data. First, the system normalizes the cable names according to a pre-defined "standard name mapping table," mapping the captured original names to standardized names, thus resolving issues of duplication, confusion, and association failures caused by naming differences between multiple data sources. Then, the system constructs a unified JSON object to carry all the cable's attributes, including but not limited to: standard name, original name, unique ID, current status, design capacity, fiber logarithm, service entry time, expiration time (eol), data source identifier, collection date, and parsing version. This JSON is persisted as a file to the corresponding subdirectory of the batch date directory. To meet the lightweight reading requirements of horizontal comparison and downstream toolchains, the system simultaneously summarizes key summary fields (such as standard name, status, design capacity, service entry time, fiber logarithm, and eol) and writes them to a CSV file, enabling "line-by-line" data browsing and export.

[0051] In this embodiment, to improve crawling stability and fault tolerance, the system is designed with a comprehensive exception and retry mechanism. For common exceptions such as network unreachability, DNS resolution failure, timeout, abnormal response status code, response body decoding failure, and missing fields, the system implements a tiered handling strategy based on the error type: transient errors trigger exponential backoff retries, while persistent errors are recorded and added to a backup queue. For objects that fail after multiple retries, the system records their ID, error category, first and last failure times, cumulative number of retries, and the last error information in the exception list, facilitating subsequent manual verification or secondary task scheduling. Logs are generated for all critical steps, including start and completion events, processing quantity, and time statistics for steps such as downloading, deduplication, parsing, requesting, merging, standardization, storage, and database entry, thereby ensuring the observability and auditability of the process.

[0052] In this embodiment, after data processing, the system writes the local structured data into a MongoDB database according to the principle of "time dimension - category dimension". In MongoDB, the system uses the current date as the database name or suffix identifier to ensure the consistency of data on the same day and the independence of cross-day comparisons; a "submarine cable" collection is created under this database as the core storage collection for this step. Before being written into the database, the system performs type validation and unit normalization on the fields according to the schema (e.g., consistency of capacity units, uniformity of time zone and format for time fields), and creates composite indexes on key fields (such as standard names and unique IDs) to support fast retrieval and aggregation analysis based on name, ID, status, and date. At the same time, the system retains data source and processing link metadata fields (such as source platform, original URL, parsing version, mapping table version, and data hash) to achieve end-to-end data lineage tracing and result verifiability.

[0053] Through the above-described process in this application embodiment, step S12 achieves full-link acquisition and governance of Infrapedia submarine optical cable data: it uses coordinate slicing as a tool to achieve system coverage of spatial range; it uses ID indexing as the main line to achieve attribute retrieval and expiration-related merging; it uses name mapping as the key to achieve standardized and unified naming across multiple sources; it uses JSON and CSV in parallel to ensure both original data preservation and structural integrity, while also satisfying lightweight viewing and rapid integration; and it uses MongoDB as the carrier to achieve online data service capabilities that are traceable by date, searchable by field, and statistically available by status. Therefore, this application, while ensuring data integrity, consistency, and availability, lays a solid data foundation for subsequent cross-source comparison, situational assessment, capacity planning, and visualization.

[0054] S13, manually simulate access to SubmarineNetworks to collect data related to submarine optical cables.

[0055] In this embodiment, step S13 is used to collect data related to submarine optical cables by manually simulating access to the SubmarineNetworks website. Unlike the aforementioned automated interface call method, this step adopts a semi-automated web page access and data extraction method, aiming to supplement submarine cable information not covered by the API and achieve more comprehensive submarine cable data integration.

[0056] In this embodiment, a file directory structure for storing the data capture results is first created based on the current system date. The system checks if a folder corresponding to the date exists in the target path; if not, the folder is created automatically. This directory is used to centrally store submarine cable information files and related logs extracted from the SubmarineNetworks website, ensuring clear time dimensions and a well-defined hierarchical structure for data storage, facilitating subsequent management and backtracking.

[0057] In this embodiment, the system employs the Selenium automated testing framework to simulate user actions in a browser by accessing the SubmarineNetworks official website. Selenium automatically launches a browser instance, loads webpage content, and simulates mouse clicks and page scrolling, thereby acquiring dynamically loaded data. The system presets target areas for access, including but not limited to multiple regional pages such as "Trans-Pacific," "Trans-Atlantic," "Inner Asia," and "Inner Europe." Each regional page contains information on several submarine optical cables and landing stations; these pages are the core part of the website showcasing the global distribution of major optical cables.

[0058] Within each regional page, the system uses Selenium to extract all title elements containing information about submarine optical cables and landing stations. It then parses the corresponding HTML structure to identify and extract the cable names and their hyperlinks, as well as the landing station names and their hyperlinks. Cable links typically lead to detailed information pages, including the cable's construction date, length, connected countries, design capacity, and operator information. Landing station links contain the landing station's geographical location, the connected optical cables, and country information. By parsing this structured information, the system obtains a complete mapping relationship between optical cables and landing stations, providing a foundation for subsequent geographic association and network topology analysis.

[0059] In this embodiment, the extracted raw data of optical cables and landing stations are processed using the ChatGPT4o large-scale model for unified structuring. The model performs semantic parsing and standardized extraction of the original webpage content, transforming text data of different formats into a unified structured output, ensuring consistency in field naming, data format, and semantic definitions across different page sources. For example, fields such as "Cable Name," "Submarine System," and "Landing Point" in different expressions on the webpage are uniformly converted into standardized fields such as "cable_name," "landing_point_name," and "link_url," thereby eliminating data inconsistencies caused by differences in manual annotation.

[0060] In this embodiment, the structured results after large-scale model processing are stored in JSON format in local files under the corresponding date directory. Each record contains the name of the optical cable or landing station and its corresponding detailed link. Some data may also include extended fields such as the region, country, and related relationships. This structured storage method ensures data traceability and machine readability, laying the foundation for subsequent database writing, visualization, and statistical analysis. When saving data, the system generates a full data file to ensure the integrity of a single task, while recording metadata such as file generation time and the number of records.

[0061] In this embodiment, to ensure the stability and integrity of data collection, the system employs a robust exception handling mechanism during crawling tasks. When issues such as webpage loading failure, element mislocation, data extraction errors, or network interruptions occur, the system automatically captures the exception information, recording the error type, URL address, timestamp, and error details in a log file. Upon capturing an exception, the system automatically skips the current task and continues processing subsequent webpages, ensuring uninterrupted task execution. The log file is saved in the same date directory for subsequent analysis, debugging, and task reproduction.

[0062] Finally, organize the local data in the local files according to time and category dimensions, and write it into the submarine cable and landing station collections of the corresponding date in the MongoDB database.

[0063] Through the steps described in this application's embodiments, it is possible to systematically collect and organize the latest data on global submarine optical cables and landing stations by manually simulating access to the SubmarineNetworks website, based on Selenium's automated browser technology and large language model structuring capabilities, even without publicly available API support. This method offers advantages such as high flexibility, broad coverage, and the ability to handle unstructured data. It effectively complements the aforementioned automated API data collection scheme, providing solid data support for establishing a multi-source fusion global submarine optical cable database.

[0064] S14 calls the Wikipedia API to collect data related to submarine optical cables.

[0065] In this embodiment, step S14 involves calling the Wikipedia API to collect data related to submarine optical cables. This step combines automated access with semantic analysis to obtain publicly available information on international submarine communication optical cables from Wikipedia, further supplementing and verifying global optical cable data. This method not only obtains the basic attributes of the optical cables but also textual information such as historical background, construction time, connected countries, and technical parameters, thereby enhancing the information dimensionality and completeness of the submarine optical cable database.

[0066] In this embodiment, to ensure the stability of network requests and the feasibility of cross-regional access, the system first configures local proxy service addresses, including HTTP and HTTPS proxies. This step is particularly suitable for restricted or firewalled environments, ensuring that the program can stably access Wikipedia's API interfaces and its web page content through proxy settings, thereby improving the connectivity and success rate of data scraping.

[0067] In this embodiment, the Wikipedia page "List of international submarine communications cables" is selected as the main entry point for data crawling. This page systematically summarizes hyperlinks to major international submarine communications cables worldwide, possessing high authority and coverage. The system first accesses the page's content via API and parses all hyperlinks contained within. Subsequently, the system performs a filtering operation on the extracted link set: removing irrelevant or duplicate links, and removing links related to country names or non-cable-related topics (such as "Japan," "Italy," and "Brazil") according to predefined exclusion rules. Simultaneously, when the system detects a specific termination marker (such as "List of domestic submarine communications cables"), it proactively stops further crawling to limit the crawling scope and prevent the collection of data unrelated to the target topic.

[0068] In this embodiment, the filtered set of links constitutes the target set of optical cables to be crawled. The system accesses each Wikipedia subpage corresponding to each target link and analyzes its content. For each page, the system checks whether its text contains the keywords "cable" and "submarine" to determine whether the page is indeed related to submarine communication optical cables. If the detection result meets the criteria, the page is considered a valid optical cable information page, and its content is included in the crawling scope; otherwise, it is marked as an invalid page, and the saving process is skipped, thereby ensuring the thematic consistency and semantic accuracy of the collected data.

[0069] In this embodiment, for Wikipedia pages deemed valid, the system saves their content as text files (TXT format) named after the fiber optic cable name to ensure file uniqueness and traceability. All files are stored in directories named according to date to achieve time-based data management and version control. Before saving a new file, the system automatically compares it with existing filenames. If a file with the fiber optic cable name already exists, the current saving step is skipped, thus avoiding duplicate data collection and resource waste. This comparison mechanism effectively improves task execution efficiency and prevents the generation of redundant data.

[0070] In this embodiment, if a Wikipedia page becomes inaccessible, a link does not exist, an error status is returned, or a network anomaly occurs, the system will automatically capture the anomaly information and record it in a log file. The error log includes the failed URL, the reason for the error, a timestamp, and the error type, which is used for subsequent task repair or re-capture. When an anomaly occurs, the system will not interrupt the overall task flow but will automatically skip the current target link and continue processing the next target, thereby ensuring task continuity and overall success rate.

[0071] In this embodiment, after all valid page text files have been generated, the system enters the data structuring processing stage. The system uses the GPT-4o model to intelligently parse each saved optical cable text file, extracting structured submarine cable information. This processing includes the identification and extraction of key fields such as cable name, country of origin, landing point, year of commissioning, total length, connecting region, operator, and design capacity. Text files that are empty, unrecognizable, or do not meet the theme requirements are automatically skipped to avoid unnecessary calculations and improve overall processing efficiency. The parsing results are formatted as JSON files, with the filename matching the input TXT files, and stored in the JSON output subdirectory under the same date directory for traceable management.

[0072] In this embodiment, all parsed structured JSON data is categorized and stored according to time and category. The system maintains a clear "date-data type" structure in the local directory, and then writes these JSON files to the corresponding database in the MongoDB database. In MongoDB, two collections are created: "Cables" and "Landing Points." Each JSON object undergoes field validation and standardization before being written to the database to ensure that the data type, field naming, and unit format are consistent with the aforementioned data sources. Simultaneously, the system appends a source identifier (Wikipedia), fetch time, file path, and parsing version information to each record to support data lineage tracing and multi-source consistency verification.

[0073] Through the steps described in this embodiment, the Wikipedia data acquisition module can obtain encyclopedic information on submarine optical cables with low interface dependency and high content coverage, supplementing the data structures of TelegeoGraphy, Infrapedia, and SubmarineNetworks, and achieving the fusion acquisition of multi-source heterogeneous data. This implementation method of this application has advantages such as flexible access, accurate semantic parsing, unified data structure, and strong traceability, providing a highly reliable data foundation for subsequent global submarine cable database construction, information analysis, and geographic visualization.

[0074] S2 uses a multi-agent collaborative architecture to collect data from submarine optical cables and the operator's official website, and transforms the data from the submarine optical cable into structured data.

[0075] In this embodiment, step S2 is used to collect data from the submarine optical cable and its operator's official website using a multi-agent collaborative architecture, and then convert the data into a structured format for storage. This step proposes a modular multi-agent collaborative crawling and processing method based on a Large Language Model (LLM), which can accurately extract target information and semantically organize it in the face of complex web page structures, dynamically loaded content, and unstructured text information, significantly improving the intelligence level and data quality of the crawling task.

[0076] In this embodiment, the system first interacts with Telegeography data stored in a local database (MongoDB) to extract official website links (URLs) of the target submarine optical cable or operator. The system performs deduplication and validity checks on these links, removing those already crawled or stored, resulting in a set of valid links to be collected. This set includes all web resources related to the submarine optical cable topic, including the homepage of the cable construction company, dedicated pages for cable projects, operation status announcement pages, and login point information pages, thus covering all dimensions of the data topic.

[0077] In this embodiment, to improve the scalability and intelligence of the crawling process, the system adopts a modular multi-agent collaborative mechanism, decomposing the overall crawling task into several independently executable agent sub-tasks. This collaborative architecture consists of the following core modules: (1) URL collection agent: responsible for collecting and verifying valid web page links from the database and external sources, filtering invalid, redirected or duplicate URLs, and ensuring the accuracy and uniqueness of the task target; (2) Web page parsing agent: responsible for simulating browser behavior, parsing web page structure, processing content containing dynamically loaded JavaScript, and ensuring that the page content is loaded completely; (3) Information extraction agent: responsible for using a large language model to extract key text information related to the target entity from the HTML document of the web page, including submarine optical cable name, technical parameters, landing point, status, operator information, etc.; (4) Data standardization agent: responsible for converting the extracted content into a unified JSON structure, performing field standardization, data validation and redundancy cleanup to ensure data consistency and integrability.

[0078] In this embodiment, the agents collaborate through message queues or intermediate caches. During task scheduling, the system dynamically generates task configuration files based on user-defined target information types (such as "optical cable name," "design capacity," "landing point country," and "operational status"). The agents automatically allocate, execute, and aggregate the crawling tasks accordingly. For each target link, the agents sequentially perform webpage access, content loading, text extraction, and structured processing. After obtaining the complete page DOM, the webpage parsing agent passes all text content to the information extraction agent, where a large language model performs semantic parsing and target information extraction. This model possesses contextual understanding and entity recognition capabilities, enabling it to accurately locate and extract key information from complex natural language descriptions, achieving efficient structured transformation of unstructured webpage content.

[0079] In this embodiment of the application, the parsing result is output in the form of multi-field structured data, including but not limited to the following: The name of the target entity (such as the name of the optical fiber system or the name of the landing station); Core attributes (such as total fiber optic cable length, design capacity, number of fiber pairs, year of construction or commissioning, technical standards, etc.); Auxiliary information (such as the country of login, the geographical location of the login site, service status, construction unit, investor, maintenance party and related links, etc.).

[0080] The system performs standardization and consistency checks immediately after data generation, automatically converting different formats of the same field (such as unit differences, inconsistent date formats, etc.) to ensure field consistency and compliance with subsequent database entry standards. The final result is stored in JSON format, with each field corresponding to a unique information type, forming a clear and semantically consistent data structure.

[0081] In this embodiment, the system is configured with a comprehensive fault tolerance and logging mechanism for abnormal and error situations. When the target page is inaccessible, its content is empty, the response times out, or the extraction results do not meet expectations, the system automatically captures the abnormal information, records the failed URL, error type, timestamp, and task number, and skips the current task to continue executing subsequent crawling, ensuring the continuity and stability of the overall task. The error log will serve as an important basis for supplementary crawling and manual verification.

[0082] In this embodiment, the crawled data is organized and managed according to date. The system generates a separate date folder for each task and names the JSON result file of each task in this directory with an incremental sequence number (e.g., "001.json", "002.json") to ensure consistency in file naming rules and ease of retrieval. After the crawling task is completed, the system performs a consistency check on all extracted results, removes redundant fields and error messages, and automatically updates the "processed link set" in the local database to prevent subsequent tasks from repeatedly crawling the same web page resources.

[0083] In this embodiment, the final processed structured data is organized and stored according to time and category dimensions, and then written to the corresponding database in the MongoDB database. In this database, the system creates two collections: "Cables" and "LandingPoints." Before data is entered into the database, the system performs field mapping and schema validation to ensure that the field types, names, and units are consistent with the data collected in steps S11 to S14, thereby achieving cross-source data fusion and unified management. Each record includes the source site identifier, collection time, capture version, and task number to achieve data traceability and version control.

[0084] Through the steps described in this embodiment, a multi-agent collaborative mechanism can be used to achieve deep analysis and data extraction of complex web page structures, effectively ensuring data consistency and traceability while improving crawling accuracy and efficiency. This method combines the semantic understanding capabilities of a large language model with an automated crawling process to form a highly scalable and robust submarine optical cable website data acquisition solution, providing solid technical support for the intelligent construction of a global submarine cable database.

[0085] S3. The submarine optical cable data from various data sources are stored in a unified graph database. Submarine optical cable nodes, landing point nodes and their related relationship data are loaded from various data sources into the graph database to obtain the integrated data. In this embodiment, step S3 is used to uniformly store and integrate submarine optical cable information from different data sources into a graph database to achieve multi-source data association fusion and structured query. Based on the aforementioned data acquisition and standardization processing, this step uses a graph database (Neo4j) to model entity nodes and relationships, thereby establishing a unified data knowledge graph. The integrated results are then written into a MongoDB database for storage, forming a traceable, queryable, and scalable global submarine optical cable data system.

[0086] In this embodiment, the data sources include five main channels: Telegeography, Infrapedia, SubmarineNetworks, Wikipedia, and data from submarine optical cable websites. These five types of data provide key information about submarine optical cable systems from different dimensions, covering geographical distribution, operational status, supplier information, landing station locations, and historical data. The system first preprocesses and structures the data to ensure consistency in field naming, encoding formats, and semantic definitions across different sources, laying the foundation for subsequent graph database import operations.

[0087] In this embodiment, the system loads the integrated data into the Neo4j graph database. The graph database uses a "node-relationship-attribute" data model to represent complex entity relationship networks. This primarily includes three types of core nodes: (1) Submarine cable node (Cable Node) is used to represent each submarine cable entity, including basic attributes such as cable name, length, design capacity, year of commissioning, and service status; (2) Supplier Node, which represents the design or construction unit of the optical cable, including attributes such as supplier name, country of origin, and number of major projects; (3) Landing Point Node: This node represents the geographical location of the optical cable landing, including the landing point name, country, geographical coordinates, and the number of connected optical cables.

[0088] In a graph model, nodes are connected by directed or undirected relationships to form a complete topology. Key relationships include: supplied_by: Connects the submarine optical cable node to the supplier node, indicating that the optical cable was built or maintained by a certain supplier; landing_at: Connects the submarine optical cable node to the landing point node, indicating that the optical cable is landed at this landing point; connects_to: indicates that two landing points are interconnected via a fiber optic cable.

[0089] Before data integration, the system first connects to the Neo4j database and performs initialization and cleanup operations on the target set, deleting historical residual data to ensure that the imported new data does not conflict with the old data. All operations are performed using the graph query language Cypher to guarantee the atomicity and controllability of data import and updates.

[0090] In this embodiment, the system loads node and relationship data from various data sources sequentially, and implements structure mapping and attribute writing through Cypher statements.

[0091] For each submarine optical cable node, the system executes the following query and integration logic: It queries the cable's supplier nodes using the `supplied_by` relationship, extracting the supplier name, region, and project participation information; it queries the cable's landing point nodes using the `landing_at` relationship, extracting the landing point ID, name, country, latitude and longitude, and other information; and it combines the cable's basic attributes with the aforementioned related information to form a complete optical cable information dictionary. This dictionary is then stored in the "cable" collection in the MongoDB database, while the cable name, unique ID, and main attribute index information are simultaneously written to the "all" collection for subsequent fast retrieval.

[0092] In this embodiment of the application, for each landing point node, the system performs a reverse query operation: by retrieving all submarine optical cable nodes connected to the landing point through the landing_at relationship, the name, capacity and status of the optical cable are extracted; combined with the basic attributes of the landing point (including ID, name, country and geographic coordinates), a complete landing point information record is formed; the sorted landing point data is stored in the "landing_point" collection of MongoDB to support retrieval and statistics by geographic dimension.

[0093] In this embodiment, MongoDB serves as the final structured data storage system, providing unified data archiving and hierarchical indexing. The integrated Neo4j data is divided into three main sets: the cable set, which stores detailed information about submarine optical cables, including basic attributes and information about all associated nodes; the landing_point set, which stores detailed attributes of landing points and reference information connecting to optical cables; and the all set, which stores basic index information about optical cables and landing points, used for front-end queries and fast matching.

[0094] During the data import process, the system performs consistency checks and format standardization on fields to ensure that identical attributes (such as country name, date format, and unit) from different sources remain consistent. Simultaneously, it adds a source identifier (such as "source: Telegeography") and an import timestamp to each node and relationship to enable cross-source traceability and data version management. The system also creates composite indexes (such as a combined index of fiber optic cable name and landing point name) for frequently used fields to improve query performance.

[0095] Through the steps described in this application's embodiments, a global submarine optical cable knowledge graph is established, enabling fusion management and complex query analysis of cross-source data. The Neo4j graph database provides powerful relational modeling capabilities and visualization query functions, while MongoDB, as a high-performance document storage system, ensures data persistence and fast access. This application's solution effectively breaks down structural barriers between multi-source data, achieving multi-dimensional association, unified modeling, and intelligent storage of submarine optical cable data, providing a solid data foundation for subsequent geographic visualization, network structure analysis, and submarine cable operation status monitoring.

[0096] S4. The integrated data sets are stored in the MongoDB database to complete the fusion of submarine optical cable information.

[0097] In this embodiment, step S4 is used to store the integrated submarine optical cable data in separate sets into a MongoDB database, thereby completing the unified fusion and storage of submarine optical cable information. This step, based on the aforementioned entity association and relationship modeling using a graph database (Neo4j), further realizes hierarchical storage and index construction of structured data, providing support for subsequent data retrieval, query optimization, and visualization analysis.

[0098] In this embodiment, the MongoDB database adopts a document-oriented storage architecture, featuring flexible schema design and high concurrency access performance, making it suitable for storing fiber optic cable and landing point data with multi-level nested structures. Before writing data, the system performs field mapping and format standardization on the Neo4j integration results to ensure consistency in data type definitions, naming rules, and hierarchical structure, and establishes unique constraints on key fields to prevent duplicate imports.

[0099] In this embodiment of the application, the structure of the MongoDB database is designed as follows: (1) Cable Collection: This collection is used to store detailed information about each submarine optical cable. Each document corresponds to a complete optical cable record, which includes the following fields: cable ID, name, length, design capacity, number of fiber pairs, and year of commissioning (RFS). It also includes information about the associated supplier and landing point.

[0100] This collection enables centralized management and conditional querying of information throughout the entire lifecycle of a single optical cable, such as filtering by supplier, country, or status.

[0101] (2) landing_point collection: This collection is used to store detailed information about the fiber optic cable landing points. Each document corresponds to a landing point entity, and the main fields include: landing point ID, name, country, geographical coordinates, and information on all fiber optic cables connected to the landing point.

[0102] This set enables reverse mapping from landing points to optical cables, supporting geographic location-based display of submarine cable network topology and path calculation.

[0103] (3) The 'all' set: This set serves as an index set, used to store basic information indexes of all optical cables and landing points for quick querying and global retrieval. Key fields include: storing basic index information for all optical cables and landing points for fast searching.

[0104] By using the all collection, the system can quickly locate the entity ID and its corresponding detailed document location, thereby accelerating the cross-collection retrieval process.

[0105] Through the design described in this application embodiment, the MongoDB database achieves the decentralized storage and structured fusion of submarine optical cable data. This solution not only ensures the consistency and manageability of data from different sources, but also provides high-performance data support for subsequent analysis, calculation, map rendering, intelligent retrieval, and visualization, thereby completing the systematic integration and unified management of submarine optical cable information.

[0106] To implement the above embodiments, the present invention also proposes a submarine optical cable information acquisition and fusion device based on a large-scale intelligent agent model. The device includes: The first module is used to collect data of the first submarine optical cable from multiple submarine cable websites through web crawling technology, and to convert the data of the first submarine optical cable into structured data through a preset large model. The second module is used to collect data from the submarine optical cable and the operator's official website using a multi-agent collaborative architecture, and to convert the data into structured data. The third module is used to store submarine optical cable data from various data sources into a graph database, and to load submarine optical cable nodes, landing point nodes and their related relationship data from various data sources into the graph database to obtain integrated data. The second module is used to store the integrated data sets into a MongoDB database to complete the fusion of submarine optical cable information.

[0107] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0108] To implement the above embodiments, the present invention also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0109] To implement the above embodiments, the present invention also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0110] To implement the above embodiments, the present invention also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0111] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0112] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for collecting and fusing submarine optical cable information based on a large-scale intelligent agent model, characterized in that, Includes the following steps: Data on the first submarine optical cable was collected from multiple submarine cable websites using web crawling technology, and then the data was transformed into structured data using a pre-set large model. A multi-agent collaborative architecture is used to collect data from submarine optical cables and operators' official websites, and the data from the second submarine optical cable is transformed into structured data. Submarine optical cable data from various data sources are stored in a unified graph database. Submarine optical cable nodes, landing point nodes, and their related relationship data are loaded from each data source into the graph database to obtain integrated data. The integrated data sets are stored in a MongoDB database to complete the fusion of submarine optical cable information.

2. The method according to claim 1, characterized in that, The process of collecting data on the first submarine optical cable from multiple submarine cable websites using web crawling technology includes: Call TelegeoGraphy's API to collect data related to submarine optical cables; Call Infrapedia's API to collect data related to submarine optical cables; Manually simulate access to SubmarineNetworks to collect data related to submarine optical cables; The API interface of Wikipedia is used to collect data related to submarine optical cables.

3. The method according to claim 2, characterized in that, The multi-agent collaborative architecture includes; The URL-collecting agent is responsible for obtaining valid webpage URLs. Set up a webpage parsing agent to handle dynamically loaded JavaScript content; The information extraction agent is responsible for extracting key information from HTML. The data standardization agent is responsible for converting the extracted information into a unified JSON format.

4. The method according to claim 3, characterized in that, The use of a multi-agent collaborative architecture to collect second submarine cable data from submarine cables and operator websites also includes: If the target page is inaccessible or the content does not meet expectations, the corresponding link and error message are recorded, and the crawling task is skipped.

5. The method according to claim 4, characterized in that, The loading process is implemented using a graph query language, where: For each fiber optic cable node, query the supplier information and login point information. For each landing point node, query its associated optical cable and basic attributes.

6. A device for collecting and fusing submarine optical cable information based on a large-scale intelligent agent model, characterized in that, include: The first module is used to collect data of the first submarine optical cable from multiple submarine cable websites through web crawling technology, and to convert the data of the first submarine optical cable into structured data through a preset large model. The second module is used to collect data from the submarine optical cable and the operator's official website using a multi-agent collaborative architecture, and to convert the data into structured data. The third module is used to store submarine optical cable data from various data sources into a graph database, and to load submarine optical cable nodes, landing point nodes and their related relationship data from various data sources into the graph database to obtain integrated data. The second module is used to store the integrated data sets into a MongoDB database to complete the fusion of submarine optical cable information.

7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-5.

Images