Unstructured data acquisition and query system combining RPA and visual language model

Abstract

This invention belongs to the field of artificial intelligence technology, specifically relating to a system for acquiring and querying unstructured data by combining RPA and a visual language model. The system includes: an RPA timed acquisition module, used to automatically access target visualization pages according to a preset scheduling strategy using RPA technology; after the page is fully rendered, it performs screenshot acquisition, converts the screenshot image into a string with a unified encoding format, and stores it in a knowledge base; simultaneously, it configures a semantic summary for each target page, associates the summary with the encoded image and the acquisition timestamp, stores it, and writes it into the knowledge base; a retrieval and recall module, used to receive natural language questions, and based on the degree of matching between the question semantics and the summary in the knowledge base, recalls the most relevant page snapshots from the page records; and an understanding and answer display interface module, used to input the encoded image of the recalled page snapshot and the user question into a visual language model, where the model performs fine-grained visual semantic understanding of the image and generates a natural language answer.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to an unstructured data acquisition and query system that combines RPA and visual language models. Background Technology

[0002] In recent years, with the rapid development of Large Language Model (LLM) and Natural Language Processing (NLP) technologies, users have placed higher expectations on interactive methods that allow them to directly query business data using natural language. They expect LLM to automatically return accurate answers or generate corresponding analytical conclusions through semantic queries. In structured database scenarios, this requirement can be partially met through technologies such as Natural Language to SQL (NL2SQL) and semantic parsers. However, for unstructured data (including documents, images, and other formats) primarily presented through visual pages, existing methods still face three core challenges:

[0003] 1. Difficulty in data acquisition: In the absence of standard interfaces, traditional web crawlers struggle to reliably extract dynamically rendered content;

[0004] 2. Weak semantic understanding: Pure text models cannot understand the spatial relationships, coordinate axis meanings, legend mappings, and other visual semantics in charts;

[0005] 3. Knowledge fragmentation: Even if a screenshot of the page or the DOM structure is obtained, there is a lack of a mechanism to transform it into a knowledge representation that is inferable, traceable, and reusable.

[0006] To address these challenges, the industry has gradually introduced various intelligent technologies to attempt to overcome bottlenecks. On the one hand, Robotic Process Automation (RPA) can simulate human operation and collect data from web or desktop application interfaces without a physical interface, demonstrating strong environmental adaptability. On the other hand, multimodal large models, represented by Vision Language Models (VLM), are beginning to demonstrate the ability to jointly understand page screenshots, charts, and interface elements. They can perform text semantic alignment, layout structure recognition, and cross-modal reasoning based on visual perception, thereby supporting a high-level understanding of "what the page is expressing and what semantic roles each area plays." In addition, Retrieval Augmented Generation (RAG) technology effectively improves the factual accuracy and interpretability of question-and-answer results by combining external knowledge bases with language models.

[0007] Despite significant progress in multimodal understanding, automated data collection, and intelligent question answering, the existing technology system still has systemic shortcomings when dealing with unstructured visual data query scenarios such as business data dashboards, complex operational report web pages, and management dashboards. These shortcomings are as follows:

[0008] 1. The lack of an effective mechanism for utilizing "front-end-only" data results in high-value information remaining in a "semantic black box" state for a long time:

[0009] Current mainstream NL2SQL systems heavily rely on structured data sources. Even when some solutions introduce multimodal capabilities, their role is primarily focused on enhancing user input understanding or result presentation, rather than extracting queryable knowledge from existing visualization interfaces. However, in many enterprise scenarios, key business metrics are often presented only in front-end visualizations after undergoing complex ETL, aggregation, or permission filtering logic, with the original data interfaces not publicly accessible due to security, cost, or architectural reasons. While these pages contain extremely high decision-making value, they become "read-only and unusable" information silos in the existing question-answering system because they cannot be directly parsed by downstream intelligent systems. Although some research has attempted to use VLM for end-to-end question answering of page screenshots, such methods typically only support isolated reasoning based on a single image and a single question, failing to establish a cognitive model of the page as a stable knowledge carrier. Especially when page content is updated daily while the layout remains unchanged, existing VLM solutions cannot reuse historical understanding results, requiring reprocessing of the entire image each time, wasting computing power and making it difficult to guarantee consistency of answers over time, thus failing to support long-term enterprise-level deployment requirements.

[0010] 2. Existing automated data collection and knowledge building methods are disconnected, making it difficult to form a sustainably evolving page-level knowledge base:

[0011] For dynamic visualization pages, while RPA technology can achieve non-intrusive data scraping and OCR / CV technology can extract some text and numerical values, the output of these methods is mostly raw pixels, fragmented text, or unannotated JSON fragments, lacking a unified knowledge representation paradigm. More importantly, existing solutions generally treat data collection and question answering as a one-off pipeline, without designing a knowledge accumulation mechanism for the characteristics of "stable layout and dynamic content." They also fail to perform template abstraction of page structure, version management of daily snapshots, or introduce manual or semi-automatic high-level semantic summarization.

[0012] Furthermore, when the number of pages grows to dozens or even hundreds, if each query needs to process all historical screenshots in full, it will lead to storage explosion and computational redundancy. The lack of an effective page-level index and summary retrieval mechanism makes it difficult for the system to quickly locate relevant pages among massive amounts of snapshots, severely limiting its usability and scalability.

[0013] 3. The multimodal question-answering process lacks a "retrieval-focusing" collaborative architecture, making it difficult to balance efficiency and accuracy:

[0014] Most current solutions integrating RPA and VLM adopt a coarse-grained end-to-end model: once a user question is received, all potentially related page images are fed into a large model for joint inference. This approach is acceptable in small-scale experiments, but faces three major bottlenecks in real-world business environments:

[0015] Inefficient computation: High-resolution screenshots or complex DOM trees cause a surge in model input tokens, significantly extending response time and making it difficult to meet real-time interaction requirements;

[0016] Limited inference accuracy: Large models are easily distracted by irrelevant decorative elements, auxiliary explanatory text, or non-target charts on the page, making it difficult to accurately focus on the area involved in the user's problem;

[0017] Poor system scalability: adding a new report page requires adjusting the entire processing flow, lacking modular and decoupled design.

[0018] Crucially, existing solutions lack a two-stage collaborative mechanism of "semantic retrieval first, then visual focusing." Specifically, they fail to first quickly filter out the most relevant page snapshots based on semantic matching between the user's question and the manually extracted page summary; then, they fail to send this snapshot along with the question to the Visual Model (VLM) for refined visual reasoning. This deficiency leads to significant fluctuations in question-answering accuracy and high resource consumption when dealing with multi-page, multi-time-point scenarios, and makes it difficult to support the natural interaction logic of "finding the right page first, then answering the right question."

[0019] Therefore, it is very important to design an unstructured data acquisition and query system that combines RPA and visual language models to achieve efficient, accurate, and scalable intelligent question answering capabilities without modifying the original business system or requiring underlying data interfaces. Summary of the Invention

[0020] This invention aims to overcome the problems in existing technologies when performing natural language queries on visual unstructured pages such as business dashboards, operational reports, and management dashboards with fixed layouts but dynamically updated data. These problems include reliance on structured interfaces, lack of knowledge accumulation mechanisms, imbalance between question-answering efficiency and accuracy, and high deployment and adaptation costs. The invention provides a system that combines RPA and visual language models to achieve efficient, accurate, and scalable intelligent question-answering capabilities without modifying existing business systems or requiring underlying data interfaces.

[0021] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0022] An unstructured data acquisition and query system that combines RPA with visual language models, including:

[0023] The RPA timed acquisition module is used to automatically access the target visualization page according to a preset scheduling strategy using RPA technology. After the page is fully rendered, it performs screenshot acquisition and converts the screenshot image into a string with a unified encoding format, which is then persistently stored in the knowledge base. At the same time, it configures a high-level semantic summary for each target page or corresponding logical area, associates the semantic summary with the corresponding encoded image and acquisition timestamp, and writes it into the knowledge base in a structured form.

[0024] The retrieval and recall module is used to receive natural language questions input by users and, based on the degree of matching between the question semantics and the semantic summary in the knowledge base, recall at least one most relevant page snapshot from the page records that meet the time constraints.

[0025] The Understanding and Answer Display Interface module is used to input the encoded image of the recalled page snapshot and the user question into the Visual Language Model (VLM). The VLM performs fine-grained visual semantic understanding of the image and generates a natural language answer.

[0026] Preferably, the RPA timed data collection module supports segmented screenshot collection of the target visualization page; the segmented screenshot collection specifically refers to:

[0027] The target visualization page is divided into multiple logical blocks, and screenshot, encoding, and summary configuration are performed independently for each logical block to achieve fine-grained knowledge modeling.

[0028] Preferably, in the RPA timed acquisition module, the screenshot images are uniformly encoded as Base64 encoded strings; the strings are stored in the knowledge base in plain text form, along with semantic summaries, acquisition timestamps, and page / region identifiers.

[0029] Preferably, in the RPA timed data collection module, the semantic summary is pre-configured by business personnel or system administrators, describing the core business meaning, key indicators, analysis dimensions, time granularity, and display logic of the page in natural language; the semantic summary does not change with the daily update of the underlying data of the page, and supports one-time configuration.

[0030] Preferably, the retrieval and recall module specifically includes the following process:

[0031] Natural language problems in receiving user input and natural language problems The set of manually summarized texts corresponding to the page records in the knowledge base that meet the time requirements. Semantic similarity matching is performed, and a context embedding model is used to map the question and each summary into dense vectors respectively: ; ;

[0032] By rearrangement model Calculate the similarity between the question vector and each summary vector. As a correlation score : ;

[0033] Finally, the candidate page set is reordered based on the reordering score, and the top K most relevant page records are selected as the recall results for subsequent use by the visual language model.

[0034] Preferably, the understanding and answer display interface module specifically includes the following process:

[0035] The Base64 encoded image of the recalled page snapshot is combined with the user's original question text to construct a multimodal input. At the same time, a visual language model is called to identify the chart type, coordinate axis, legend and key data points of the encoded image. Combined with the semantics of the user's question, cross-modal reasoning is completed, and finally the answer in structured or natural language format is output.

[0036] Preferably, the unstructured data acquisition and query system that combines RPA and visual language models supports multi-turn dialogue context management and historical data backtracking query. By reusing historical snapshots and summary records, it enables data comparison and trend analysis at different points in time on the same page.

[0037] Preferably, the target visualization page refers to a front-end display interface with a fixed layout, dynamic data content, and no open data interface, specifically including: business monitoring dashboard, operation analysis dashboard, real-time report page, and management dashboard.

[0038] Compared with the prior art, the beneficial effects of this invention are: (1) This invention uses RPA to automatically access the target visualization page at regular intervals, captures the complete view after ensuring that the page is fully rendered, and converts the image to Base64 in a unified manner. The structured storage of encoded strings fundamentally solves the problem of unreliable and unreproducible data collection in scenarios such as remote desktop, no API interface or front-end dynamic loading, and realizes stable, lossless and standardized retention of unstructured page data; (2) The present invention adopts the knowledge construction paradigm of "artificial semantic summary + page snapshot", in which business personnel configure a high-level summary describing the core business meaning, indicator dimensions and display logic of the page at one time. The summary does not change with the daily data changes. Only the image and timestamp need to be updated later, which greatly reduces the maintenance cost of the knowledge base and naturally supports advanced analysis needs such as historical backtracking and cross-period comparison; (3) The present invention constructs a two-stage hybrid question-and-answer architecture of "semantic retrieval and positioning + visual focus and answering". When the user asks a question, the snapshot ranked in the top K positions is quickly recalled from the candidate pages that meet the time constraints based on the embedding model and re-ranking mechanism, avoiding indiscriminate reasoning on massive images and significantly improving system efficiency; (4) The present invention will recall highly relevant page snapshots. The VLM is input in conjunction with user questions. It utilizes the VLM's fine-grained understanding of chart structure, coordinate axes, legends and numerical labels to generate accurate and interpretable natural language answers, realizing a complete closed loop from "collection-storage-retrieval-understanding-question-answering"; (5) This invention can achieve efficient, accurate and stable querying of complex visualization pages with fixed layout but dynamically changing data without relying on the underlying structured data interface, without modifying the original business system, and without intruding on the front-end code; It not only breaks through the strong dependence of traditional intelligent question-and-answer systems on structured sources, but also takes into account high accuracy and low resource consumption through a phased processing mechanism, significantly improving the applicability, robustness and scalability of the system in large-scale, high-concurrency enterprise environments; At the same time, it greatly reduces the threshold and waiting cost for business personnel to obtain key data, enabling them to obtain insights instantly like using a search engine, providing a new technical path with industrial value for building a highly available, zero-adaptation, intelligent unstructured data question-and-answer system for real business scenarios. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the functional architecture of the unstructured data acquisition and query system that combines RPA and visual language model according to the present invention. Figure 2 This is a functional flowchart of the retrieval and recall module in this invention; Figure 3This is a schematic diagram illustrating a process for an unstructured data acquisition and query system combining RPA and a visual language model to acquire data offline in a data question-and-answer scenario on an e-commerce platform, as provided in an embodiment of the present invention. Figure 4 This is a flowchart illustrating the online question-and-answer process of an unstructured data acquisition and query system combining RPA and visual language models provided in an embodiment of the present invention, in a data question-and-answer scenario on an e-commerce platform. Detailed Implementation

[0040] To more clearly illustrate the embodiments of the present invention, specific implementation methods will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0041] This invention provides a system for acquiring and querying unstructured data that combines RPA and a visual language model. It is suitable for visualization page scenarios with fixed layouts but dynamically updated data content. The overall technical solution is divided into an offline knowledge construction stage and an online intelligent question-answering stage. Through the collaborative work of core functional modules, it enables natural language query capabilities for front-end visualized data.

[0042] The system functional architecture proposed in this invention is as follows: Figure 1 As shown, the overall method and system mainly consist of three modules:

[0043] The RPA timed data collection module automatically launches the browser according to a preset scheduling strategy (fixed time daily or event-triggered) using RPA technology. It simulates real user operations, completing interactive processes such as login, redirection, and filtering, and accurately accesses preset visual page URLs. After ensuring the page is fully rendered by element visibility detection or loading status judgment, the system automatically captures the complete view or a specified area image. The acquired screenshots are then converted into Base64 encoded strings and persistently stored as a unified, database-embedded plain text format. This encoding method effectively avoids the path dependency, permission restrictions, format compatibility, and cross-platform synchronization problems associated with traditional file storage, while fully preserving the original page's visual layout, colors, chart styles, and other key contextual information, providing a high-quality, high-fidelity input foundation for accurate subsequent understanding of VLM.

[0044] Then, business personnel or system administrators pre-configure a high-level semantic summary for each target page (or a specific data area within the page). This summary describes the core business meaning, key indicators, analysis dimensions, time granularity, and display logic of the page in natural language, exhibiting high stability and remaining unchanged regardless of daily changes in the underlying data. Therefore, only one summary needs to be configured for the same page or area, and subsequent RPA collections only require updating the corresponding Base64 image and collection timestamp, significantly reducing maintenance costs. For visualization pages with high information density and complex content, the system supports a region-based capture strategy: dividing the page into multiple logical blocks, and independently executing screenshot and summary configuration for each block, thereby achieving finer-grained knowledge modeling and subsequent accurate question answering.

[0045] Finally, all collected results are written into a structured knowledge base storage table. Each record contains the following core fields: a unique page identifier, a region identifier, a collection timestamp, a Base64-encoded screenshot of the page, and a manually configured semantic summary. This structured knowledge base not only supports efficient semantic retrieval and historical backtracking, but also naturally adapts to advanced application scenarios such as cross-time comparison and multi-turn dialogue context management, providing a solid data foundation for the entire intelligent question-answering system.

[0046] The retrieval and recall module: After receiving the natural language question input by the user, the system first pre-filters the page records in the knowledge base based on the time range implied or explicit temporal semantics in the question. Then, it uses a context embedding model to return multiple semantically related summary information. Finally, it uses a reranker model to filter out the Top-K page records. The retrieval and recall module's workflow is as follows: Figure 2 As shown:

[0047] The system's natural language input problem And compare it with the set of manually summarized texts corresponding to the page records in the knowledge base that meet the time requirements. Semantic similarity matching is performed, and an embedding model is used to map the question and each summary into dense vectors: ; ;

[0048] By rearrangement model Calculate the similarity between the question vector and each summary vector. As a correlation score : ;

[0049] This model captures fine-grained semantic matching between questions and summaries through a cross-attention mechanism, significantly outperforming the shallow similarity calculation of single-tower embedding models. Finally, the system reorders the candidate set based on the reordering score and selects the Top-K page records that best match the user's intent as the recall results for use by downstream visual language models.

[0050] Understanding the Answer Display Interface: After the retrieval and recall module filters out the Top-K most relevant page records, the system enters the Understanding the Answer and Display Interface module. First, relevant page records are selected from the recall results, and their corresponding Base64 encoded screenshots are extracted. At the same time, the user's original question text is retained, together forming the multimodal input of VLM. Subsequently, VLM performs fine-grained analysis on the image, identifies the chart type, coordinate axis, legend and key data points, and combines the question semantics to complete cross-modal alignment and reasoning, finally generating a structured natural language answer, which is returned to the user through a web interface, enterprise IM or API.

[0051] Based on the technical solution of this invention, the implementation process of this invention in practical applications will be illustrated through the following case scenarios. The specific application implementation scheme is as follows:

[0052] like Figure 3 and Figure 4 As shown, the following detailed example illustrates a typical application scenario—a data-driven question-and-answer scenario on an e-commerce platform:

[0053] Scene background:

[0054] The operations team of an e-commerce platform tracks sales performance across various regions nationwide daily through an internal "Sales Monitoring Dashboard." This page is dynamically generated by the front-end system and includes the following core components:

[0055] Top KPI card: Displays the day's GMV (GMV represents the total value of all completed orders on the platform), order volume, and number of new users;

[0056] The line chart in the middle section shows the national GMV trend over the past 30 days.

[0057] The table at the bottom lists yesterday's sales figures and the month-on-month change rate for each province.

[0058] The page has a fixed layout and clear structure, but the data is automatically updated every day at midnight, and no API interface or database access is provided to the outside world, making it impossible for traditional intelligent data query systems based on structured data to connect. Operations staff frequently need to answer questions such as "What was the sales volume of a certain district today?" and "What was the month-on-month change rate for each province yesterday?" Relying on manual viewing of the dashboard is inefficient and prone to errors.

[0059] Implementation steps:

[0060] 1. Offline storage: This module can capture updates after data is updated, achieving synchronous updates without affecting users who are currently using the data. For example... Figure 3 The following is a flowchart of the offline data acquisition process:

[0061] 1) RPA scheduled page access: Configure the system to use an RPA task to automatically start the browser at 01:00 every day, simulate login operation, navigate to the target page URL, and wait for the page to be fully rendered before entering the screenshot stage.

[0062] 2) RPA Multi-Region Capture: Due to the dense information and distinct modules on the page, the system adopts a multi-region capture strategy: the page is divided into three logical regions: Region A: Top KPI card area; Region B: Middle line chart area; Region C: Bottom table area. RPA performs precise captures on each of these three regions to ensure that the image of each region is independent and complete, avoiding cross-region interference.

[0063] 3) Image Encoding and Manual Summary Configuration: Each region's screenshot is converted to a Base64 encoded string and stored in a unified format. Business personnel configure a high-level semantic summary for each region for subsequent semantic retrieval: Region A Summary: "Core business metrics for the day, including GMV, order volume, and new user count."; Region B Summary: "National GMV trend curve for the past 30 days, including year-on-year change rate."; Region C Summary: "Lists yesterday's sales revenue and month-on-month change rate for each province."

[0064] 4) Information Storage: All collected results are written to the knowledge base storage table in a structured format. Each record contains the following fields: Table 1 Data table of data collection results storage fields

[0065] Note: Multiple regions can be recorded on the same page, supporting fine-grained knowledge modeling.

[0066] 2. Online Question Answering System: This module is responsible for real-time semantic retrieval and intelligent question answering based on a pre-built knowledge base when a user initiates a natural language query, achieving a complete closed loop from question input to answer output. The system adopts a two-stage architecture of "first retrieval and location, then visual understanding," significantly improving response efficiency while ensuring high accuracy, and supporting multi-turn dialogues and historical data backtracking. The entire process is as follows: Figure 4 As shown:

[0067] 1) Receiving user questions: Users enter questions in the enterprise's internal intelligent assistant interface, such as "What is the sales volume of a certain district today?" After receiving the question, the system first identifies the time keywords in it and determines the date range of the data to be queried.

[0068] 2) Semantic Matching and Page Location: The system performs a semantic comparison between the user's question and all page summaries in the knowledge base that meet the time criteria. Since a clear business description has been configured for each region during the offline phase, the system can quickly determine that the content the user is most interested in is most likely located in the "bottom table area." Therefore, the system retrieves the snapshot record for region C collected today from the knowledge base.

[0069] 3) Using the VLM model for question-and-answer display: The user's question, "What was the sales revenue of a certain district today?", along with the Base64 encoded location of the image, is fed into the VLM. The model automatically recognizes the information in the image, locates the content "a certain district" and "sales revenue," reads the corresponding value "¥2,380,000," and generates a natural language answer: "According to the collected data from the large screen, the sales revenue of a certain district today is ¥2,380,000."

[0070] This invention proposes an unstructured data acquisition and query system that combines RPA technology with a visual language model. By automating the collection of data from visualized pages, semantic summarization modeling, semantic retrieval and localization, and visual focusing understanding, it enables natural language querying of business data presented only in a front-end format. This invention transforms the originally "read-only and unusable" visualized pages into a knowledge carrier that can be continuously perceived, traced, and reused. Without modifying the original business system or requiring underlying data interfaces, it achieves efficient, accurate, and scalable intelligent question-answering capabilities, significantly improving the efficiency of enterprises in utilizing unstructured data assets.

[0071] This invention specifically addresses the following three key technical problems:

[0072] 1. Breaking free from the strong dependence on structured data sources, this invention enables direct querying of purely front-end visualized data. Addressing real-world scenarios where many key business metrics are only displayed through web page charts and lack open APIs, this invention utilizes a combined RPA+VLM solution to enable the system to "understand" page content, supporting natural language queries without rebuilding data pipelines or intruding into existing IT systems.

[0073] 2. Construct a sustainable knowledge management system for dynamic pages, supporting complex queries across time and multiple rounds. Addressing the typical characteristics of "unchanging page structure and daily data updates," this invention transforms visual pages into indexable, comparable, and traceable knowledge units through a combination of timed snapshots, manual summarization, and versioned storage mechanisms, effectively supporting advanced interactive needs requiring temporal context.

[0074] 3. This invention achieves a two-stage collaborative question answering process of "semantic retrieval—visual focusing," balancing efficiency, accuracy, and scalability. Unlike existing end-to-end full-graph reasoning models, this invention employs a phased processing architecture: first, lightweight text retrieval narrows the candidate pool, then VLM is invoked for refined analysis of highly relevant snapshots. This design significantly reduces resource consumption, improves response speed, and avoids interference from irrelevant areas, ensuring system stability and answer reliability in multi-page, high-concurrency scenarios.

[0075] The innovative aspects of this invention are as follows:

[0076] 1. This invention proposes an unstructured, visualized page knowledge modeling method based on "page snapshot + stable semantic summary". Unlike existing technologies that only perform isolated parsing or end-to-end reasoning on a single screenshot, this method overcomes the technical bottleneck of traditional unstructured data being "unreusable and unevolvable".

[0077] 2. This invention constructs a non-intrusive, visual page continuous acquisition and standardized storage mechanism based on RPA. This invention simulates manual operation through RPA, automatically accessing the target page without interface or code intrusion. After the page is fully rendered, it performs full-page or region-by-region screenshot acquisition, and then uniformly converts the acquired images into an encoded form for structured storage. This acquisition and storage mechanism represents a significant technological advancement in the field of unstructured data querying.

[0078] 3. This invention proposes a two-stage collaborative question-answering architecture of "semantic retrieval and localization + visual language model focused understanding". Unlike the existing solution's coarse processing method of directly inputting all page images into a multimodal model for reasoning, this two-stage architecture effectively reduces computational resource consumption, improves response speed, and significantly reduces irrelevant visual interference, achieving a balance between efficiency, accuracy, and system scalability.

[0079] 4. This invention establishes an end-to-end intelligent query closed-loop system architecture for visualized unstructured data. Without relying on structured data sources or modifying existing business systems, it enables natural language query capabilities for complex visualization pages, and can be widely applied to various practical scenarios such as enterprise business dashboards, operational reports, and management dashboards.

[0080] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. An unstructured data acquisition and query system that combines RPA and visual language models, characterized in that, include: The RPA timed acquisition module is used to automatically access the target visualization page according to a preset scheduling strategy using RPA technology. After the page is fully rendered, it performs screenshot acquisition and converts the screenshot image into a string with a unified encoding format, which is then persistently stored in the knowledge base. At the same time, it configures a high-level semantic summary for each target page or corresponding logical area, associates the semantic summary with the corresponding encoded image and acquisition timestamp, and writes it into the knowledge base in a structured form. The retrieval and recall module is used to receive natural language questions input by users and, based on the degree of matching between the question semantics and the semantic summary in the knowledge base, recall at least one most relevant page snapshot from the page records that meet the time constraints. The Understanding and Answer Display Interface module is used to input the encoded image of the recalled page snapshot and the user question into the Visual Language Model (VLM). The VLM performs fine-grained visual semantic understanding of the image and generates a natural language answer.

2. The unstructured data acquisition and query system combining RPA and visual language model according to claim 1, characterized in that, The RPA timed data collection module supports segmented screenshot collection of the target visualization page; the segmented screenshot collection specifically refers to: The target visualization page is divided into multiple logical blocks, and screenshot, encoding, and summary configuration are performed independently for each logical block to achieve fine-grained knowledge modeling.

3. The unstructured data acquisition and query system combining RPA and visual language model according to claim 1, characterized in that, In the RPA timed data collection module, the screenshot images are uniformly encoded as Base64 encoded strings; these strings are stored in the knowledge base in plain text form, along with semantic summaries, data collection timestamps, and page / region identifiers.

4. The unstructured data acquisition and query system combining RPA and visual language model according to claim 1, characterized in that, In the RPA scheduled data collection module, the semantic summary is pre-configured by business personnel or system administrators and describes the core business meaning, key indicators, analysis dimensions, time granularity and display logic of the page in natural language; the semantic summary does not change with the daily update of the underlying data of the page and supports one-time configuration.

5. The unstructured data acquisition and query system combining RPA and visual language model according to claim 3, characterized in that, The retrieval and recall module specifically includes the following process: Receiving a natural language question input by a user and mapping the natural language question to a set of human-generated summary texts corresponding to page records in a knowledge base that satisfy a temporal requirement of the natural language question performing semantic similarity matching using a contextual embedding model converting the summary texts to high-dimensional dense semantic vectors : ； ； By rearranging the model Computing the similarity between the problem vector and each summary vector , as the relevance score : ； Finally, the candidate page set is reordered based on the reordering score, and the top K most relevant page records are selected as the recall results for subsequent use by the visual language model.

6. The unstructured data acquisition and query system combining RPA and visual language model according to claim 5, characterized in that, The understanding and answer display interface module specifically includes the following processes: The Base64 encoded image of the recalled page snapshot is combined with the user's original question text to construct a multimodal input. At the same time, a visual language model is called to identify the chart type, coordinate axis, legend and key data points of the encoded image. Combined with the semantics of the user's question, cross-modal reasoning is completed, and finally the answer in structured or natural language format is output.

7. The unstructured data acquisition and query system combining RPA and visual language model according to claim 1, characterized in that, The unstructured data acquisition and query system that combines RPA and visual language models supports multi-turn dialogue context management and historical data backtracking query. By reusing historical snapshots and summary records, it enables data comparison and trend analysis at different points in time on the same page.

8. The unstructured data acquisition and query system combining RPA and visual language model according to claim 1, characterized in that, The target visualization page refers to a front-end display interface with a fixed layout, dynamic data content, and no open data interfaces.

Images