Web page automatic screenshot and anomaly recognition system, method and equipment based on vision-language multi-mode large model and storage medium

Abstract

The invention discloses a Web page automatic screenshot and anomaly recognition system, method and device based on a vision-language multi-mode large model and a storage medium, and the system comprises a self-adaptive automatic image collection module, a page automatic screenshot module, a picture image preprocessing module and an image anomaly recognition module based on the multi-mode large model. Automatic execution of page anomaly detection is driven through web automation and a multi-modal large model technology, manual intervention is reduced, errors caused by human factors are avoided, comprehensive inspection can be rapidly and accurately performed on a system page, and the inspection efficiency and accuracy are remarkably improved; abnormal detection of typical scenes such as page data missing, data abnormity and curve jumping is also realized, manual page inspection work is replaced, and the manual workload and the operation and maintenance cost are reduced; the problems of low efficiency, large abnormal detection workload, high repeatability of the same work and the like in the process of dispatching the inspection of the three-area information management system by an automatic watch keeper are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, in particular to a web page automatic screenshot and abnormality identification system, method, device and storage medium based on visual-linguistic multimodal large model. BACKGROUND

[0002] In the evolution process of web information acquisition technology, the academic and engineering circles have always been faced with the core challenge of how to efficiently extract target information from massive Internet resources. The traditional manual collection mode is difficult to adapt to the exponential growth of network data size due to its high repeatability and low efficiency. Before the breakthrough of artificial intelligence technology, although the crawler technology based on HTML structure analysis realizes automation, it is plagued by the non-standardization of web page structure, the fragmentation of page layout, and the parsing difficulties of unstructured data such as charts and tables.

[0003] Image recognition technology, as a core pillar of artificial intelligence application, has been widely deployed in the power industry in recent years. From power line inspection, substation equipment monitoring, to electric meter box state recognition, wind turbine blade defect detection, computer vision systems are gradually replacing traditional manual inspection methods. The main challenge currently faced by the power system is that the detection object content is diverse, and the available training model is limited. The key features of different detection objects differ significantly, such as between different categories of data such as tables, column charts, line charts, and text, and within the same category of data. Existing algorithms often require customized feature engineering or model structure adjustment for specific objects, making it difficult to develop a general feature extractor, with high development cost and long development cycle. SUMMARY

[0004] The present application aims to provide a web page automatic screenshot and abnormality identification system based on visual-linguistic multimodal large model for detecting data missing, data abnormality, and curve jump in various web interface data; another object of the present application is to provide a web page automatic screenshot and abnormality identification method, device and storage medium based on visual-linguistic multimodal large model.

[0005] Technical solution: the web page automatic screenshot and abnormality identification system based on visual-linguistic multimodal large model, comprising

[0006] An adaptive automatic image acquisition module for automatically capturing various web interface generated during task execution;

[0007] A page automatic screenshot module for capturing target images in various web interfaces;

[0008] A picture image preprocessing module for labeling detection regions and task scene prompt words for target images;

[0009] The image anomaly recognition module based on the multi-modal large model is used for guiding the multi-modal large model to lock the analysis attention to the detection area, performing information recognition on the target image, and recognizing the abnormal scene according to the preset natural language prompt word.

[0010] Further, the adaptive automatic image acquisition module intelligently arranges the inspection process through process automation technology, and uses the selenium WEB automation framework to perform timed automatic screenshot saving of the webpage interface.

[0011] Further, the page automatic screenshot module edits the operation category and operation object of each step of operation, translates each step of operation category into a python script code, executes the python script code, and completes the automatic webpage interface screenshot work.

[0012] Further, the picture image preprocessing module performs image preprocessing on the target image according to the analysis role, the detection area frame selection, the type annotation and the prompt word.

[0013] Further, the analysis role includes a text analyst, a table analyst and a chart analyst.

[0014] Further, the detection area type is divided into a chart type, a table type and a text type, wherein the text type is divided into text missing and error display, the table type is divided into table content missing, table missing and data error in the table, and the chart type is divided into chart part display missing, single data missing and chart complete missing.

[0015] The webpage automatic screenshot and abnormal recognition method based on the visual-language multi-modal large model is realized through the webpage automatic screenshot and abnormal recognition system based on the visual-language multi-modal large model, and the method comprises

[0016] The adaptive automatic image acquisition module automatically captures the various webpage interfaces generated during task execution;

[0017] The page automatic screenshot module automatically acquires the target image according to the preset rule.

[0018] The picture image preprocessing module annotates the target image with a detection area and a task scene prompt word.

[0019] The image anomaly recognition module based on the multi-modal large model guides the multi-modal large model to lock the analysis attention to the detection area, performs information recognition on the target image, and recognizes the abnormal scene according to the preset natural language prompt word.

[0020] Further, the abnormal scene includes a data abnormal scene, an information missing scene and a curve jump scene.

[0021] The computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program.

[0022] The computer readable storage medium stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0023] Advantages: Compared with the prior art, the present application has the following advantages: 1. The present application drives page anomaly detection automation execution through web automation and multi-modal large model technology, reduces manual intervention, avoids errors caused by human factors, can quickly and accurately perform comprehensive inspection on system pages, and significantly improves inspection efficiency and accuracy; 2. The present application realizes anomaly detection of typical scenarios such as page data loss, data anomaly and curve jump, replaces manual page inspection and improves inspection efficiency and accuracy, reduces manual workload and operation and maintenance cost; 3. The present application solves the problems of low efficiency, large anomaly detection workload and high repetition of the same work of automatic duty officers in the process of inspecting the dispatching three-zone information management system. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The flowchart of the method of the present application. DETAILED DESCRIPTION

[0025] The Web page automatic screenshot and anomaly recognition system based on the visual-language multi-modal large model comprises:

[0026] (1) Page automatic screenshot module: responsible for image acquisition task, with adaptive and automatic characteristics, can automatically acquire images of various web interfaces according to preset rules, flexibly and accurately capture target images, and provide original image data for subsequent processing. The page automatic screenshot module adopts the logic of editing first, testing second and executing last. In the editing process, the operation category and operation object of each step can be edited. In the process of testing the results, each step of operation category is translated into executable python script code. Finally, when executing, the system can directly call the python script code to complete the automatic screenshot of the three-zone web interface.

[0027] (2) Picture image preprocessing module: on the basis of the image transmitted from the adaptive automatic image acquisition module, responsible for fine processing of the image, accurately marking the detection area, and clearly defining the specific detection task, providing clear guidance for subsequent anomaly identification. The picture image preprocessing module can perform image preprocessing on the page screenshot through role analysis, region framing, type labeling, and prompt word engineering. Role analysis includes text analyst, table analyst, and chart analyst. Region framing uses the horizontal and vertical coordinates of the upper left corner and the lower right corner as parameters. The picture image preprocessing module adopts a set of fine and scientific classification and processing mechanism. This module first implements accurate classification of the detection content, which is divided into three categories: charts, tables, and text. In these three categories, text is further divided into text missing and error display, table is further divided into table content missing, table missing, and data error in table, and chart is further divided into chart partial display missing, single data missing, and chart complete missing.

[0028] (3) Image anomaly identification module based on multi-modal large model: this module guides the large model to accurately lock the key area of the picture, and combines the pre-set natural language prompt (Prompt) to promote the model's deep understanding and logical reasoning of the image content, and accurately identify whether the key area meets the description requirements in the prompt. Finally, the analysis result is output in a standardized structured data.

[0029] During the execution of the detection task, the system will call the script generated by the adaptive automatic image acquisition module to automatically capture new images generated during task execution. These images intuitively reflect the state characteristics of the detection object at different times and under different conditions. The adaptive automatic image acquisition module can intelligently arrange the inspection process using process automation technology, and use selenium and other WEB automation frameworks to perform timed automatic screenshot saving of system pictures.

[0030] In terms of process, first, the adaptive automatic image acquisition module obtains the real-time screenshot of the Web running page; then, according to the labeling area and prompt word content preset by the picture image preprocessing module, the screenshot data is input into the image anomaly identification module of the multi-modal large model for in-depth analysis. Finally, the standardized and structured analysis result is output, providing accurate basis for subsequent decision-making.

[0031] The Web page automatic screenshot and anomaly identification method based on the visual-linguistic multi-modal large model of the present application is implemented by the above-mentioned Web page automatic screenshot and anomaly identification system based on the visual-linguistic multi-modal large model. The method comprises

[0032] The adaptive automatic image acquisition module automatically captures the various web interfaces generated during task execution;

[0033] The page automatic screenshot module acquires images of various web interfaces automatically according to preset rules, and captures target images;

[0034] The picture image preprocessing module labels a detection region and a task scene prompt word of the target image;

[0035] The image anomaly recognition module based on the multi-modal large model guides the multi-modal large model to lock the analysis attention on the detection region, performs information recognition on the target image, and recognizes an abnormal scene according to a preset natural language prompt word. The abnormal scene includes a data abnormal scene, an information missing scene and a curve jump scene.

[0036] The intelligent screenshot process arrangement system constructed based on a Web automation technology (such as a Selenium framework) supports full-process automatic operation on multiple data display pages. The system can be triggered regularly according to preset rules, and automatically completes tasks such as page element positioning, interactive clicking, high-definition screenshot and structured storage, thereby solving problems such as frequent repetitive operation and low work efficiency in traditional manual detection;

[0037] Picture image preprocessing technology: an intelligent detection task configuration framework for multi-modal large models is constructed, core function points such as rectangular region framing, detection frame type labeling and multi-modal prompt word engineering are integrated, accurate capture of key information of a Web page and standard definition of model input and output are realized, and the accuracy of model anomaly detection is improved.

[0038] Image anomaly recognition technology based on multi-modal large models: relying on the cross-modal correlation capability of multi-modal large models, the image region information (text blocks, table cells, curve segments, etc.) after preprocessing is detected for abnormalities in a natural language question and answer manner, complex detection problems such as consistency of model output text description and text or table values, matching degree of curve trend and threshold interval are captured, and accurate detection of multi-scene page abnormalities based on generalization means is realized.

[0039] The present application uses selenium and other automation frameworks to automatically save screenshots of system pictures. The page screenshots are preprocessed by region framing and type labeling, the model input and output are standardized, and the quality of model question and answer is improved. A multi-modal large model is used for general image detection, and the narrow applicability of traditional image recognition models is solved.

[0040] The computer device disclosed by the present application includes a memory, a processor and a computer program stored on the memory and executable on the processor. The processor executes the computer program to realize the steps of the above method.

[0041] The computer readable storage medium disclosed by the present application has a computer program stored thereon. The computer program is executed by the processor to realize the steps of the above method.

Claims

1. A web page automated screenshot and anomaly recognition system based on a visual-language multimodal large model, characterized in that, include An adaptive automated image acquisition module is used to automatically capture various web interfaces generated during task execution; An automated screenshot module for capturing target images across various web page interfaces; The image preprocessing module is used to annotate the detection area and task scene prompts in the target image; The image anomaly recognition module based on a multimodal large model is used to guide the multimodal large model to focus its analysis attention on the detection area, perform information recognition on the target image, and identify abnormal scenes based on preset natural language prompts.

2. The automated web page screenshotting and anomaly recognition system based on a visual-language multimodal large model according to claim 1, characterized in that, The adaptive automated image acquisition module intelligently orchestrates the inspection process through process automation technology and uses the Selenium WEB automation framework to automatically capture and save web page screenshots at regular intervals.

3. The automated web page screenshotting and anomaly recognition system based on a visual-language multimodal large model according to claim 1, characterized in that, The automated page screenshot module allows you to edit the operation category and object for each step, translate each operation category into Python script code, execute the Python script code, and complete the automated webpage screenshot task.

4. The automated web page screenshotting and anomaly recognition system based on a visual-language multimodal large model according to claim 1, characterized in that, The image preprocessing module performs image preprocessing on the target image based on the analysis role, detection area selection, type labeling, and prompt words.

5. The automated web page screenshotting and anomaly recognition system based on a visual-language multimodal large model according to claim 4, characterized in that, Analytical roles include text analysts, table analysts, and chart analysts.

6. The automated web page screenshotting and anomaly recognition system based on a visual-language multimodal large model according to claim 5, characterized in that, The detection area types are divided into chart, table, and text. Among them, text is further divided into missing text and error display; table is divided into missing table content, missing table, and incorrect data in the table; and chart is divided into partial missing chart display, missing single data, and missing chart.

7. A method for automated screenshotting and anomaly recognition of web pages based on a visual-linguistic multimodal large model, characterized in that, This is achieved through the automated web page screenshotting and anomaly recognition system based on a visual-language multimodal large model as described in any one of claims 1-6, the method comprising: The adaptive automated image acquisition module automatically captures various web interfaces generated during task execution; The automated page screenshot module automatically acquires images of various web page interfaces and captures target images according to preset rules; The image preprocessing module marks the detection area and task scene prompts in the target image; The image anomaly recognition module based on the multimodal large model guides the multimodal large model to focus its analysis attention on the detection area, perform information recognition on the target image, and identify abnormal scenes based on preset natural language prompts.

8. The method for automated screenshotting and anomaly recognition of web pages based on a visual-language multimodal large model according to claim 7, characterized in that, Abnormal scenarios include data anomaly scenarios, information missing scenarios, and curve jump scenarios.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 7 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 7 to 8.

Images