Screenshot content extraction method, system and device based on multimodal large model

Abstract

The present invention provides a screenshot content extraction method based on a multimodal large model, which is used to extract OTA hotel structured information, including the following steps: obtaining a screenshot image with hotel information; performing text recognition on the image through an optical character recognition model to obtain a text information box set; performing text area positioning on the image through an image positioning model, and extracting the text content of the text area to obtain a text content set; inputting the text content set into a text large model, extracting structured information through preset prompt words, and obtaining first structured information; inputting the image into a visual language model, extracting structured information through prompt words, and obtaining second structured information; comparing and fusing the first structured information and the second structured information to obtain hotel structured information. The present invention can understand the image content more comprehensively and accurately, significantly improve the accuracy of hotel structured information extraction, and reduce the error rate of information extraction.

Description

Background Art

[0002] In the OTA (Online Travel Agency) industry, accurate extraction and rapid updating of hotel information are of great significance to the operation of the platform. However, the existing technology has significant deficiencies in dealing with unstructured hotel screenshot content. The hotel information contained in these screenshot images is usually unstructured and difficult to be directly used for data analysis, information retrieval or automated processing, which brings inconvenience to users' information management and use. Efficiently and accurately extracting structured information from these screenshot images has become a technical problem that needs to be solved urgently. Traditional image information extraction methods have many limitations. Some methods rely on manual annotation or predefined templates for matching, which is inefficient, costly, and difficult to adapt to the frequent changes in the interface and information layout of the OTA platform. Although optical character recognition (OCR) technology can convert text in an image into electronic text, traditional OCR technology usually only focuses on the recognition of text content, ignoring the contextual information such as the position and layout of the text in the image, resulting in the lack of structured information extracted and difficult to be directly used for subsequent processing. In addition, problems such as complex background, image noise or low resolution that may exist in the screenshot image will also affect the recognition accuracy of traditional OCR technology. More importantly, OTA hotel information screenshots contain rich visual and textual information. Traditional methods usually only focus on information of a single modality, ignoring the correlation and complementarity between the two modal information, resulting in insufficient information utilization and incomplete or inaccurate extracted structured information. At the same time, traditional methods lack targeted structured information extraction methods, and it is difficult to effectively extract structured data that meets specific needs from the identified text, such as hotel name, price, address, rating, etc., which cannot meet the needs of practical applications. Therefore, there is an urgent need for a method that can effectively integrate multimodal information and extract structured information in a targeted manner to improve the efficiency and accuracy of extracting structured information from OTA hotel information screenshots.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] In view of the problems in the prior art, the purpose of the present invention is to provide a screenshot content extraction method, system and device based on a multimodal large model, which is used to extract structured information from hotel screenshots on the OTA platform, so as to solve the problems existing in the prior art of low efficiency of manual screening, poor robustness of automated methods, and difficulty in handling complex formats and diversified scenarios.

[0005] The embodiment of the present invention provides a screenshot content extraction method based on a multimodal large model, which is used to extract OTA hotel structured information, comprising the following steps:

[0006] Get a screenshot image with hotel information;

[0007] Perform text recognition on the screenshot image through an optical character recognition model to obtain a set of text information boxes containing location information;

[0008] The text area of ​​the screenshot image is located by using the image positioning model, and the text content of the text area is extracted to obtain a text content set containing position information;

[0009] Inputting a text content set containing position information into a large text model, extracting structured information using preset prompt words, and obtaining first structured information;

[0010] Inputting the screenshot image into the visual language model, extracting structured information through preset prompt words, and obtaining second structured information;

[0011] The first structured information and the second structured information are compared and integrated to obtain hotel structured information.

[0012] In some embodiments, performing text recognition on the screenshot image by using an optical character recognition model includes the following steps:

[0013] Preprocess the screenshot images, including image denoising, image enhancement and image binarization;

[0014] Use deep learning models to detect and recognize text on preprocessed images;

[0015] Output a text information box containing the bounding box of each recognized text area, the text information box contains the recognized text content and the position information of the bounding box;

[0016] Post-process the recognition results, including spelling correction, layout analysis and special character processing.

[0017] In some embodiments, the image localization model uses the yolov5 target detection model to detect the text area in the image and output the bounding box of the text area.

[0018] In some embodiments, the text model uses a pre-trained language model based on the Transformer architecture.

[0019] In some embodiments, the preset prompt words include keywords for specifying extraction fields, instructions for limiting output formats, and a small number of examples.

[0020] In some embodiments, the visual language model is trained by the following steps:

[0021] Build a training dataset containing screenshot images with hotel information and corresponding structured information;

[0022] The visual language model is fine-tuned using the training dataset to optimize the model parameters. The training goal is to minimize the difference between the predicted structured information and the actual structured information.

[0023] In some embodiments, comparing and fusing the first structured information and the second structured information comprises the following steps:

[0024] Comparing values ​​and confidence levels of the same fields in the first structured information and the second structured information;

[0025] If the values ​​of the same field are consistent, then this value is used as the final result, and the average or maximum value of the two confidence levels is taken as the final confidence level;

[0026] If the values ​​of the same field are inconsistent, they are selected or merged according to preset rules or confidence levels. The preset rules include giving priority to values ​​with higher confidence levels, or making weighted selections based on the importance of the field.

[0027] In some embodiments, a parallel reasoning framework is used to accelerate reasoning of large text models and visual language models.

[0028] An embodiment of the present invention also provides a screenshot content extraction system based on a multimodal large model, which is used to implement the above-mentioned screenshot content extraction method based on a multimodal large model.

[0029] The embodiment of the present invention further provides a screenshot content extraction device based on a multimodal large model, comprising:

[0030] processor;

[0031] a memory in which executable instructions for the processor are stored;

[0032] Among them, the processor is configured to perform the steps of the above-mentioned screenshot content extraction method based on the multimodal large model by executing executable instructions.

[0033] An embodiment of the present invention also provides a computer-readable storage medium for storing a program, which implements the steps of a screenshot content extraction method based on a multimodal large model when the program is executed by a processor.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

[0035] The screenshot content extraction method based on the multimodal large model of the present invention has the following beneficial effects: by fusing multimodal information such as optical character recognition, image positioning, text large model and visual language model, the present invention can understand the image content more comprehensively and accurately, significantly improve the accuracy of hotel structured information extraction, and reduce the error rate of information extraction. The present invention does not require manual annotation or template matching, reduces manual intervention and maintenance costs, and uses deep learning models for rapid reasoning, greatly improving the efficiency of information extraction and shortening processing time. The present invention utilizes the powerful feature extraction and generalization capabilities of deep learning models, as well as the advantages of multimodal information fusion, to enhance the adaptability to various complex images, improve the information extraction performance under different image quality conditions, and has stronger robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0037] Figure 1 is a flowchart of a screenshot content extraction method based on a multimodal large model according to an embodiment of the present invention;

[0038] Figure 2 is a flow chart of performing text recognition on a screenshot image by an optical character recognition model according to an embodiment of the present invention;

[0039] Figure 3 It is a structural schematic diagram of a screenshot content extraction device based on a multimodal large model according to an embodiment of the present invention;

[0040] Figure 4 It is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0042] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps can be decomposed, while some steps can be combined or partially combined. Therefore, the actual execution order may change according to the actual situation. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0043] The present invention aims to comprehensively and accurately understand and convert unstructured screenshot data through multimodal information processing technology, and overcome the challenges faced by existing technologies in processing such data. Unstructured screenshot content usually contains a large amount of complex and irregularly distributed text and image information, and its format varies, which makes it difficult for traditional parsing methods to effectively process it. In order to solve this technical problem, the present invention innovatively introduces a multimodal processing method, takes text and images as input, and comprehensively utilizes the complementary characteristics of the two to achieve efficient extraction and fusion of information, thereby improving the accuracy and robustness of information extraction. Specifically, the present invention comprehensively uses multiple technologies such as optical character recognition (OCR), image positioning, text large model, and visual language model, and compares and fuses the extracted structured information to form a complete technical solution. First, OCR technology is used to extract original text information from the screenshot image, laying the foundation for subsequent structured information extraction. OCR technology converts the text in the image into an electronic text that can be processed by a computer through steps such as image preprocessing (such as image noise reduction, binarization, etc.), text detection (precisely locating the text area in the image), character recognition (converting the image in the text area into character encoding), and post-processing (such as spelling correction, layout analysis, etc.). At the same time, combined with image positioning technology, the text area in the screenshot image is accurately located, and accurate location information is provided for the subsequent extraction of the text content of the corresponding area from the OCR result, thereby realizing the effective association between the text content and the image area. On this basis, the present invention introduces a pre-trained language model based on the Transformer architecture, that is, a large text model, which is used to perform deep semantic understanding of the extracted text content and extract structured information in combination with preset prompt words. This type of model is pre-trained on a large-scale text corpus to acquire rich language knowledge and contextual information, and can more effectively understand the semantics of the text, thereby improving the accuracy of structured information extraction. In addition, the present invention also adopts a visual language model to directly process the screenshot image, and extracts structured information in combination with preset prompt words, thereby making full use of the visual information in the image to assist in the understanding and extraction of text information. The visual language model integrates image features and text features and uses a deep learning model for joint learning to achieve effective association between image content and its related text description, thereby introducing visual information in the process of structured information extraction to improve the comprehensiveness and accuracy of information extraction. The present invention inputs the text information obtained by OCR and image positioning into the text large model, and simultaneously inputs the screenshot image into the visual language model to extract structured information separately. Finally, the present invention compares and integrates the first structured information and the second structured information output by the text large model and the visual language model to obtain the final structured information.The comparison and fusion process includes comparing the values ​​and confidences of the same field, and selecting or fusion according to preset rules (such as giving priority to values ​​with higher confidence, or weighted selection according to the importance of the field, etc.), so as to make full use of the advantages of multimodal information and overcome the limitations that may exist in single modal information, such as text information may be ambiguous or misidentified, and visual information may be affected by image quality or occlusion. Through the organic combination and collaborative work of the above technical solutions, the present invention effectively solves the technical problem of efficiently and accurately extracting structured information from screenshot images such as OTA hotel information. Compared with traditional methods, it can process complex images more effectively, understand image and text content more accurately, and extract the required structured information more flexibly, thereby significantly improving the performance of structured information extraction.

[0044] like Figure 1 As shown, an embodiment of the present invention provides a screenshot content extraction method based on a multimodal large model, which is used to extract OTA hotel structured information, and aims to solve the accuracy and efficiency problems of hotel screenshot content parsing on the OTA platform in an automated and intelligent manner, including the following steps:

[0045] S100, obtaining a screenshot image with hotel information;

[0046] The embodiment of the present invention involves obtaining a screenshot image containing hotel information from an OTA platform or other relevant data sources, and obtaining an image file containing hotel information on the OTA platform through various methods (such as manual screenshots by users, automatic screenshots by applications, etc.). These screenshot images may come from platforms such as websites and mobile applications, and their content generally includes hotel name, type, address, room information, facilities, ratings, etc. The screenshot image can be in various common image formats, such as JPEG, PNG, BMP, etc.

[0047] S200, performing text recognition on the screenshot image by using an optical character recognition model to obtain a set of text information boxes containing position information;

[0048] The optical character recognition model refers to the use of optical character recognition (OCR) technology to convert text in an image into computer-editable electronic text. In an embodiment of the present invention, the optical character recognition model can be various known OCR models, such as Tesseract, PaddleOCR, etc., or it can be an OCR model based on deep learning, such as CRNN, Transformer-based OCR, etc. The text information box refers to a rectangular box containing the recognized text content and the position information of the text content in the image. The position information can be the coordinates of the bounding box (such as the upper left and lower right corner coordinates), or it can be the center point coordinates and width and height information. This step aims to extract text information from the screenshot image and record its position in the image to provide text and position information for subsequent information extraction.

[0049] S300, locating the text area of ​​the screenshot image by using an image positioning model, and extracting the text content of the text area to obtain a text content set containing position information;

[0050] In the embodiment of the present invention, the image localization model can be based on traditional image processing methods (such as edge detection, connected domain analysis, etc.), or it can be a target detection model based on deep learning, such as YOLO, Faster R-CNN, etc. The image localization model is intended to locate the text area more accurately and filter out interference information in non-text areas in the image.

[0051] Furthermore, the image localization model can perform validity filtering on the text information recognized by OCR according to pre-set rules or the model's own judgment, thereby improving the efficiency and accuracy of subsequent structured information extraction. The image localization model can perform validity filtering in the following ways:

[0052] Based on the position filtering, the image positioning model can filter according to the absolute or relative position of the text area in the image. For example, the position filtering rule is preset. If the coordinates of the center point of the detected text area exceed the preset threshold range of the center area of ​​the image (for example, 80% of the center area), or the distance between the coordinates of any vertex of the text area and the edge of the image is less than the preset threshold (for example, 5% of the image width or height), it is considered that the information in the area may be an advertisement, watermark, navigation bar or other irrelevant information, and it is filtered out of the positioning results. This method can be implemented by comparing the coordinate values ​​of the text area boundary box with the size parameters of the image.

[0053] Based on the filtering of area or aspect ratio, the image positioning model can filter according to the area or aspect ratio of the text area. For example, if the area of ​​the detected text area is less than a preset threshold (such as 0.1% of the image area), or the aspect ratio exceeds the preset range (such as the aspect ratio is less than 0.5 or greater than 5), it is considered that the information in the area may be noise, misidentified characters or other irrelevant information, and it is filtered. This method can be implemented by calculating the area and aspect ratio of the text area boundary box.

[0054] Based on the model confidence filtering, the image localization model usually outputs a confidence score while outputting the text area location information, indicating the degree of certainty of the model that the area contains text. You can set a confidence threshold, such as 0.8. If the confidence score of a text area is lower than the threshold, the localization result of the area is considered unreliable and filtered.

[0055] Based on the filtering of context information, combined with the OCR results, the image positioning model can filter according to the context information of the text area. For example, if the OCR recognition results in a certain text area contain specific keywords or character combinations (such as "advertisement", "promotion", "buy now", etc.), the information in this area is considered to be advertising information and will be filtered. For another example, if the text layout or font in a certain area is obviously inconsistent with the layout or font commonly used on the OTA hotel information page, such as the font is too small, the layout is too dense or sparse, then the font size, line spacing, character spacing and other features can be combined for judgment and filtering.

[0056] Through the precise positioning of the text area by the image positioning model and the above-mentioned validity filtering, the amount of data for subsequent processing can be effectively reduced, and the interference of irrelevant information on the structured information extraction process can be avoided, thereby improving the efficiency and accuracy of information extraction. Specifically, by filtering out advertising information at the edge of the image, small-area noise areas, recognition results with low confidence, and areas where the layout or content is obviously inconsistent, the amount of data input to the subsequent large text model and visual language model can be significantly reduced, the computational burden of the model can be reduced, and the accuracy of the model output results can be improved. The above filtering methods can be used alone or in combination, for example, first performing location-based filtering and then performing model confidence-based filtering to achieve better filtering effects. In addition, the setting of the above thresholds can be adjusted and optimized according to specific application scenarios and data sets. The above embodiments are only exemplary and do not constitute a limitation of the present invention. Without departing from the concept of the present invention, those skilled in the art may also adopt other suitable filtering rules and methods.

[0057] S400, inputting a text content set containing position information into a large text model, extracting structured information through preset prompt words, and obtaining first structured information;

[0058] The text big model refers to a pre-trained language model based on the Transformer architecture, such as BERT, RoBERTa, GPT series, LLaMA, etc. These models are pre-trained on a large-scale text corpus, learn rich language knowledge and contextual information, can better understand the semantics of the text, and are applied to various downstream tasks, such as text classification, named entity recognition, relationship extraction, etc. In an embodiment of the present invention, the text big model is used to perform deep semantic understanding of the extracted text content, and extract structured information in combination with the "preset prompt word (Prompt)". "Preset prompt word" refers to an instruction or template for guiding the text big model to extract structured information, such as "Please extract the hotel name, price and address from the following text: [text content]". Through carefully designed prompt words, the text big model can be guided to extract the required structured information from the input text content, such as hotel name, price, address, rating, facilities, etc., and output in a predetermined format (such as JSON, XML, etc.).

[0059] S500, inputting the screenshot image into a visual language model, extracting structured information through preset prompt words, and obtaining second structured information;

[0060] A visual language model refers to a model that can understand the relationship between images and texts, and realize cross-modal information fusion and understanding. Common visual language models include CLIP, BLIP, ALIGN, etc. These models usually learn the correspondence between images and texts through methods such as contrastive learning or generative models, so that they can understand the content of the image and link it with the relevant text description. In an embodiment of the present invention, the visual language model directly processes the screenshot image, and extracts structured information in combination with preset prompt words, thereby using the visual information in the image (such as the hotel's logo, picture layout, hotel pictures, etc.) to assist in the understanding and extraction of text information and improve the accuracy of information extraction.

[0061] S600: Compare and merge the first structured information and the second structured information to obtain hotel structured information.

[0062] Compare the structured information output by the large text model and the visual language model, for example, compare whether the values ​​of the same field are consistent, and fuse them according to the preset rules. For example, if the hotel names extracted by the two models are consistent, the name is used as the final result; if the prices extracted by the two models are inconsistent, they can be selected or fused according to the confidence of the model or pre-set rules (such as taking the average, selecting the result with higher confidence, etc.). This step further improves the accuracy and robustness of structured information extraction by fusing multimodal information.

[0063] The present invention combines OCR, image positioning, text large model and visual language model, and utilizes the effectiveness filtering mechanism of the image positioning model to achieve efficient and accurate extraction of structured information from OTA hotel information screenshot images, thus solving the problems of low information extraction accuracy and poor anti-interference ability in traditional methods.

[0064] In some optional embodiments, such as Figure 2 The optical character recognition model is used to perform text recognition on the screenshot image, including the following steps:

[0065] S210, preprocessing the screenshot image, including image noise reduction, image enhancement and image binarization;

[0066] Image preprocessing is intended to improve the accuracy of subsequent text detection and recognition. Image denoising can use various image denoising algorithms, such as median filtering, Gaussian filtering, bilateral filtering, etc., to remove noise in the image, such as salt and pepper noise, Gaussian noise, etc. In this embodiment, median filtering is preferably used, which has a better effect on removing salt and pepper noise and can better retain the edge information of the image. The window size of the median filter can be adjusted according to the noise level of the image, such as 3x3, 5x5, etc. Image enhancement, using various image enhancement algorithms, such as histogram equalization, contrast stretching, gamma correction, etc., to enhance the contrast and clarity of the image, so that the text area is more prominent. In this embodiment, histogram equalization is preferably used, which can effectively improve the overall contrast of the image. Image binarization, using various image binarization algorithms, such as global threshold method, local threshold method (such as Otsu algorithm, adaptive threshold method, etc.), converting the image into a binary image, that is, setting the pixel value in the image to 0 or 255, so that the text area and the background are clearer. In this embodiment, the Otsu algorithm is preferably used, which can adaptively select the optimal threshold and has a good effect on images with uneven illumination. The above preprocessing steps can be adjusted and combined according to actual conditions, for example, only image denoising and binarization can be performed, or image enhancement can be performed first and then image denoising. These steps are non-limiting and can be selected according to specific application scenarios and image quality.

[0067] S220, using a deep learning model to perform text detection and recognition on the preprocessed image;

[0068] A deep learning model refers to a text detection and recognition model based on a deep neural network. In this embodiment, an end-to-end OCR model based on a convolutional neural network (CNN) and a recurrent neural network (RNN), such as CRNN, DBNet, PSENet, etc., and an OCR model based on a Transformer, such as TrOCR, etc., are preferably used. These models can perform text detection and recognition at the same time, thereby improving recognition efficiency. Text detection aims to locate text areas in an image, usually in the form of a bounding box; text recognition aims to convert the detected text area into a corresponding text string.

[0069] S230, outputting a text information box including a bounding box of each recognized text area, wherein the text information box includes recognized text content and position information of the bounding box;

[0070] The text information box contains the recognized text content and the position information of the bounding box. The position information of the bounding box is usually represented by a four-tuple (x1, y1, x2, y2), where (x1, y1) is the coordinate of the upper left corner of the bounding box, and (x2, y2) is the coordinate of the lower right corner of the bounding box. The output format of the text information box can be a structured format such as JSON, XML, etc., which is convenient for subsequent processing.

[0071] S240, post-processing the recognition results, including spelling correction, layout analysis and special character processing.

[0072] Spelling correction can use various spelling correction algorithms, such as dictionary-based correction, statistics-based correction, deep learning-based correction, etc., to correct spelling errors in the recognized text. Layout analysis refers to the layout analysis of the recognized text, such as determining the typesetting direction and paragraph structure of the text to improve the readability of the text. Special character processing refers to the processing of recognized special characters, such as removing invisible characters, converting special symbols, etc., to ensure the standardization of the text. The above post-processing steps are also non-limiting and can be selected and combined according to actual conditions. For example, if the recognition accuracy requirement is not high, spelling correction can be omitted.

[0073] Through the steps of the embodiment of the present invention, the optical character recognition model can effectively extract text information from the screenshot image and output it in the form of a text information box, providing a reliable data basis for subsequent structured information extraction. The accuracy of text detection and recognition is improved through image preprocessing; the recognition efficiency and accuracy are improved by adopting advanced deep learning models; and the quality of recognition results is further improved through post-processing. It can effectively solve the accuracy and robustness problems of text recognition in screenshot images and provide high-quality text input for subsequent steps.

[0074] In some optional embodiments, the image positioning model uses the yolov5 target detection model to detect the text area in the image and output the bounding box of the text area. In this embodiment, the yolov5 model performs validity filtering in the following ways: location-based filtering, filtering according to the coordinate information of the bounding box. For example, if the coordinates of the center point of the bounding box exceed 80% of the center area of ​​the image, or the distance from the coordinates of any vertex of the bounding box to the edge of the image is less than 5% of the width or height of the image, it will be filtered. Based on the filtering of the area or aspect ratio, the area and aspect ratio of each bounding box are calculated. If the area is less than a preset threshold or the aspect ratio exceeds the preset range, it will be filtered. Based on the filtering of the model confidence, the yolov5 model will output a confidence score while outputting the bounding box. Set a confidence threshold, such as 0.8. If the confidence score of a bounding box is lower than the threshold, it will be filtered.

[0075] By using the YOLOv5 model for text area detection and combining it with the above-mentioned validity filtering method, the text area in the image can be efficiently and accurately located, and the interference of irrelevant information and noise can be removed, providing high-quality input for subsequent structured information extraction. This embodiment has the advantages of fast speed, high accuracy, and strong robustness, and can effectively solve the problem of text area detection in OTA hotel information screenshot images. It should be pointed out that this embodiment is not restrictive, and other target detection models, such as Faster R-CNN, Mask R-CNN, etc., or other text detection models, such as TextBoxes++, CRAFT, etc., can also be used.

[0076] In some optional embodiments, the text large model adopts a pre-trained language model based on the Transformer architecture. In the embodiment of the present invention, the text large model refers to a large-scale pre-trained language model that can understand and generate natural language text. These models are usually based on the Transformer architecture. By pre-training on a large-scale text corpus, rich language knowledge and contextual information are learned, and the semantics of the text can be better understood, and applied to various downstream tasks, such as text classification, named entity recognition, relationship extraction, text generation, etc. In this embodiment, the text large model can effectively extract structured information from unstructured text by understanding the input text content and prompt words, and combining the language knowledge learned by itself. By using a pre-trained language model based on the Transformer architecture, the language knowledge learned in the large-scale corpus can be fully utilized to improve the accuracy and robustness of structured information extraction. It solves the problem that the traditional method cannot effectively process complex text information, and provides high-quality text information for subsequent multimodal information fusion. It should be pointed out that this embodiment is not restrictive, and other text processing models based on deep learning may also be used, such as CNN, RNN, etc., but the pre-trained language model based on Transformer can usually achieve better results.

[0077] In some optional embodiments, the preset prompt words include keywords for specifying the extraction field, instructions for limiting the output format, and a small number of examples. In this embodiment, the preset prompt words consist of the following three parts:

[0078] Keywords used to specify extraction fields: This part of the prompt word explicitly specifies which information fields need to be extracted from the input text. These keywords are usually commonly used terms or concepts in the target field. For example, in the present invention, the fields that need to be extracted from the OTA hotel information may include: Hotel Name, Hotel Address, Hotel Price, Hotel Star Rating, User Rating / Score, Hotel Facilities / Amenities, Room Type, Check-in / Check-out Time. In the prompt word, clear keywords can be used to specify these fields, such as "Extract the following information: Hotel Name, Address, Price, and Rating".

[0079] Instructions for limiting output formats: These prompts are used to specify the format of LLMs output results for subsequent processing and use. Common output formats include:

[0080] JSON (JavaScript Object Notation): A lightweight data exchange format that is easy to parse and generate.

[0081] XML (Extensible Markup Language): A markup language used to describe structured data.

[0082] CSV (Comma-Separated Values): A simple text format for storing tabular data.

[0083] In the prompt, you can use explicit instructions to specify the output format, such as "output in JSON format", "output in XML format", "output in CSV format". For example, if you need to output in JSON format, you can add the following instruction in the prompt: "Please return the extracted information in JSON format, in the following format: {"hotel_name":"","address":"","price":"","rating":""}".

[0084] Few-shot Examples: To further improve the extraction accuracy of LLMs, you can include a few examples in the prompt. Each example contains an input text and the corresponding structured output. By providing examples, LLMs can better understand the requirements and goals of the task. For example:

[0085] Input text: Beijing Mouxiang Hotel, address: XX Road, Chaoyang District, Beijing, price: 1200 yuan per night, rating: 4.8 points. JSON output: {"hotel_name":"Beijing Mouxiang Hotel","address":"XX Road, Chaoyang District, Beijing","price":"1200 yuan","rating":"4.8 points"}

[0086] Input text: A certain hotel in Shanghai is located on the Bund, offering luxurious accommodations, with room rates starting from 2,000 yuan and a user rating of 4.9 points. JSON output: {"hotel_name":"Shanghai Hotel","address":"The Bund","price":"2,000 yuan","rating":"4.9 points"}

[0087] By using prompt words containing keywords, output format instructions and a small number of examples, we can effectively guide the large text model to extract structured information, improve the accuracy and efficiency of extraction, and facilitate subsequent data processing and use. This solves the problem of traditional methods requiring a large amount of labeled data for model training, reduces development costs, and improves the generalization ability of the model.

[0088] In some embodiments, the visual language model is trained by the following steps:

[0089] Construct a training dataset containing screenshot images with hotel information and corresponding structured information. The construction process includes:

[0090] Data collection: Collect screenshots containing hotel information from various OTA platforms. These images should cover different page types (such as hotel list page, detail page, search result page, etc.), different interface styles, and different hotel types. Data collection can be done by manual screenshots, crawlers, etc.

[0091] Data annotation: annotate each screenshot image collected, and the annotation content is the structured information corresponding to the image. The fields of the structured information should be consistent with the target fields extracted by the text model, such as hotel name, address, price, rating, etc. The annotation format can be JSON, XML, etc. The annotation tools can be LabelImg, LabelMe, etc. The accuracy and consistency of the annotation should be ensured as much as possible.

[0092] Data cleaning and preprocessing: Clean and preprocess the labeled data, including removing duplicate data, processing missing values, format conversion, etc. For example, price information can be uniformly converted to numeric types, and address information can be normalized.

[0093] Use the training dataset to fine-tune the visual language model and optimize the model parameters. The training goal is to minimize the difference between the predicted structured information and the actual structured information, including:

[0094] Select a suitable visual language model. In this embodiment, models such as CLIP, BLIP, and ALIGN can be selected. These models usually include an image encoder and a text encoder, and can map images and texts to the same vector space.

[0095] Fine-tune the selected visual language model using the training dataset. The goal of fine-tuning is to enable the model to predict the corresponding structured information based on the input screenshot image. The specific method of fine-tuning may vary depending on the model. For example, for the CLIP model, you can use the contrastive learning method for fine-tuning, that is, maximize the similarity between the image and the corresponding text description, and minimize the similarity between the image and the non-corresponding text description. For the BLIP model, you can use a generative method for fine-tuning, that is, given an image, let the model generate the corresponding text description, and use a loss function such as the cross entropy loss function to optimize the model parameters.

[0096] Use a suitable loss function to measure the difference between the structured information predicted by the model and the actual structured information. Common loss functions include: Cross entropy loss function: used for classification tasks, such as predicting the star rating of a hotel; Mean square error loss function, used for regression tasks, such as predicting the price or rating of a hotel; Edit distance, used to measure the difference between two strings, such as comparing the difference between the predicted hotel name and the actual hotel name; Optimizer, use a suitable optimizer to update the model parameters, such as Adam, SGD, etc.; Training parameters, set appropriate training parameters, such as learning rate, batch size, number of iterations, etc.

[0097] Evaluation indicators: Use appropriate evaluation indicators to measure the performance of the model. Common evaluation indicators include: Precision, the proportion of correctly predicted structured information to all predicted information; Recall, the proportion of correctly predicted structured information to all true information; F1-score, the harmonic mean of precision and recall; Accuracy: For each field, the proportion of samples whose predicted values ​​are completely consistent with the true values.

[0098] Through the above steps, a visual language model that can effectively extract structured information from screenshot images can be trained. By learning the association between images and texts, the model can use the visual information in the image to assist in the extraction of structured information, thereby improving the accuracy and robustness of the extraction. The embodiment of the present invention and the large text model together constitute the basis for multimodal information fusion, solving the problems of information loss and understanding bias caused by traditional methods that rely solely on text information for extraction. It should be pointed out that this embodiment is not restrictive, and other visual language models and training methods can be used, such as using different loss functions, optimizers and training parameters. The key is to build a high-quality training data set and select a suitable model and training method to achieve the best performance.

[0099] In some optional embodiments, the first structured information and the second structured information are compared and merged to combine the information of the two modalities and improve the accuracy and robustness of the final structured information, including the following steps:

[0100] Comparing values ​​and confidence levels of the same fields in the first structured information and the second structured information;

[0101] In the embodiment of the present invention, it is first necessary to determine which fields in the two structured information are the same. For example, if both structured information contain the fields of "hotel name", "hotel address" and "hotel price", these fields are the same fields that need to be compared. Then, the values ​​and confidences of these same fields are compared. The value refers to the specific content of the field, for example, the value of the hotel name may be "Beijing XX Hotel". The confidence refers to the degree of trustworthiness of the model for the value, usually expressed as a probability value or score.

[0102] If the values ​​of the same field are consistent, then this value is used as the final result, and the average or maximum value of the two confidence levels is taken as the final confidence level;

[0103] In an embodiment of the present invention, if the values ​​of the same field are consistent, it means that the two models have the same understanding of the field, and the value is used as the final result. For confidence, the following two methods can be used for processing: taking the average value, taking the average of the confidences of the two models as the final confidence. For example, the confidence of the text model for the hotel name is 0.9, and the confidence of the visual language model for the hotel name is 0.8, then the final confidence is (0.9+0.8) / 2=0.85. Take the maximum value and take the higher confidence of the two models as the final confidence. For example, in the above example, the final confidence is 0.9. Which method is used in the specific implementation process depends on the specific application scenario and requirements. Taking the average value can smooth the confidence and avoid misleading the high confidence of a single model; taking the maximum value is more inclined to believe in a more confident model.

[0104] If the values ​​of the same field are inconsistent, they are selected or merged according to preset rules or confidence levels. The preset rules include giving priority to values ​​with higher confidence levels, or making weighted selections based on the importance of the field.

[0105] In the embodiment of the present invention, if the values ​​of the same field are inconsistent, it indicates that the two models have different understandings of the field. In this case, selection or fusion needs to be performed according to preset rules or confidence levels. The preset rules include:

[0106] The value with higher confidence is preferred. Compare the confidence of the two models for the field and select the value with higher confidence as the final result. For example, the hotel name extracted by the text model is "Beijing XX Hotel" with a confidence of 0.7; the hotel name extracted by the visual language model is "Shanghai XX Hotel" with a confidence of 0.9, then "Shanghai XX Hotel" is selected as the final result.

[0107] Make weighted selections based on the importance of the fields. Different fields may have different importance in structured information. For example, the hotel name and address are usually more important than a specific facility in the hotel. Therefore, different weights can be assigned according to the importance of the fields, and then selections can be made based on the weighted confidence. For example, assuming that the weight of the hotel name is 0.6 and the weight of the hotel facilities is 0.4, if the two models have inconsistent values ​​for the hotel name and hotel facilities, you need to calculate the weighted confidence separately and select the value with the higher weighted confidence. The calculation method is as follows: weighted confidence = confidence * weight.

[0108] In addition to confidence, contextual information can also be used to make judgments. For example, if the address extracted by the text model is geographically close to the address extracted by the visual language model, it can be considered that the results of both models have a certain degree of credibility, and the two address information can be considered to be merged, such as merging into a more detailed address description.

[0109] Manual intervention: In some cases, if the model cannot make an effective judgment, or the accuracy of the results is very high, manual intervention can be introduced to manually judge and select the final value.

[0110] In some optional embodiments, a parallel reasoning framework is used to accelerate the reasoning of large text models and visual language models. Since large text models and visual language models usually have huge parameters and high computational complexity, optimization techniques need to be used in practical applications to improve reasoning efficiency. The parallel reasoning framework is one of the important optimization methods. It distributes the calculation tasks of the model to multiple computing devices (such as GPU, CPU) for parallel execution, thereby shortening the reasoning time. vLLM is an efficient reasoning and service framework designed specifically for large language models (LLM), aiming to improve the reasoning speed and accuracy by optimizing the model structure and algorithm. It is particularly good at processing the reasoning of autoregressive language models, such as the GPT series models. Since large text models and visual language models often contain such model components, vLLM can effectively accelerate the reasoning process. By adopting the vLLM parallel reasoning framework, the reasoning process of large text models and visual language models can be effectively accelerated, and the efficiency and performance of the system can be improved. It makes it possible to efficiently use large-scale models for structured information extraction even when resources are limited. It should be noted that this embodiment is not restrictive and can be combined with other optimization techniques, such as model pruning, knowledge distillation, etc., to further improve the reasoning efficiency. At the same time, with the development of technology, more and more efficient reasoning frameworks will emerge, which can be selected and applied according to actual conditions.

[0111] An embodiment of the present invention also provides a screenshot content extraction system based on a multimodal large model, which is used to implement the steps of executing a screenshot content extraction method based on a multimodal large model.

[0112] In the screenshot content extraction method based on the multimodal large model of this embodiment, the serial number of each step is only used to distinguish the steps, and is not used as a limitation on the specific execution order of the steps. The execution order between the above steps can be adjusted and changed as needed.

[0113] In the screenshot content extraction system based on the multimodal large model of the present invention, the functions of each module can be implemented by adopting the specific implementation method of the screenshot content extraction method based on the multimodal large model as described above, which will not be described in detail here.

[0114] An embodiment of the present invention also provides a screenshot content extraction device based on a multimodal large model, comprising a processor; a memory, in which executable instructions of the processor are stored; wherein the processor is configured to execute the steps of a screenshot content extraction method based on a multimodal large model by executing the executable instructions.

[0115] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0116] Refer to the following Figure 3 The electronic device 600 according to this embodiment of the present invention is described. Figure 3 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0117] like Figure 3 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0118] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present invention described in the screenshot content extraction method based on the multimodal large model in the above description of this specification. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0119] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0120] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0121] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0122] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0123] In the screenshot content extraction device based on the multimodal large model, the steps of the screenshot content extraction method based on the multimodal large model are implemented when the program in the memory is executed by the processor. Therefore, the device can also obtain the technical effect of the above-mentioned screenshot content extraction method based on the multimodal large model.

[0124] The embodiment of the present invention also provides a computer-readable storage medium for storing a program, and the steps of the screenshot content extraction method based on the multimodal large model are implemented when the program is executed by the processor. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is executed on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned screenshot content extraction method based on the multimodal large model section of this specification.

[0125] refer to Figure 4 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be executed on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0126] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0127] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0128] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0129] The steps of the screenshot content extraction method based on the multimodal large model are implemented when the program in the computer storage medium is executed by the processor. Therefore, the computer storage medium can also obtain the technical effect of the above-mentioned screenshot content extraction method based on the multimodal large model.

[0130] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A screenshot content extraction method based on a multimodal large model for extracting OTA hotel structured information, characterized in that: The steps include: Get a screenshot image with hotel information; Performing text recognition on the screenshot image by using an optical character recognition model to obtain a set of text information boxes containing position information; Locating the text area of ​​the screenshot image by using an image positioning model, and extracting the text content of the text area to obtain a text content set containing position information; Inputting the text content set containing the position information into the text macro model, extracting structured information through preset prompt words, and obtaining first structured information; Inputting the screenshot image into a visual language model, extracting structured information through preset prompt words, and obtaining second structured information; The first structured information and the second structured information are compared and integrated to obtain hotel structured information.

2. The screenshot content extraction method based on a multimodal large model according to claim 1 is characterized in that: The step of performing text recognition on the screenshot image by using an optical character recognition model comprises the following steps: Preprocessing the screenshot image, including image noise reduction, image enhancement and image binarization; Use deep learning models to detect and recognize text on preprocessed images; Outputting a text information box containing a bounding box of each recognized text area, wherein the text information box contains recognized text content and position information of the bounding box; Post-process the recognition results, including spelling correction, layout analysis and special character processing.

3. The screenshot content extraction method based on a multimodal large model according to claim 1 is characterized in that: The image localization model adopts the yolov5 target detection model, which is used to detect the text area in the image and output the bounding box of the text area.

4. The screenshot content extraction method based on a multimodal large model according to claim 1, characterized in that: The text model uses a pre-trained language model based on the Transformer architecture.

5. The screenshot content extraction method based on a multimodal large model according to claim 1, characterized in that: The preset prompt words include keywords for specifying extraction fields, instructions for limiting output formats, and a small number of examples.

6. The screenshot content extraction method based on a multimodal large model according to claim 1 is characterized in that: The visual language model is trained by the following steps: Build a training dataset containing screenshot images with hotel information and corresponding structured information; The visual language model is fine-tuned using the training data set to optimize model parameters, with the training goal being to minimize the difference between the predicted structured information and the actual structured information.

7. The screenshot content extraction method based on a multimodal large model according to claim 1, characterized in that: Comparing and fusing the first structured information and the second structured information comprises the following steps: Comparing values ​​and confidence levels of the same fields in the first structured information and the second structured information; If the values ​​of the same field are consistent, then this value is used as the final result, and the average or maximum value of the two confidence levels is taken as the final confidence level; If the values ​​of the same field are inconsistent, they are selected or merged according to preset rules or confidence levels. The preset rules include giving priority to values ​​with higher confidence levels, or performing weighted selection according to the importance of the field.

8. The screenshot content extraction method based on a multimodal large model according to claim 1, characterized in that: A parallel reasoning framework is used to accelerate the reasoning of the large text model and the visual language model.

9. A screenshot content extraction system based on a multimodal large model, characterized in that: Used to implement the screenshot content extraction method based on a multimodal large model as described in any one of claims 1 to 8.

10. A screenshot content extraction device based on a multimodal large model, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to execute the steps of the screenshot content extraction method based on a multimodal large model described in any one of claims 1 to 8 by executing the executable instructions.

Images