Picture violation detection method, device and system and medium

Through the multimodal feature extraction and chain layout strategy of multimodal large model, the problem of difficulty in integrating document picture layout information in the existing technology is solved, and the accuracy and interpretability of violation detection are improved.

CN119027956APending Publication Date: 2024-11-26PING AN TECH (SHENZHEN) CO LTD

Patent Information

Application Number
CN202411144590.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

When the existing violation detection model processes document pictures containing complex layout information, it is difficult to effectively integrate layout information, resulting in the impact of the accuracy of violation detection.

Method used

A multimodal large model is used to extract multimodal features of document pictures, text information and layout information through pre-trained encoder, and a large language model is used to extract content of chain layout strategy, and position the designated layout area for violation detection.

Benefits of technology

It improves the accuracy and interpretability of violation detection in document pictures, ensuring effective processing of complex layout information and accurate identification of violations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027956A_ABST
    Figure CN119027956A_ABST
Patent Text Reader

Abstract

The invention discloses a picture violation detection method, device and system and a medium, and the method comprises the steps: obtaining a to-be-detected document picture, extracting text information and layout information in the document picture, and inputting the text information and layout information into a trained multi-mode large model, feature extraction is carried out on the document pictures, the text information and the layout information through a pre-training encoder, multi-modal features are obtained, and the pre-training encoder is obtained through training of layout perception pre-training tasks of different levels; in response to the violation detection instruction, carrying out chained layout strategy content extraction on the multi-modal features and the violation detection instruction through a large language model in the multi-modal large model, and obtaining the content of a specified layout area in the document picture; and performing violation detection on the content of the specified layout area, and outputting a corresponding violation detection result. The multi-modal large model locates the related area in the document picture through the multi-modal feature, and the content is extracted in a concentrated manner for violation detection, so that the violation detection accuracy of the document picture is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a method, device, system and medium for detecting image violations. Background Art

[0002] In the financial industry, based on the regulations of financial regulatory agencies on financial service promotion, customer privacy protection, etc., customer service needs to strictly abide by relevant regulations when providing services. Therefore, financial companies need to monitor customer service’s violations to improve financial security and regulatory efficiency.

[0003] At present, based on artificial intelligence models, it is possible to monitor violations and automatically detect whether there are any violations in the communication content between customer service and customers. The process is usually as follows: collect normal and illegal chat sample data, label and preprocess them, build its artificial intelligence model through training models, collect chat sample data for testing, label and preprocess them, and pass the test samples through the artificial intelligence model to obtain the predicted violation category and predicted violation score of the test samples. Based on the test results, the artificial intelligence model for customer service violation detection in financial scenarios is judged to determine whether it is available. If it is available, it is deployed to the production environment to monitor customer service violations.

[0004] However, in the process of customer service chatting with customers, the actual situation is that text information and document images are often accompanied. For document images, they contain complex layout information of the document, and some violations are easily hidden in these layouts. The existing violation detection model is difficult to effectively integrate the layout information of the document when understanding the document, which makes it easy for omissions or errors to occur during violation detection, affecting the accuracy of violation detection. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide an image violation detection method, device, system and medium that can be applied to financial technology or other related fields, and its main purpose is to improve the accuracy of violation detection for document images.

[0006] The technical solution of the present invention is as follows:

[0007] A first aspect of the present invention provides a method for detecting image violations, comprising:

[0008] Obtaining a document image to be detected, and extracting text information and layout information from the document image;

[0009] Inputting the document image, text information and layout information into a trained multimodal large model, wherein the multimodal large model includes a pre-trained encoder and a large language model;

[0010] Extracting features of the document image, text information and layout information through the pre-trained encoder to obtain multimodal features, wherein the pre-trained encoder is trained through layout perception pre-training tasks at different levels;

[0011] In response to the violation detection instruction, extracting the content of the chain layout strategy from the multimodal features and the violation detection instruction through the large language model to obtain the content of the specified layout area in the document image;

[0012] Perform violation detection on the content of the designated layout area and output corresponding violation detection results.

[0013] In one embodiment, the extracting features of the document image, text information and layout information by the pre-trained encoder to obtain multimodal features includes:

[0014] Encoding the document image, text information and layout information by the pre-trained encoder to obtain visual features and text layout features;

[0015] The visual features and the text layout features are projected from the output space of the pre-trained encoder to the embedding space of the large language model.

[0016] In one embodiment, in response to the violation detection instruction, extracting the content of the chain layout strategy on the multimodal features and the violation detection instruction through the large language model to obtain the content of the specified layout area in the document image includes:

[0017] receiving a violation detection instruction, and inputting the multimodal features and the violation detection instruction into a trained large language model;

[0018] Analyzing the question in the violation detection instruction by using the large language model, and determining a designated layout area in the document image associated with the question according to the multimodal feature;

[0019] The designated layout area is centrally extracted for content according to the question to obtain the answer content of the question.

[0020] In one embodiment, the performing violation detection on the content of the designated layout area and outputting the corresponding violation detection result specifically includes:

[0021] The violation category and violation score of the content in the designated layout area are predicted, and a violation detection result is outputted indicating whether a preset violation type exists and a corresponding violation score.

[0022] In one embodiment, the training process of the multimodal large model includes:

[0023] Collecting training pictures and identifying text data and layout data in the training pictures;

[0024] Constructing input information of different levels according to the training images, text data and layout data, wherein the input information includes the overall document level, the layout area level and the paragraph sentence level;

[0025] The multimodal large model is subjected to layout-aware pre-training and supervised fine-tuning according to the input information to obtain a trained multimodal large model.

[0026] In one embodiment, performing layout-aware pre-training and supervised fine-tuning on the multimodal large model according to the input information to obtain a trained multimodal large model includes:

[0027] Inputting different levels of input information into the multimodal large model, and performing different levels of layout perception pre-training on the pre-trained encoder through corresponding pre-training task instructions;

[0028] Input information from all levels is fed into a large multimodal model, which is fine-tuned under a supervised chained layout strategy via instructions containing multiple layout cues.

[0029] In one embodiment, after collecting the training image and identifying the text data and layout data in the training image, the method further includes:

[0030] The training images, text data and layout data are preprocessed accordingly.

[0031] A second aspect of the present invention provides an image violation detection device, comprising:

[0032] A text layout extraction module is used to obtain a document image to be detected and extract text information and layout information from the document image;

[0033] An input module, used to input the document image, text information and layout information into a trained multimodal large model, wherein the multimodal large model includes a pre-trained encoder and a large language model;

[0034] A feature extraction module, used to extract features from the document image, text information and layout information through the pre-trained encoder to obtain multimodal features, wherein the pre-trained encoder is trained through layout perception pre-training tasks at different levels;

[0035] A content extraction module, configured to, in response to a violation detection instruction, extract content of a chain layout strategy from the multimodal features and the violation detection instruction through the large language model to obtain content of a specified layout area in the document image;

[0036] The violation detection module is used to perform violation detection on the content of the specified layout area and output corresponding violation detection results.

[0037] A third aspect of the present invention provides a system for detecting image violations, the system comprising at least one processor; and

[0038] a memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned image violation detection method.

[0040] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned image violation detection method.

[0041] Beneficial effects: The present invention discloses a method, device, system and medium for detecting image violations. Compared with the prior art, the embodiment of the present invention obtains a document image to be detected, extracts text information and layout information from the document image, and then inputs them into a trained multimodal large model. The pre-trained encoder therein performs feature extraction on the document image, text information and layout information to obtain multimodal features, and the pre-trained encoder is trained through layout perception pre-training tasks at different levels; in response to a violation detection instruction, the large language model in the multimodal large model performs content extraction of a chain layout strategy on the multimodal features and the violation detection instruction to obtain the content of a specified layout area in the document image; the content of the specified layout area is subjected to violation detection, and the corresponding violation detection result is output. The multimodal large model is enabled to locate relevant areas in the document image through multimodal features, and the content is extracted intensively for violation detection, thereby improving the accuracy of violation detection on the document image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0043] Figure 1 A flow chart of a method for detecting image violations provided by an embodiment of the present invention;

[0044] Figure 2 A flow chart of step S103 in the image violation detection method provided by an embodiment of the present invention;

[0045] Figure 3A flow chart of step S104 in the image violation detection method provided by an embodiment of the present invention;

[0046] Figure 4 A flow chart of a multimodal large model training process in the image violation detection method provided by an embodiment of the present invention;

[0047] Figure 5 A flowchart of step S403 in the image violation detection method provided by an embodiment of the present invention

[0048] Figure 6 A schematic diagram of the functional modules of the image violation detection device provided by an embodiment of the present invention;

[0049] Figure 7 A schematic diagram of the hardware structure of the image violation detection system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0051] In the financial industry, based on the regulations of financial regulatory agencies on financial service promotion, customer privacy protection, etc., customer service needs to strictly abide by relevant regulations when providing services. Therefore, financial companies need to monitor customer service’s violations to improve financial security and regulatory efficiency.

[0052] At present, it is possible to monitor violations based on artificial intelligence models and automatically detect whether there are any violations in the communication between customer service and customers. However, in the process of chatting between customer service and customers, the actual situation is that text information and document images are often accompanied. For document images, which contain complex layout information of the document, some violations are easy to be hidden in these layouts. The existing violation detection model is difficult to effectively integrate the layout information of the document when understanding the document, which makes it easy for omissions or errors to occur during violation detection, affecting the accuracy of violation detection.

[0053] In order to solve the above problems, the present invention proposes a method for detecting image violations. Figure 1 , Figure 1 A flowchart of an embodiment of the image violation detection method provided by the present invention. The system architecture of the image violation detection method provided by this embodiment may include a first terminal device, a second terminal device, a third terminal device, a network and a server. The network is used to provide a medium for a communication link between the first terminal device, the second terminal device, the third terminal device and the server. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0054] The user may use the first terminal device, the second terminal device, and the third terminal device to interact with the server through the network to receive or send messages, etc. Various communication client applications may be installed on the first terminal device, the second terminal device, and the third terminal device, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0055] The first terminal device, the second terminal device, and the third terminal device may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0056] The server may be a server that provides various services, such as a background management server that provides support for the content browsed by the user using the first terminal device, the second terminal device, and the third terminal device (for example only). The background management server may analyze and process the received data such as the user request, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0057] It should be noted that the image violation detection method provided in the embodiment of the present invention can generally be executed by the first terminal device, the second terminal device or the third terminal device. Correspondingly, the image violation detection device provided in the embodiment of the present invention can also be set in the first terminal device, the second terminal device or the third terminal device.

[0058] Alternatively, the image violation detection method provided by the embodiment of the present invention may also be generally executed by a server. Accordingly, the image violation detection device provided by the embodiment of the present invention may generally be arranged in a server. The image violation detection method provided by the embodiment of the present invention may also be executed by a server or a server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server. Accordingly, the image violation detection device provided by the embodiment of the present invention may also be arranged in a server or a server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server.

[0059] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0060] like Figure 1 As shown, the image violation detection method provided in this embodiment specifically includes the following steps:

[0061] S101, obtaining a document image to be detected, and extracting text information and layout information from the document image.

[0062] In this embodiment, the document image to be detected can be a chat image of the customer service during the service process. By collecting the chat image between the customer service and the customer, document understanding and violation detection are performed to ensure the compliance of the service process. The document image contains text and the corresponding document layout. When understanding the document, the text information and layout information in the document image can be identified by OCR (optical character recognition) and other methods, and then the position and structure of the text in the document can be determined. The text information is specifically the text content identified from the document image by OCR technology. This information usually includes the text string in the document, and does not involve the spatial position of the text or the relationship with other elements; the layout information describes the spatial arrangement of the text in the document, such as the position of the text line, paragraph spacing, table structure, etc. The specific layout information may include the position coordinates of the text box, the text direction, the boundaries of the columns and rows, etc.

[0063] S102: input the document image, text information and layout information into a trained multimodal large model, wherein the multimodal large model includes a pre-trained encoder and a large language model.

[0064] In this embodiment, the original document image and the extracted text information and layout information are input into a trained multimodal large model for document understanding. The multimodal large model includes a pre-trained encoder and a large language model. By using images, texts and related image layouts as input, the richness of the processed data types is improved compared to using only text input. In addition, the small-shot learning, contextual connection and mathematical deduction capabilities of the large language model are utilized to further improve the accuracy of document content extraction by combining multimodal information, thereby improving the accuracy of violation detection.

[0065] S103, extracting features from the document image, text information and layout information through the pre-trained encoder to obtain multimodal features, wherein the pre-trained encoder is trained through layout-aware pre-training tasks at different levels.

[0066] In this embodiment, the pre-trained encoder in the multimodal large model is trained through layout-aware pre-training tasks at different levels. Specifically, a document pre-training model is used as an encoder. The pre-trained encoder is pre-trained at different levels of the document (such as the overall document, regional level, and paragraph level) according to different levels of input information and corresponding task instructions, so that the model can better understand the global information of the document and carefully analyze local details. The pre-trained encoder extracts features from the input document image, text information, and layout information to obtain multimodal features. The extracted multimodal features are the basis for the model to understand the content and structure of the document. The pre-trained encoder obtained by learning the document layout from the global to the local can effectively and reliably extract multimodal feature representations from multimodal information when processing complex document structures in real application scenarios, thereby improving the model's ability to understand documents and ensuring the accuracy of violation detection.

[0067] In one embodiment, Figure 2 As shown, step S103 includes:

[0068] S201, encoding the document image, text information and layout information by the pre-trained encoder to obtain visual features and text layout features;

[0069] S202: Project the visual features and the text layout features, and project the visual features and the text layout features from the output space of the pre-trained encoder to the embedding space of the large language model.

[0070] In this embodiment, the multimodal large model receives input document images, text information and layout information, wherein the text information may be a text sequence, and the layout information may be a bounding box coordinate, etc. The document images, text information and layout information are encoded by the pre-trained encoder in the model, and the visual features and text layout features of the document are extracted, wherein the visual features are features directly extracted from the document images, mainly focusing on the visual content of the image, such as the visual representation of text, images, charts, etc., and may include, for example, the geometric shape, color, texture of the elements in the document image, and their positions in the image, etc.; the text layout features focus on the spatial arrangement and structure of the text in the document, such as the position of the text line, the layout of the paragraph, the row and column structure of the table, etc., and may include, for example, the bounding box coordinates of the text, the direction of the text, the relationship between text blocks, etc. Among the extracted features, the visual features help the model recognize and understand the elements in the image, and the text layout features help the model understand the arrangement of the text content and the logical structure of the document. The two features are combined to form a multimodal feature representation, so that the model can not only understand the visual information of the document, but also understand the text content and its organization in the document, thereby improving the accuracy and depth of document understanding.

[0071] Furthermore, the extracted visual features and text layout features are projected respectively. Specifically, two different multi-layer perceptrons can be used as a visual projector and a text and layout projector to project the visual features and text layout features from the output space of the pre-trained encoder to the embedding space of the large language model to adapt to the input format of the large language model. This allows the feature vector to be seamlessly input into the large language model, ensuring that the extracted multimodal features can be effectively processed by the large language model.

[0072] S103 . In response to the violation detection instruction, extract the content of the multimodal features and the violation detection instruction using the large language model using a chain layout strategy to obtain the content of the specified layout area in the document image.

[0073] In this embodiment, for the violation detection instruction input by the user, the instruction contains prompt words of corresponding layout information, such as "whether there is sensitive information leakage in the xx table of the document", "whether there is exaggerated propaganda in the xx paragraph of the document", etc., the large language model extracts the content of the chain layout strategy based on the received violation detection instruction and the extracted multimodal features, and then obtains the content of the specified layout area corresponding to the violation detection instruction in the document image. Compared with the traditional model that directly generates answers based on questions, the chain layout strategy is used in this embodiment to improve the model's understanding of questions and answer inference ability when extracting content from documents, so that the model can locate relevant areas in the document, focus attention and accurately infer answers, providing more accurate judgment content for the judgment of violation detection, thereby improving the accuracy and explainability of violation detection.

[0074] In one embodiment, Figure 3 As shown, step S104 includes:

[0075] S301, receiving a violation detection instruction, and inputting the multimodal features and the violation detection instruction into a trained large language model;

[0076] S302, analyzing the question in the violation detection instruction by using the large language model, and determining a designated layout area in the document image associated with the question according to the multimodal feature;

[0077] S303: extracting content from the designated layout area in a centralized manner according to the question, and obtaining answer content to the question.

[0078] In this embodiment, the chain layout strategy specifically answers questions through a step-by-step reasoning process, that is, each reasoning step should be based on the previous step, so that when the model combines layout hints to reason to extract the content of the corresponding area, it can not only provide the answer content, but also provide the process of how to find the answer content, which helps to improve the credibility and interpretability of the model.

[0079] Specifically, a violation detection instruction input by a user is received and the multimodal features and the violation detection instruction are input into a trained large language model. As mentioned above, the instruction contains prompt words for corresponding layout information, such as "whether there is sensitive information leakage in the xx table of the document", "whether there is exaggerated publicity in the xx paragraph of the document", etc. When the large language model extracts content using a chain layout strategy, it first performs a question analysis, that is, analyzes the question in the violation detection instruction, such as analyzing the type of question and key information to determine which parts of the document need to be paid attention to; then, it performs relevant area concentration, that is, according to the problem analysis results, combined with the multimodal features, it determines the designated layout area in the document image that is associated with the question, so that the model recognizes and concentrates on the area in the document that is related to the question, which helps the model ignore irrelevant information and improves the efficiency of document content extraction; finally, it performs answer formation, that is, according to the question, it performs concentrated content extraction on the designated layout area to obtain the answer content of the question. Content extraction through the chain layout strategy provides an intermediate step for answering questions, which is beneficial to improving the performance of the model in document understanding. When detecting violations, the model can be guided by the layout prompt words in the violation detection instructions to focus on extracting content in the relevant layout areas for violation detection without the need to detect other irrelevant areas. This improves the document understanding performance while also helping to improve the efficiency of violation detection.

[0080] S105 , performing violation detection on the content of the designated layout area, and outputting corresponding violation detection results.

[0081] In this embodiment, violation detection is performed based on the extracted content of the designated layout area to obtain the corresponding violation detection result. Specifically, a classifier or rule engine can be trained separately to perform violation detection on the extracted content. Of course, the large language model can also be fine-tuned accordingly to integrate the corresponding violation detection logic so that it can identify the corresponding violation content. Specifically, when performing the reasoning of violation detection, the violation category and violation score of the content in the designated layout area are predicted, that is, by training a classification model or fine-tuning the large language model, the model can identify different types of violations (such as fraud, harassment, sensitive information leakage, etc.), based on the extracted content of the designated layout area, the classifier or large language model is used to perform classification prediction of violation detection on the content, according to the classification result, the detected violation type is output, and a score is assigned to the violation, which can be based on the confidence of the model, the severity of the violation or other related factors. The higher the violation score, the greater the possibility or more serious the violation.

[0082] This embodiment obtains a pre-trained encoder through different levels of layout-aware pre-training tasks, performs multimodal feature extraction on the document image, text information and layout information, and performs content extraction of a chain layout strategy through the multimodal features of a large language model and violation detection instructions, so that the multimodal large model can accurately locate relevant areas in the document image and concentrate on extracting content for violation detection, thereby effectively improving the accuracy of violation detection on document images.

[0083] In one embodiment, Figure 4 As shown, the training process of the multimodal large model includes the following steps:

[0084] S401, collecting training pictures and identifying text data and layout data in the training pictures;

[0085] S402, constructing input information of different levels according to the training image, text data and layout data, wherein the input information includes an overall document level, a layout area level and a paragraph sentence level;

[0086] S403: Perform layout-aware pre-training and supervised fine-tuning on the multimodal large model according to the input information to obtain a trained multimodal large model.

[0087] In this embodiment, in order to enable the multimodal large model to understand the global information of the document and carefully analyze the local details, and further improve the model's ability to utilize the document layout information, data for training is collected and constructed during model training, and the multimodal large model is trained through a training strategy adjusted by layout instructions. Specifically, training images are collected and the text data and layout data in the training images are identified. For example, a large number of document images in customer service history chat records are collected in the business system as training images, and the text data and layout data in the training images are obtained through automatic recognition, manual annotation, and other methods.

[0088] Preferably, after collecting the training data, the training images, text data and layout data can be preprocessed accordingly. For example, the preprocessing of text data includes cleaning the text, removing special characters, HTML tags, URL links and other non-text content, and retaining only the pure text part; converting traditional and simplified Chinese to unify the font, marking the paragraphs and sentence levels, etc.; preprocessing of training images includes zooming operations, adjusting contrast, etc.; preprocessing of layout data includes masking the text in the image, that is, blanking it, setting the required layout tags, such as tables, plate distribution (pie charts, bar charts), and recording relevant coordinate information, such as the location of the table, for subsequent model training.

[0089] The layout instruction adjustment training strategy adopted in this embodiment is divided into two stages, layout-aware pre-training and layout-aware supervised fine-tuning. In order to achieve the model's comprehensive understanding of documents of different levels during the training process, different levels of input information are constructed based on the collected training images, text data and layout data. The specific input information includes the overall document level, layout area level and paragraph sentence level, so that the multimodal large model can be pre-trained and supervised according to the input information of different levels, guiding the model to learn the structure and content of the document layout, improving the overall understanding ability of the model, and enabling the model to locate relevant areas in the document, focus attention and accurately infer answers, thereby improving the accuracy and explainability of violation quality inspection.

[0090] In one embodiment, Figure 5 As shown, step S403 includes:

[0091] S501, inputting input information of different levels into the multimodal large model, and performing layout perception pre-training of different levels on the pre-trained encoder through corresponding pre-training task instructions;

[0092] S502: Input information of all levels is input into the multimodal large model, and supervised fine-tuning of the large language model under the chain layout strategy is performed through instructions containing multiple layout hints.

[0093] In this embodiment, in the pre-training stage, three different levels of pre-training strategies are simultaneously applied to the multimodal large model, that is, three groups of input information at different levels, including the overall document level, the layout area level and the paragraph sentence level, are input into the multimodal large model, and the pre-training encoder in the model is pre-trained at different levels of layout perception through corresponding pre-training task instructions. For the overall document level, through the input of the entire picture level and the corresponding text content, under the pre-training task instructions with prompt words of layout information, the pre-training encoder is pre-trained for the document intensive description task and the text and layout reconstruction task, wherein the document intensive description task is used to train the encoder to generate a detailed description of the input document, and the text and layout reconstruction task is used to train the encoder to reconstruct the complete text and layout information of the document and output it in a specific format; At the layout area level, through the input of the picture level with layout information and the corresponding text content, under the pre-training task instruction with the prompt word with layout information, the pre-trained encoder is pre-trained for the document layout analysis task and the table understanding task. The document layout analysis task is used to train the encoder to recognize the layout types of different areas in the document, such as titles, lists, tables, etc., and the table understanding task is used to train the encoder to understand the row and column information and cell content of the table in the document; at the paragraph sentence level, through the picture input with the paragraph sentence level and the corresponding text content, under the pre-training task instruction with the prompt word with layout information, the pre-trained encoder is pre-trained for the masked visual language modeling task. The masked visual language modeling task is to input text by randomly masking, and the training encoder model needs to predict these masked texts. Through instruction adjustment, the pre-trained encoder is pre-trained at three different levels of layout perception, so that the model can better understand the structure of the document from different levels, and can improve the model's overall understanding of the document image when detecting image violations, thereby ensuring the reliability of content extraction.

[0094] After the pre-training is completed, the large language model in the multimodal large model is further fine-tuned under the chain layout strategy through all levels of input information. Specifically, in the pre-training stage, the parameters of the large language model are frozen, and only the parameters of the pre-trained model encoder and the two multi-layer perceptrons are updated; during the supervised fine-tuning, the parameters of the pre-trained model encoder are kept frozen, and the parameters of the large language model, LLM and the two multi-layer perceptrons are fine-tuned. In the supervised fine-tuning stage, the large language model is supervised and fine-tuned under the chain layout strategy through instructions containing multiple layout hints, so that the large language model can analyze the problem according to each instruction to determine the corresponding layout area, and then perform the reasoning process of relevant area concentration and answer formation, so that the large language model can obtain the ability to locate the relevant area in the document after analyzing the problem, and then concentrate on accurately inferring the answer during the supervised training process of the layout hint instruction, thereby accurately and reliably narrowing the scope of violation detection and improving the accuracy and efficiency of violation detection.

[0095] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0096] Another embodiment of the present invention provides a device for detecting image violations. Figure 6 As shown, the device 1 comprises:

[0097] The text layout extraction module 11 is used to obtain the document image to be detected and extract the text information and layout information in the document image;

[0098] An input module 12, used to input the document image, text information and layout information into a trained multimodal large model, wherein the multimodal large model includes a pre-trained encoder and a large language model;

[0099] A feature extraction module 13, configured to extract features from the document image, text information and layout information through the pre-trained encoder to obtain multimodal features, wherein the pre-trained encoder is trained through layout perception pre-training tasks at different levels;

[0100] A content extraction module 14 is used to extract content of the multimodal features and the violation detection instruction using the large language model in response to the violation detection instruction, and obtain the content of the specified layout area in the document image;

[0101] The violation detection module 15 is used to perform violation detection on the content of the specified layout area and output corresponding violation detection results.

[0102] The module referred to in the present invention refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of image violation detection than a program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0103] In one embodiment, the feature extraction module 13 includes:

[0104] An encoding unit, used for encoding the document image, text information and layout information through the pre-trained encoder to obtain visual features and text layout features;

[0105] A projection unit is used to project the visual features and the text layout features from the output space of the pre-trained encoder to the embedding space of the large language model.

[0106] In one embodiment, the content extraction module 14 includes:

[0107] An instruction input unit, used for receiving a violation detection instruction, and inputting the multimodal features and the violation detection instruction into a trained large language model;

[0108] A question analysis and layout positioning unit, configured to analyze the question in the violation detection instruction by using the large language model, and determine a designated layout area in the document image associated with the question according to the multimodal features;

[0109] The content extraction unit is used to extract the content of the designated layout area in a centralized manner according to the question, so as to obtain the answer content of the question.

[0110] In one embodiment, the violation detection module 15 is specifically used to:

[0111] The violation category and violation score of the content in the designated layout area are predicted, and a violation detection result is outputted indicating whether a preset violation type exists and a corresponding violation score.

[0112] In one embodiment, the device 1 further comprises:

[0113] A data acquisition module, used to acquire training images and identify text data and layout data in the training images;

[0114] An information construction module, used to construct input information of different levels according to the training images, text data and layout data, wherein the input information includes the overall document level, the layout area level and the paragraph sentence level;

[0115] The training module is used to perform layout-aware pre-training and supervised fine-tuning on the multimodal large model according to the input information to obtain a trained multimodal large model.

[0116] In one embodiment, the training module includes:

[0117] A pre-training unit, used for inputting different levels of input information into the multimodal large model, and performing different levels of layout perception pre-training on the pre-trained encoder through corresponding pre-training task instructions;

[0118] A fine-tuning unit is used to input all levels of input information into the multimodal large model, and to perform supervised fine-tuning of the large language model under a chain layout strategy through instructions containing multiple layout hints.

[0119] In one embodiment, the training module further includes:

[0120] The preprocessing unit is used to perform corresponding preprocessing on the training images, text data and layout data respectively.

[0121] Another embodiment of the present invention provides a system for detecting image violations. Figure 7 As shown, the system 10 includes:

[0122] One or more processors 110 and memory 120, Figure 7 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0123] The processor 110 is used to complete various control logics of the system 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0124] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the image violation detection method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the system 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implementing the image violation detection method in the above method embodiment.

[0125] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the system 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the system 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0126] One or more units are stored in the memory 120, and when executed by one or more processors 110, the image violation detection method in any of the above method embodiments is executed, for example, the following steps are executed:

[0127] Obtaining a document image to be detected, and extracting text information and layout information from the document image;

[0128] Inputting the document image, text information and layout information into a trained multimodal large model, wherein the multimodal large model includes a pre-trained encoder and a large language model;

[0129] Extracting features of the document image, text information and layout information through the pre-trained encoder to obtain multimodal features, wherein the pre-trained encoder is trained through layout perception pre-training tasks at different levels;

[0130] In response to the violation detection instruction, extracting the content of the chain layout strategy from the multimodal features and the violation detection instruction through the large language model to obtain the content of the specified layout area in the document image;

[0131] Perform violation detection on the content of the designated layout area and output corresponding violation detection results.

[0132] An embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the image violation detection method in any of the above method embodiments is executed, for example, the following steps are performed:

[0133] Obtaining a document image to be detected, and extracting text information and layout information from the document image;

[0134] Inputting the document image, text information and layout information into a trained multimodal large model, wherein the multimodal large model includes a pre-trained encoder and a large language model;

[0135] Extracting features of the document image, text information and layout information through the pre-trained encoder to obtain multimodal features, wherein the pre-trained encoder is trained through layout perception pre-training tasks at different levels;

[0136] In response to the violation detection instruction, extracting the content of the chain layout strategy from the multimodal features and the violation detection instruction through the large language model to obtain the content of the specified layout area in the document image;

[0137] Perform violation detection on the content of the designated layout area and output corresponding violation detection results.

[0138] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0139] In summary, the present invention discloses a method, device, system and medium for detecting image violations, the method comprising: obtaining a document image to be detected, extracting text information and layout information from the document image and inputting them into a trained multimodal large model, extracting features of the document image, text information and layout information through a pre-trained encoder therein to obtain multimodal features, the pre-trained encoder being trained through layout perception pre-training tasks at different levels; in response to a violation detection instruction, extracting content of a chain layout strategy from the multimodal features and the violation detection instruction through a large language model in the multimodal large model to obtain the content of a specified layout area in the document image; performing violation detection on the content of the specified layout area, and outputting corresponding violation detection results. The multimodal large model is enabled to locate relevant areas in the document image through multimodal features, and the content is extracted intensively for violation detection, thereby improving the accuracy of violation detection on the document image.

[0140] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0141] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for detecting image violations. It is characterized in that include: Obtaining a document image to be detected, and extracting text information and layout information from the document image; Inputting the document image, text information and layout information into a trained multimodal large model, wherein the multimodal large model includes a pre-trained encoder and a large language model; Extracting features of the document image, text information and layout information through the pre-trained encoder to obtain multimodal features, wherein the pre-trained encoder is trained through layout perception pre-training tasks at different levels; In response to the violation detection instruction, extracting the content of the chain layout strategy from the multimodal features and the violation detection instruction through the large language model to obtain the content of the specified layout area in the document image; Perform violation detection on the content of the designated layout area and output corresponding violation detection results.

2. The image violation detection method according to claim 1, It is characterized in that The pre-trained encoder is used to extract features from the document image, text information, and layout information to obtain multimodal features, including: Encoding the document image, text information and layout information by the pre-trained encoder to obtain visual features and text layout features; The visual features and the text layout features are projected from the output space of the pre-trained encoder to the embedding space of the large language model.

3. The image violation detection method according to claim 1, It is characterized in that In response to the violation detection instruction, extracting the content of the chain layout strategy from the multimodal features and the violation detection instruction through the large language model to obtain the content of the specified layout area in the document image includes: receiving a violation detection instruction, and inputting the multimodal features and the violation detection instruction into a trained large language model; Analyzing the question in the violation detection instruction by using the large language model, and determining a designated layout area in the document image associated with the question according to the multimodal feature; The designated layout area is centrally extracted based on the question to obtain the answer content of the question.

4. The image violation detection method according to claim 1, It is characterized in that The performing violation detection on the content of the designated layout area and outputting the corresponding violation detection result specifically includes: The violation category and violation score of the content in the designated layout area are predicted, and a violation detection result is outputted indicating whether a preset violation type exists and a corresponding violation score.

5. The image violation detection method according to any one of claims 1 to 4, It is characterized in that The training process of the multimodal large model includes: Collecting training pictures and identifying text data and layout data in the training pictures; Constructing input information of different levels according to the training images, text data and layout data, wherein the input information includes the overall document level, the layout area level and the paragraph sentence level; The multimodal large model is subjected to layout-aware pre-training and supervised fine-tuning according to the input information to obtain a trained multimodal large model.

6. The image violation detection method according to claim 5, It is characterized in that The performing layout-aware pre-training and supervised fine-tuning on the multimodal large model according to the input information to obtain a trained multimodal large model includes: Inputting different levels of input information into the multimodal large model, and performing different levels of layout perception pre-training on the pre-trained encoder through corresponding pre-training task instructions; Input information from all levels is fed into a large multimodal model, which is fine-tuned under a supervised chained layout strategy via instructions containing multiple layout cues.

7. The image violation detection method according to claim 5, It is characterized in that After collecting the training pictures and identifying the text data and layout data in the training pictures, the method further includes: The training images, text data and layout data are preprocessed accordingly.

8. A device for detecting image violations, It is characterized in that include: A text layout extraction module is used to obtain a document image to be detected and extract text information and layout information from the document image; An input module, used to input the document image, text information and layout information into a trained multimodal large model, wherein the multimodal large model includes a pre-trained encoder and a large language model; A feature extraction module, used to extract features from the document image, text information and layout information through the pre-trained encoder to obtain multimodal features, wherein the pre-trained encoder is trained through layout perception pre-training tasks at different levels; A content extraction module, configured to, in response to a violation detection instruction, extract content of a chain layout strategy from the multimodal features and the violation detection instruction through the large language model to obtain content of a specified layout area in the document image; The violation detection module is used to perform violation detection on the content of the specified layout area and output corresponding violation detection results.

9. A system for detecting image violations. It is characterized in that The system includes at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image violation detection method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, It is characterized in that The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the image violation detection method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Cross-modal sensitive information identification method, system and terminal

    CN115758282A

  • Method and device for realizing Internet harmful image detection processing based on large language model, processor and computer readable storage medium

    CN117876911A

Cited By

  • Method, apparatus, and electronic device for connector anomaly recognition based on multimodal large model fine-tuning

    TWI931285B