Illegal behavior detection method and device based on multi-modal information fusion, and terminal
By adopting sliding window slices and multimodal information fusion methods in the financial field, the problem of multimodal model not being detailed enough image description in financial scenarios is solved, the accuracy and efficiency of violation detection is improved, and high resolution is achieved Effective processing of rate image and text data.
Patent Information
- Application Number
- CN202411185643.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-11-15
AI Technical Summary
When existing multimodal models process high-resolution images and text data in financial scenarios, the image description is not detailed enough, which limits the multimodal models' ability to understand image-text relationships, resulting in low accuracy in detection of violations.
Using a multimodal information fusion method based on sliding windows, the artificial intelligence model is trained by slicing high-resolution images and combining text data to form a multimodal information fusion model to improve the accuracy of violation detection .
It effectively solves the problem of long-term graphics and text information processing capabilities in the financial field of customer service violation detection, improves the accuracy and efficiency of violation detection, and can better capture visual details and understand long text content.
Smart Images

Figure CN118965079A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of financial technology, and in particular to a method, device and terminal for detecting illegal behavior based on multimodal information fusion. Background Art
[0002] At present, large multimodal models (LMMs) have made significant progress in processing image and text data, especially in tasks such as image description and visual question answering, which has attracted widespread attention in the academic community. However, there are still some significant challenges and shortcomings when applying these technologies in financial scenarios.
[0003] Existing solutions include using pre-trained vision modules to support larger input resolutions, such as LLaVA1.5, and gradually increasing the resolution during training through curriculum learning, such as Qwen-VL, PaLI-3, and PaLI-X. Although these methods have improved the performance of the model to some extent, they have high requirements for training resources and still face challenges when processing larger image sizes.
[0004] Especially in the financial field, it is necessary to process high-resolution images and accurately capture subtle visual details for accurate quality inspection and analysis. The performance of existing technologies in these aspects still cannot fully meet the needs of the financial industry, and there is a lack of systems and methods for efficiently processing multimodal data. Therefore, it is an urgent need to develop more efficient and accurate multimodal training and inference methods for financial scenarios to support key tasks such as financial security and risk management.
[0005] In summary, although the existing technology has made certain progress in the development and application of multimodal models, in processing financial scenarios, the image descriptions in existing data sets are usually not detailed enough to fully improve the multimodal model's ability to understand the image-text relationship, which limits the application effect of multimodal models in processing high-demand financial scenarios and the accuracy of violation detection is low.
[0006] Therefore, the prior art still needs to be improved and developed. Summary of the invention
[0007] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method, device and terminal for detecting violations based on multimodal information fusion, aiming to solve the problem that the image descriptions in the data set in the prior art are usually not detailed enough, which cannot fully improve the multimodal model's ability to understand the image-text relationship, limits the application effect of the multimodal model in processing high-demand financial scenarios, and has a low accuracy rate in detecting violations.
[0008] The technical solution of the present invention is as follows:
[0009] The first embodiment of the present invention provides a method for detecting illegal behavior based on multimodal information fusion, the method comprising:
[0010] Collect chat data during the chat process between the customer service and the customer, wherein the chat data is the chat content sent by the customer service, and the chat content includes chat text information and chat long pictures;
[0011] Slicing the long chat picture based on a sliding window to obtain picture slice data;
[0012] Generate training samples based on the chat text information, the long chat picture and the picture slice data, train the artificial intelligence model based on the training samples to obtain a multimodal information fusion model, wherein the artificial intelligence model includes a multimodal information fusion processing module and an output module;
[0013] The customer service chat content to be detected is obtained, the customer service chat content to be detected is input into the multimodal information fusion model, and the violation detection result output by the multimodal information fusion model is obtained.
[0014] Furthermore, the collection of chat data during the chat process between the customer service and the customer includes:
[0015] Collect chat content sent by customer service during the chat between customer service and customers, including normal chat content and illegal chat content;
[0016] Annotating the customer service chat content to obtain annotated customer service chat content;
[0017] The annotated customer service chat content is preprocessed to obtain preprocessed customer service chat content.
[0018] Furthermore, the slicing operation is performed on the long chat picture based on the sliding window to obtain picture slice data, including:
[0019] Matching the chat text content with the long chat picture one by one to obtain corresponding information between the chat text content and the long chat picture;
[0020] Slicing the long chat picture based on a sliding window to obtain picture slices;
[0021] Based on the corresponding information, obtaining chat text content corresponding to the image slice, and truncating the corresponding chat text content to obtain truncated text content;
[0022] Based on the picture slice and the truncated text content, picture slice data is obtained.
[0023] Furthermore, generating training samples based on the chat text information, the long chat picture and the picture slice data includes:
[0024] Inputting the chat text content into a text feature extraction module to obtain chat text features;
[0025] Inputting the long chat picture into a picture feature extraction module to obtain features of the long chat picture;
[0026] Inputting the image slice data into an image slice data extraction module to obtain image slice features;
[0027] A training sample is obtained based on the chat text features, the chat long picture features, and the picture slice text features.
[0028] Furthermore, the artificial intelligence model is trained based on the training samples to obtain a multimodal information fusion model, including:
[0029] Training the artificial intelligence model based on the training samples to obtain an initial multimodal information fusion model;
[0030] Collecting chat sample data for testing, and testing the initial multimodal information fusion model based on the chat sample data;
[0031] If the test is successful, the initial multimodal information fusion model is used as the final multimodal information fusion model;
[0032] If the test fails, the network parameters of the initial multimodal information fusion model are adjusted until the initial multimodal information fusion model test succeeds, and the initial multimodal information fusion model when the test succeeds is used as the multimodal information fusion model.
[0033] Further, the acquiring of customer service chat content to be detected and inputting the customer service chat content to be detected into the multimodal information fusion model includes:
[0034] Acquire customer service chat content to be detected, and pre-process the detected customer service chat content to obtain initial customer service chat content;
[0035] Extracting features from the initial customer service chat content to obtain customer service chat features to be detected;
[0036] The customer service chat features to be detected are input into the multimodal information fusion model.
[0037] Furthermore, the obtaining of the violation result output by the multimodal information fusion model includes:
[0038] Obtain the violation type, violation score and violation identification result output by the multimodal information fusion model.
[0039] Another embodiment of the present invention provides a violation detection device based on multimodal information fusion, the device comprising:
[0040] A data collection module is used to collect chat data during the chat process between the customer service and the customer. The chat data is the chat content sent by the customer service, and the chat content includes chat text information and chat long pictures;
[0041] A picture slicing module, used to slice the long chat picture based on a sliding window to obtain picture slice data;
[0042] A model training module, used to generate training samples based on the chat text information, the long chat picture and the picture slice data, and train the artificial intelligence model based on the training samples to obtain a multimodal information fusion model, wherein the artificial intelligence model includes a multimodal information fusion processing module and an output module;
[0043] The behavior recognition module is used to obtain customer service chat content to be detected, input the customer service chat content to be detected into the multimodal information fusion model, and obtain the violation behavior detection result output by the multimodal information fusion model.
[0044] Another embodiment of the present invention provides a terminal, the terminal comprising at least one processor; and
[0045] a memory communicatively connected to the at least one processor; wherein,
[0046] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned violation detection method based on multimodal information fusion.
[0047] Another embodiment of the present invention further provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned violation detection method based on multimodal information fusion.
[0048] Beneficial effects: The violation detection method based on multimodal information fusion of the embodiment of the present invention mainly solves the problem of long image and text information processing capability of customer service violation quality inspection in the financial field. The technical framework includes a text, image and slice image data feature extraction module, a multimodal information fusion processing module, and training on image, text, and slice image data to learn multimodal feature representation. In the inference stage, the violation situation is detected by the violation type and violation score inferred by the large model. Different from the previous violation detection method based on large models, a large model and multi-resolution and multi-slice image information fusion are introduced, and a multi-level image feature fusion is integrated from the feature space to provide rich information. Moreover, because the large model has the ability of few-shot learning and contextual connection, it can further obtain feature information for judging violations. It can effectively solve the problem of customer service violation detection in the financial field and improve the accuracy of customer service violation detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:
[0050] Figure 1 It is a flow chart of a preferred embodiment of a method for detecting illegal behavior based on multimodal information fusion of the present invention;
[0051] Figure 2 A schematic diagram of the network structure of a multimodal information fusion model of a preferred embodiment of a method for detecting illegal behavior based on multimodal information fusion according to the present invention;
[0052] Figure 3 It is a functional module diagram of a preferred embodiment of a violation detection device based on multimodal information fusion of the present invention;
[0053] Figure 4 The figure is a schematic diagram of the hardware structure of a preferred embodiment of a terminal of the present invention. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0055] In the process of chatting between customer service and customers in the prior art, the actual situation is that long pictures and a large amount of text information are often accompanied, which makes customer service violation detection challenging, which brings great difficulties to the large model based on artificial intelligence. In response to the above technical problems, this application proposes a violation detection method based on multimodal information fusion, which mainly solves the problem of long image and text information processing capability of customer service violation quality inspection in the financial field. The technical framework includes text, picture and sliced image data feature extraction modules, multimodal information fusion processing modules, and training on pictures, texts, and sliced image data to learn multimodal feature representation. In the inference stage, the technology detects the violation situation through the violation type and violation score inferred by the large model, which mainly solves the problem of long image and text information processing capability of customer service violation quality inspection in the financial field. The technical framework includes text, picture and sliced image data feature extraction modules, multimodal information fusion processing modules, and training on pictures, texts, and sliced image data to learn multimodal feature representation. In the inference stage, the technology detects violations through the violation types and violation scores inferred by the large model, thereby improving the accuracy of customer service violation detection.
[0056] The embodiments of the present invention are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0057] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. Here, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0058] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robots, natural speech processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.
[0059] In view of the above problems, the present invention provides a method for detecting illegal behaviors based on multimodal information fusion. Figure 1 , Figure 1 FIG. 1 is a flow chart of a preferred embodiment of a method for detecting illegal behaviors based on multimodal information fusion according to the present invention. Figure 1 As shown, it includes:
[0060] Step S100: collecting chat data of a chat process between a customer service representative and a customer, wherein the chat data is chat content sent by the customer service representative, and the chat content includes chat text information and chat long pictures;
[0061] Step S200: Slicing the long chat picture based on a sliding window to obtain picture slice data;
[0062] Step S300: generating training samples based on the chat text information, the long chat picture and the picture slice data, and training an artificial intelligence model based on the training samples to obtain a multimodal information fusion model, wherein the artificial intelligence model includes a multimodal information fusion processing module and an output module;
[0063] Step S400: obtaining customer service chat content to be detected, inputting the customer service chat content to be detected into the multimodal information fusion model, and obtaining the violation detection result output by the multimodal information fusion model.
[0064] In specific implementation, the violation detection method based on multimodal information fusion of the embodiment of the present invention is mainly used for violation detection in the chat process between customer service and customers in financial scenarios.
[0065] The chat content sent by the customer service during the chat between the customer service and the customer is collected. The chat content during the chat between the customer service and the customer is often accompanied by long pictures and a large amount of text information. Therefore, the chat content includes but is not limited to chat text information and long chat pictures.
[0066] Compared with the method of directly interpolating ViT (Vision Transformer) to increase the input resolution, a new module is used in the embodiment of the present invention to divide the high-resolution image into smaller pictures by a sliding window method to obtain picture slice data. Each small picture is processed independently by a static visual encoder and trained by LoRA to obtain a small visual encoder. Among them, LoRA (Low-Rank Adaptation of Large Language Models) is a technology for fine-tuning large language models (such as GPT series models). It aims to reduce computational complexity and storage requirements by introducing low-rank matrices while maintaining the improvement of model performance. LoRA technology allows researchers and developers to quickly and effectively adjust the model to adapt to specific tasks or data sets without having to train the entire model from scratch. LoRA achieves fine-tuning by adding a small trainable low-rank matrix to the pre-trained model. This low-rank matrix is used to adjust the weights of the pre-trained model without changing most of the parameters of the original model. Specifically, LoRA decomposes the weight matrix of the original model into the product of two smaller matrices and inserts a trainable low-rank matrix between the two matrices. In this way, LoRA is able to fine-tune the model by only updating this low-rank matrix, significantly reducing the required computing resources and storage space. LoRA is computationally efficient: Since LoRA only updates the low-rank matrix, the required computing resources are greatly reduced compared to fully fine-tuning the entire model. Low storage requirements: The introduction of low-rank matrices significantly reduces the storage requirements after model fine-tuning. High flexibility: LoRA allows users to train multiple low-rank matrices for different tasks or data sets and easily switch when needed, thereby achieving rapid adaptation of the model. Maintain original model performance: Through a carefully designed low-rank matrix, LoRA is able to maintain most of the performance of the original model during fine-tuning while optimizing for specific tasks.
[0067] After processing the chat text information, long chat pictures and picture slice data respectively, training samples are generated, and the artificial intelligence model is trained based on the training samples to obtain a multimodal information fusion model. The artificial intelligence model includes a multimodal information fusion processing module for fusing chat text information, long chat pictures and picture slice data, and an output module for outputting the recognition results. Preferably, the artificial intelligence model can adopt the QWEN model. The QWEN model is a large-scale language model based on the Transformer architecture. The model is pre-trained by predicting the next word, focusing on the expansion of scale and data to ensure the simplicity and stability of the model. The Qwen model has: Multilingual processing capabilities: The Qwen large model supports multiple languages, especially in English and Chinese, and can effectively cope with other languages such as Spanish, French, and Japanese. This is due to the multilingual nature of its pre-training data and the proprietary efficient word segmenter used. Long text processing capabilities: The Qwen model pays special attention to the expansion of context length in the pre-training stage, and uses rotational position encoding (RoPE) with longer context length and larger cardinality for continuous training. This allows the open source Qwen model to typically support context windows of up to 32K word tokens, ensuring accuracy and coherence when processing long texts. Deep optimization: On top of the base model, Tongyi Qianwen deeply optimizes Qwen through post-training techniques such as SFT (supervised fine-tuning) and RLHF (reinforcement learning human feedback) to achieve a higher level of alignment and interactivity. These technologies ensure the model's outstanding performance in following instructions, showing innovation, and gaining human satisfaction. Tool usage and agent-playing capabilities: The Qwen model is able to generate responses that include thinking and actions, and perform situational reasoning based on previous steps and observations. It can not only understand and use known tools (such as function calls, code interpreters, Hugging Face agents), but also master tools that have never been touched through situational learning.
[0068] A customer service violation detection instruction is detected, customer service chat content to be detected is obtained, the customer service chat content to be detected is input into a trained multimodal information fusion model, and a violation detection result output by the multimodal information fusion model is obtained.
[0069] The embodiment of the present invention monitors customer service violations in financial scenarios, and uses an artificial intelligence model to help monitor violations, automatically detect whether there are violations in the area, and improve financial security and regulatory efficiency.
[0070] In one embodiment, the chat data of the chat process between the customer service and the customer is collected, including:
[0071] Collect chat content sent by customer service during the chat between customer service and customers, including normal chat content and illegal chat content;
[0072] Annotating the customer service chat content to obtain annotated customer service chat content;
[0073] The annotated customer service chat content is preprocessed to obtain preprocessed customer service chat content.
[0074] During specific implementation, the chat content sent by the customer service during the chat between the customer service and the customer is collected, and the chat content sent by the customer service during the chat between the customer service and the customer is collected, and the chat content is marked and pre-processed to obtain the pre-processed customer service chat content.
[0075] The text data preprocessing process is as follows:
[0076] Collect text data. Clean the text to remove special characters, HTML tags, URL links and other non-text content, and only keep the plain text part. After the conversion from traditional to simplified Chinese, the pre-processed text data is used pre express.
[0077] Image data preprocessing process:
[0078] Collect image data; perform zooming operations; adjust contrast; use i pre express.
[0079] In one embodiment, a slicing operation is performed on the long chat picture based on a sliding window to obtain picture slice data, including:
[0080] Matching the chat text content with the long chat picture one by one to obtain corresponding information between the chat text content and the long chat picture;
[0081] Slicing the long chat picture based on a sliding window to obtain picture slices;
[0082] Based on the corresponding information, obtaining chat text content corresponding to the image slice, and truncating the corresponding chat text content to obtain truncated text content;
[0083] Based on the picture slice and the truncated text content, picture slice data is obtained.
[0084] In specific implementation, the chat text content i pre Chat with long pictures pre Make correspondence and obtain corresponding information.
[0085] Based on the preset sliding window, i pre For slicing use p pre,i Indicates that the corresponding text also needs to be truncated with p pre,t express.
[0086] The sliced images and the corresponding truncated texts are matched one by one to form a patch, and the image slice data p is used pre Indicates. pre Equal to p pre =(p pre,i , p pre,t ).
[0087] In one embodiment, generating training samples based on the chat text information, the long chat picture and the picture slice data includes:
[0088] Inputting the chat text content into a text feature extraction module to obtain chat text features;
[0089] Inputting the long chat picture into a picture feature extraction module to obtain features of the long chat picture;
[0090] Inputting the image slice data into an image slice data extraction module to obtain image slice features;
[0091] A training sample is obtained based on the chat text features, the chat long picture features, and the picture slice text features.
[0092] In the specific implementation, the image and graphic slice data i pre and p pre,i Input into the image feature extraction module Φ i And the image and text slice feature extraction module Φ p , and get the corresponding feature information:
[0093] F i =Φ i (i pre )
[0094] F pi =Φ pi (p pre,i )|
[0095] F p 、F i and t pre Input into the multimodal information fusion processing module Ψ to obtain fusion feature information. The formula is as follows:
[0096] F pt =Φ pt (p pre,t , F pi )
[0097] F f =Ψ(F pt , F i , t pre ).
[0098] In one embodiment, the artificial intelligence model is trained based on the training samples to obtain a multimodal information fusion model, including:
[0099] Training the artificial intelligence model based on the training samples to obtain an initial multimodal information fusion model;
[0100] Collecting chat sample data for testing, and testing the initial multimodal information fusion model based on the chat sample data;
[0101] If the test is successful, the initial multimodal information fusion model is used as the final multimodal information fusion model;
[0102] If the test fails, the network parameters of the initial multimodal information fusion model are adjusted until the initial multimodal information fusion model test succeeds, and the initial multimodal information fusion model when the test succeeds is used as the multimodal information fusion model.
[0103] In specific implementation, F f Input to the output module Ω to get the violation type Violation Score And recognition results
[0104]
[0105] Collect chat sample data for testing, and annotate and preprocess it.
[0106] The test sample passes through the artificial intelligence model to obtain the predicted violation category and predicted violation score of the test sample.
[0107] Based on the test results, the artificial intelligence model used for detecting customer service violations in financial scenarios is judged to determine whether it is usable.
[0108] If the test is successful, the initial multimodal information fusion model is used as the final multimodal information fusion model;
[0109] If the test fails, the network parameters of the initial multimodal information fusion model are adjusted until the initial multimodal information fusion model test succeeds, and the initial multimodal information fusion model when the test succeeds is used as the multimodal information fusion model.
[0110] In one embodiment, obtaining customer service chat content to be detected and inputting the customer service chat content to be detected into the multimodal information fusion model includes:
[0111] Acquire customer service chat content to be detected, and pre-process the detected customer service chat content to obtain initial customer service chat content;
[0112] Extracting features from the initial customer service chat content to obtain customer service chat features to be detected;
[0113] The customer service chat features to be detected are input into the multimodal information fusion model.
[0114] In the specific implementation, the customer service chat content to be detected is obtained, the type of the customer service chat content is obtained, and if the customer service chat content is text, the text data is collected. The text is cleaned to remove special characters, HTML tags, URL links and other non-text content, and only the plain text part is retained.
[0115] After the traditional and simplified Chinese characters are converted, the initial text content is obtained;
[0116] If the customer service chat content is a long picture, the picture data is collected; the zoom operation is performed; the contrast is adjusted to obtain the initial long picture content. The initial long picture content is sliced based on the preset sliding window to obtain the initial picture slice content.
[0117] The initial text content is input into the text feature extraction module to obtain the initial text features; the initial long picture content is input into the picture feature extraction module to obtain the initial long picture features, and the initial picture slice content is input into the slice text feature extraction module to obtain the initial picture slice features; the initial text features, initial long picture features, and initial picture slice features are input into the multimodal information fusion model for fusion, and then the violation identification results are output through the output module.
[0118] In one embodiment, obtaining the violation result output by the multimodal information fusion model includes:
[0119] Obtain the violation type, violation score and violation identification result output by the multimodal information fusion model.
[0120] When implementing it, Figure 2 As shown in FIG, a large model based on multimodal information fusion includes a text feature extraction module, an image feature extraction module, a graphic and text slice data feature extraction module, a multimodal information fusion processing module, and an output module. The text, image, and graphic slice are respectively input into the corresponding feature extraction module, and then the feature information is input into the multimodal information fusion processing module. Then it is input into the output module to output the violation type, violation score, and violation identification result. The violation identification result includes the existence of a violation or the absence of a violation. The types of violations include exaggerating profits, inducing loans, and leaking private information.
[0121] The slice text feature extraction module and the image feature extraction module use the QWEN-1.5B model and the VIT model respectively. Among them, the QWEN-1.5B model is a medium-sized model in the QWEN series, with a specific parameter scale of 1.5B (i.e. 1.5 billion). The QWEN1.5 series also includes models of other different scales, such as 0.5B, 1.8B, 4B, 7B, 14B and 72B, as well as a MoE (Mixture of Experts) model. The QWEN-1.5B model has: Language generation and understanding capabilities: The QWEN-1.5B model has strong language generation and understanding capabilities and can handle various complex natural language tasks. Multilingual capabilities: The model supports multiple languages, including Chinese and English, and is suitable for applications in cross-language scenarios. Long sequence support: The QWEN-1.5B model supports long sequence processing and can handle long text inputs containing a lot of information. Fine-tuning and customization: Users can use fine-tuning technology to customize and optimize the model according to specific tasks or scenarios to improve the performance of the model on specific tasks.
[0122] The VIT model, full name Vision Transformer, is an innovative method to apply the Transformer model to computer vision tasks. The VIT model is a deep learning model based on the Transformer architecture, specifically designed for computer vision tasks. It achieves global understanding and processing of images by dividing the image into small patches and inputting these patches as sequences into the Transformer model.
[0123] The model principle is as follows:
[0124] Image segmentation: The input image is first divided into small patches of fixed size.
[0125] Each small block is converted into a vector representation after linear embedding, which is similar to word embedding in natural language processing.
[0126] Position encoding: In order to preserve the spatial information between image patches, the ViT model adds position encoding to each image patch. These position encodings can be learnable or fixed (such as sinusoidal position encoding).
[0127] Transformer Encoder: The sequence of image patch vectors is then fed into the encoder layer of the Transformer for processing. The encoder layer contains multiple layers of multi-head self-attention mechanisms and feed-forward neural networks. The self-attention mechanism enables the model to capture global context information and long-term dependencies.
[0128] Classification head: The output of the Transformer encoder passes through a classification head for tasks such as image classification. Usually a classification token (CLS token) is added at the beginning of the image block sequence, and the final classification result is based on the output of the classification token.
[0129] The embodiment of the present invention increases the input resolution in a large multimodal model framework. Compared with the method of directly interpolating ViT to increase the input resolution, a new module is used to divide the high-resolution image into smaller pictures through a sliding window method. Each small picture is processed independently by a static visual encoder, and a small visual encoder is obtained by training with LoRA.
[0130] This technology leverages existing LMMs while avoiding the need for full pre-training. These encoders are typically trained at lower resolutions (such as 448×448), which is expensive to train from scratch. By adjusting each patch to its supported resolution, the distribution of the encoder's training data is maintained. This method leverages a variety of trainable patches to enhance resolution. In addition, each small image's corresponding text information also has a corresponding small text encoder trained. The one-to-one correspondence and feature learning between global images and global text, local images and local text are guaranteed, achieving OCR-Free (no need to introduce an additional OCR model), being able to process high-resolution images and accurately capture subtle visual details, and improving the ability to understand long text content for accurate quality inspection and analysis.
[0131] Compared with the prior art, the violation detection method based on multimodal information fusion in the embodiment of the present invention has the following technical advantages:
[0132] Applicable to long image and text data: The image resolution is higher than the 448×448 resolution commonly used in LMMs. Higher resolution helps to better identify and understand small or tightly grouped objects and dense text. The features of the global image and the local image of the long image are fused and trained uniformly to reduce the loss of local image information. It is better suitable for anomaly detection of long images in the financial field.
[0133] Ability to handle long text and image context understanding: A multi-level description generation method is introduced to improve the model's ability to grasp the relationship between multiple targets and more effectively use common sense to generate text descriptions. This method has obvious advantages in capturing the extensive visual details and semantic information of financial images, and can more comprehensively analyze user behavior and identify potential violations.
[0134] No additional training is required, and it has a certain ability to learn in a small number of times: By training a small module feature extraction model through LoRA, there is no need to pre-train a large model, and the learning ability based on the large model can achieve some exceptional samples in a small number of times.
[0135] Scalability and adaptability: Large models can acquire extensive knowledge and language expression capabilities through pre-training, thus having strong scalability and adaptability. This enables them to handle violation detection tasks based on multimodal information fusion in different fields and contexts, and have a certain generalization ability.
[0136] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.
[0137] Another embodiment of the present invention provides a violation detection device based on multimodal information fusion, such as Figure 3 As shown, the device 1 comprises:
[0138] The data collection module 11 is used to collect chat data during the chat process between the customer service and the customer, wherein the chat data is the chat content sent by the customer service, and the chat content includes chat text information and chat long pictures;
[0139] The picture slicing module 12 is used to slice the long chat picture based on a sliding window to obtain picture slice data;
[0140] A model training module 13, used to generate training samples based on the chat text information, the long chat picture and the picture slice data, and train the artificial intelligence model based on the training samples to obtain a multimodal information fusion model, wherein the artificial intelligence model includes a multimodal information fusion processing module and an output module;
[0141] The behavior recognition module 14 is used to obtain customer service chat content to be detected, input the customer service chat content to be detected into the multimodal information fusion model, and obtain the violation behavior detection result output by the multimodal information fusion model.
[0142] The specific implementation method is shown in the method embodiment, which will not be described in detail here.
[0143] In one embodiment, the data acquisition module 11 is specifically used for:
[0144] Collect chat content sent by customer service during the chat between customer service and customers, including normal chat content and illegal chat content;
[0145] Annotating the customer service chat content to obtain annotated customer service chat content;
[0146] The annotated customer service chat content is preprocessed to obtain preprocessed customer service chat content.
[0147] The specific implementation method is shown in the method embodiment, which will not be described in detail here.
[0148] In one embodiment, the data acquisition module 12 is specifically used for:
[0149] Matching the chat text content with the long chat picture one by one to obtain corresponding information between the chat text content and the long chat picture;
[0150] Slicing the long chat picture based on a sliding window to obtain picture slices;
[0151] Based on the corresponding information, obtaining chat text content corresponding to the image slice, and truncating the corresponding chat text content to obtain truncated text content;
[0152] Based on the picture slice and the truncated text content, picture slice data is obtained.
[0153] The specific implementation method is shown in the method embodiment, which will not be described in detail here.
[0154] In one embodiment, the model training module 13 is also used to:
[0155] Inputting the chat text content into a text feature extraction module to obtain chat text features;
[0156] Inputting the long chat picture into a picture feature extraction module to obtain features of the long chat picture;
[0157] Inputting the image slice data into an image slice data extraction module to obtain image slice features;
[0158] A training sample is obtained based on the chat text features, the chat long picture features, and the picture slice text features.
[0159] The specific implementation method is shown in the method embodiment, which will not be described in detail here.
[0160] In one embodiment, the model training module 13 is also used to:
[0161] Training the artificial intelligence model based on the training samples to obtain an initial multimodal information fusion model;
[0162] Collecting chat sample data for testing, and testing the initial multimodal information fusion model based on the chat sample data;
[0163] If the test is successful, the initial multimodal information fusion model is used as the final multimodal information fusion model;
[0164] If the test fails, the network parameters of the initial multimodal information fusion model are adjusted until the initial multimodal information fusion model test succeeds, and the initial multimodal information fusion model when the test succeeds is used as the multimodal information fusion model.
[0165] The specific implementation method is shown in the method embodiment, which will not be described in detail here.
[0166] In one embodiment, the behavior recognition module 14 is further used to:
[0167] Acquire customer service chat content to be detected, and pre-process the detected customer service chat content to obtain initial customer service chat content;
[0168] Extracting features from the initial customer service chat content to obtain customer service chat features to be detected;
[0169] The customer service chat features to be detected are input into the multimodal information fusion model.
[0170] The specific implementation method is shown in the method embodiment, which will not be described in detail here.
[0171] In one embodiment, the behavior recognition module 14 is further used to:
[0172] Obtain the violation type, violation score and violation identification result output by the multimodal information fusion model.
[0173] The specific implementation method is shown in the method embodiment, which will not be described in detail here.
[0174] Another embodiment of the present invention provides a terminal, such as Figure 4 As shown, the terminal 10 includes:
[0175] One or more processors 110 and memory 120, Figure 4 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.
[0176] The processor 110 is used to complete various control logics of the terminal 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISCMachine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware control, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.
[0177] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the method for detecting violations based on multimodal information fusion in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implementing the method for detecting violations based on multimodal information fusion in the above method embodiment.
[0178] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store applications required for operating the device and at least one function; the data storage area may store data created according to the use of the device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0179] One or more units are stored in the memory 120, and when executed by one or more processors 110, the violation detection method based on multimodal information fusion in any of the above method embodiments is executed, for example, the above described Figure 1 The method comprises steps S100 to S400.
[0180] An embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the above-described Figure 1 The method comprises steps S100 to S400.
[0181] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory controls or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.
[0182] Another embodiment of the present invention provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the violation detection method based on multimodal information fusion of the above method embodiment. For example, executing the above described Figure 1 The method comprises steps S100 to S400.
[0183] The embodiments described above are merely illustrative, in which the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0184] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can exist in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.
[0185] Conditional language such as "can," "can," "might," or "may," among other things, unless specifically stated otherwise or otherwise understood within the context as used, is generally intended to convey that particular embodiments can include (while other embodiments do not) particular features, elements, and / or operations. Thus, such conditional language is also generally intended to imply that features, elements, and / or operations are anyway required for one or more embodiments or that one or more embodiments must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or will be performed in any particular embodiment.
[0186] What has been described herein in this specification and the accompanying drawings includes examples of methods and devices for detecting violations based on multimodal information fusion. Of course, it is not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of the present disclosure, but it is recognized that many other combinations and permutations of the disclosed features are possible. Therefore, it is obvious that various modifications can be made to the present disclosure without departing from the scope or spirit of the present disclosure. In addition, or in an alternative, other embodiments of the present disclosure may be apparent from consideration of the present specification and the accompanying drawings and from the practice of the present disclosure as presented herein. It is intended that the examples set forth in this specification and the accompanying drawings are considered to be illustrative and not restrictive in all respects. Although specific terms are employed herein, they are used in a general and descriptive sense and are not used for limiting purposes.
Claims
1. A violation detection method based on multimodal information fusion, Features , the method comprises: Collect chat data during the chat process between the customer service and the customer, wherein the chat data is the chat content sent by the customer service, and the chat content includes chat text information and chat long pictures; Slicing the long chat picture based on a sliding window to obtain picture slice data; Generate training samples based on the chat text information, the long chat picture and the picture slice data, train the artificial intelligence model based on the training samples to obtain a multimodal information fusion model, wherein the artificial intelligence model includes a multimodal information fusion processing module and an output module; The customer service chat content to be detected is obtained, the customer service chat content to be detected is input into the multimodal information fusion model, and the violation detection result output by the multimodal information fusion model is obtained.
2. The method according to claim 1, It is characterized in that The chat data collected during the chat process between the customer service and the customer includes: Collect chat content sent by customer service during the chat between customer service and customers, including normal chat content and illegal chat content; Annotating the customer service chat content to obtain annotated customer service chat content; The annotated customer service chat content is preprocessed to obtain preprocessed customer service chat content.
3. The method according to claim 2, It is characterized in that The slicing operation on the long chat picture based on the sliding window to obtain picture slice data includes: Matching the chat text content with the long chat picture one by one to obtain corresponding information between the chat text content and the long chat picture; Slicing the long chat picture based on a sliding window to obtain picture slices; Based on the corresponding information, obtaining chat text content corresponding to the image slice, and truncating the corresponding chat text content to obtain truncated text content; Based on the picture slice and the truncated text content, picture slice data is obtained.
4. The method according to claim 3, It is characterized in that The generating of training samples based on the chat text information, the long chat picture and the picture slice data includes: Inputting the chat text content into a text feature extraction module to obtain chat text features; Inputting the long chat picture into a picture feature extraction module to obtain features of the long chat picture; Inputting the image slice data into an image slice data extraction module to obtain image slice features; A training sample is obtained based on the chat text features, the chat long picture features, and the picture slice text features.
5. The method according to claim 4, It is characterized in that The artificial intelligence model is trained based on the training samples to obtain a multimodal information fusion model, including: Training the artificial intelligence model based on the training samples to obtain an initial multimodal information fusion model; Collecting chat sample data for testing, and testing the initial multimodal information fusion model based on the chat sample data; If the test is successful, the initial multimodal information fusion model is used as the final multimodal information fusion model; If the test fails, the network parameters of the initial multimodal information fusion model are adjusted until the initial multimodal information fusion model test succeeds, and the initial multimodal information fusion model when the test succeeds is used as the multimodal information fusion model.
6. The method according to claim 5, It is characterized in that The acquiring the customer service chat content to be detected and inputting the customer service chat content to be detected into the multimodal information fusion model includes: Acquire customer service chat content to be detected, and pre-process the detected customer service chat content to obtain initial customer service chat content; Extracting features from the initial customer service chat content to obtain customer service chat features to be detected; The customer service chat features to be detected are input into the multimodal information fusion model.
7. The method according to claim 6, It is characterized in that The obtaining of the violation result output by the multimodal information fusion model includes: Obtain the violation type, violation score and violation identification result output by the multimodal information fusion model.
8. A violation detection device based on multimodal information fusion, It is characterized in that The device comprises: A data collection module is used to collect chat data during the chat process between the customer service and the customer. The chat data is the chat content sent by the customer service, and the chat content includes chat text information and chat long pictures; A picture slicing module, used to slice the long chat picture based on a sliding window to obtain picture slice data; A model training module, used to generate training samples based on the chat text information, the long chat picture and the picture slice data, and train the artificial intelligence model based on the training samples to obtain a multimodal information fusion model, wherein the artificial intelligence model includes a multimodal information fusion processing module and an output module; The behavior recognition module is used to obtain customer service chat content to be detected, input the customer service chat content to be detected into the multimodal information fusion model, and obtain the violation behavior detection result output by the multimodal information fusion model.
9. A terminal, It is characterized in that The terminal comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the violation detection method based on multimodal information fusion described in any one of claims 1-7.
10. A non-volatile computer-readable storage medium, It is characterized in that The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the violation detection method based on multimodal information fusion as described in any one of claims 1-7.
