Hierarchical decision violation scene identification method and device, equipment, medium and product

Abstract

The application discloses a kind of hierarchical decision-making violation scene identification method, device, equipment, medium and product, the method first obtains the image and audio data of the scene to be identified;Again with audio event detection, speech recognition model and keyword hierarchical screening module first-level decision-making model processing, obtain first-level result;If first-level result is not hit or violation feature is not obvious, with the second-level decision-making model of text, image feature extraction model processing, obtain second-level result;If second-level result is still so, image, audio data and first, second-level result are input into the third-level decision-making model containing multimodal large language model, obtain third-level result;Finally, whether it is the target violation scene according to third-level result is determined. Through audio event detection, keyword hierarchical screening strategy, double-tower model and multimodal large model combined hierarchical strategy, violation scene efficient identification is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence risk control technology, and in particular to a method, apparatus, equipment, medium and product for identifying violation scenarios using a tiered decision-making system. Background Technology

[0002] In scenarios requiring real-time identification of violations in audio, video, and multimodal content, such as financial fraud prevention, content security auditing, and online live streaming regulation, accurate real-time detection of violations has become a critical need. Current gambling detection solutions primarily employ single-modal detection or directly call large-scale multimodal models. Single-modal detection includes plain text filtering using keyword libraries, pure image detection using image recognition technology, and pure audio judgment based on audio analysis technology. Some solutions, in pursuit of comprehensive detection, directly call large-scale multimodal models to process all data, attempting to achieve a multi-dimensional comprehensive judgment.

[0003] However, the aforementioned existing technologies have obvious drawbacks: First, single-modality approaches have inherent limitations. Pure text keyword filtering is easily bypassed by variant slang, homophones, and code words; pure image detection cannot identify voice gambling behavior; and pure audio analysis struggles to distinguish between background noise and gambling sounds, making comprehensive and accurate detection impossible. Second, there are resource and efficiency bottlenecks. Directly calling large-scale multimodal models to process all data leads to high response latency and prohibitive computational costs, lacking an efficient hierarchical processing mechanism. Third, the strategies are rigid and lack adaptability. Fixed threshold judgment mechanisms are prone to high false interception rates, and there is a lack of collaborative and conflict resolution mechanisms for multimodal evidence, making it difficult to adapt to complex and ever-changing real-world application scenarios. Summary of the Invention

[0004] This invention provides a hierarchical decision-making method, apparatus, device, medium, and product for identifying violation scenarios, so as to achieve efficient identification of violation scenarios.

[0005] According to one aspect of the present invention, a hierarchical decision-making method for identifying violation scenarios is provided, comprising:

[0006] Acquire image and audio data of the scene to be identified;

[0007] The audio data is processed based on the first-level decision model to obtain the first-level decision result corresponding to the scene to be identified. The first-level decision model includes an audio event detection model, a speech recognition model and a keyword hierarchical screening module.

[0008] When the primary decision result is a miss or the violation features are not obvious, the image data and audio data are processed based on the secondary decision model to obtain the secondary decision result corresponding to the scene to be identified. The secondary decision model includes a text feature extraction model and an image feature extraction model.

[0009] When the secondary decision result is a miss or the violation feature is not obvious, the image data, audio data, primary decision result and secondary decision result are input into the tertiary decision model to obtain the tertiary decision result corresponding to the scene to be identified. The tertiary decision model includes a multimodal large language model.

[0010] The decision is made based on the results of the three-level decision-making process to determine whether the scene to be identified is a target violation scene.

[0011] According to another aspect of the present invention, a violation scene identification device with hierarchical decision-making is provided, comprising:

[0012] The data acquisition module is used to acquire image and audio data of the scene to be identified;

[0013] The first-level decision module is used to process audio data based on the first-level decision model to obtain the first-level decision result corresponding to the scene to be identified. The first-level decision model includes an audio event detection model, a speech recognition model, and a keyword hierarchical filtering module.

[0014] The secondary decision module is used to process image and audio data based on the secondary decision model when the primary decision result is a miss or the violation feature is not obvious, so as to obtain the secondary decision result corresponding to the scene to be identified. The secondary decision model includes a text feature extraction model and an image feature extraction model.

[0015] The Level 3 decision module is used to input image data, audio data, Level 1 decision results, and Level 2 decision results into the Level 3 decision model when the Level 2 decision result is a miss or the violation feature is not obvious, so as to obtain the Level 3 decision result corresponding to the scene to be identified. The Level 3 decision model includes a multimodal large language model.

[0016] The scene judgment module is used to determine whether the scene to be identified is a target violation scene based on the three-level decision results.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] At least one processor;

[0019] and memory that is communicatively connected to at least one processor;

[0020] The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to perform the violation scene identification method of hierarchical decision-making according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the violation scene identification method of hierarchical decision-making according to any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer program product is provided, the computer program product including a computer program, which, when executed by a processor, implements the violation scene identification method of hierarchical decision-making according to any embodiment of the present invention.

[0023] The technical solution of this invention involves acquiring image and audio data of the scene to be identified; processing the audio data based on a first-level decision model to obtain a first-level decision result corresponding to the scene to be identified; if the first-level decision result is a miss or the violation characteristics are not obvious, processing the image and audio data based on a second-level decision model to obtain a second-level decision result corresponding to the scene to be identified; if the second-level decision result is a miss or the violation characteristics are not obvious, inputting the image data, audio data, first-level decision result, and second-level decision result into a third-level decision model to obtain a third-level decision result corresponding to the scene to be identified; and determining whether the scene to be identified is a target violation scene based on the third-level decision result. This technical solution, through a three-level progressive detection architecture, solves the problems of single-modality limitations, resource and efficiency bottlenecks, and rigid and unadaptable strategies in existing violation scene identification technologies by employing a hierarchical progression of rapid screening, in-depth analysis, and final verification, achieving a balance between detection efficiency and accuracy.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of a hierarchical decision-making method for identifying violation scenarios provided in an embodiment of the present invention.

[0027] Figure 2 A flowchart of another hierarchical decision-making method for identifying violation scenarios provided in an embodiment of the present invention.

[0028] Figure 3A flowchart of a preferred hierarchical decision-making method for identifying violation scenarios provided in an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram of a hierarchical decision-making violation scene recognition device provided in an embodiment of the present invention.

[0030] Figure 5 A schematic diagram of the structure of an electronic device for implementing the hierarchical decision-making violation scene identification method of this invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Figure 1 This is a flowchart illustrating a hierarchical decision-making method for identifying violation scenarios, provided by an embodiment of the present invention. This embodiment is applicable to scenarios requiring real-time violation identification of audio, video, and multimodal content, such as financial anti-fraud, content security auditing, and online live streaming supervision. The method can be executed by a hierarchical decision-making violation scenario identification device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method specifically includes the following steps:

[0034] S110. Acquire image and audio data of the scene to be identified.

[0035] The scenarios to be identified can be those requiring violation identification, such as financial anti-fraud, content security review, and online live streaming supervision.

[0036] S120. Based on the first-level decision model, the audio data is processed to obtain the first-level decision result corresponding to the scene to be identified.

[0037] The first-level decision model includes an audio event detection model, a speech recognition model, and a keyword hierarchical screening module. The first-level decision model is used to initially determine whether a scene violates regulations, and the corresponding judgment result is the first-level decision result.

[0038] Specifically, image and audio data of the scene to be identified can be obtained. To improve the efficiency of recognition, a first-level decision model can be used to process the audio data and obtain the first-level decision result.

[0039] In some possible implementations, processing audio data based on a first-level decision model to obtain the first-level decision result corresponding to the scene to be identified may include: inputting audio data into a pre-trained audio event detection model to obtain the non-speech event detection result corresponding to the audio data; inputting audio data into a speech recognition model to obtain the text data corresponding to the audio data, and matching the text data with a keyword database through a keyword hierarchical filtering module to obtain the keyword matching result; and determining the first-level decision result based on the non-speech event detection result and the keyword matching result.

[0040] The audio event detection model can be a pre-trained model used to detect non-speech sound events in the audio that are highly relevant to the illegal activity. Preferably, the audio event detection model is a music recognition model (YAMNet) or a custom convolutional neural network. The speech recognition model can be a model that converts speech in the audio data into text data; the keyword library can be a set of keywords related to the target illegal scenario, such as gambling-related keywords like "betting" and "chips".

[0041] Specifically, audio data can be input into a pre-trained audio event detection model to obtain non-speech event detection results. Alternatively, audio data can be input into a speech recognition model to obtain corresponding text data. Then, a keyword hierarchical filtering module matches the text data with a keyword database to obtain keyword matching results. Further, based on the non-speech event detection results and keyword matching results, a primary decision is determined. The advantage of this approach is that it allows for a preliminary assessment from both non-speech and speech-text keyword perspectives, quickly identifying potential violation scenarios.

[0042] In some embodiments, when identifying violations in gambling scenarios, a pre-trained audio event detection model can be used to detect non-voice sound events highly related to gambling activities in real time from the audio data stream, such as the sound of shuffling mahjong tiles, the sound of chips colliding, the sound of dice shaking, and the sound of cards being dealt at the card table.

[0043] Furthermore, the input audio stream is converted into text using automatic speech recognition technology, and a three-tiered keyword database related to gambling is established for matching. For example, high-risk words include "betting" and "all-in"; medium-risk words include "playing cards" and "winning or losing"; and low-risk words include "entertainment" and "gathering". If a high-confidence gambling-related audio event is detected, for example, a non-voice sound event with a confidence level > 0.8, and high-risk gambling keywords are identified, the scenario is immediately determined to be gambling and blocked, terminating the process. In other words, the first-level decision result is "hit," and the scenario to be identified is determined to be the target violation scenario.

[0044] S130. If the first-level decision result is a miss or the violation feature is not obvious, the image data and audio data are processed based on the second-level decision model to obtain the second-level decision result corresponding to the scene to be identified.

[0045] The secondary decision model includes a text feature extraction model and an image feature extraction model. A primary decision result of "miss" or "insignificant violation features" may mean that the primary decision model failed to effectively determine whether the scene to be identified is a violation scene, or that the violation features extracted by the primary decision model from the audio data are not significant enough to determine whether the scene to be identified is a violation scene. The secondary decision result can refer to the violation determination result of the secondary decision model for the scene to be identified.

[0046] Specifically, if the first-level decision result is a miss or the violation characteristics are not obvious, the text feature extraction model and image feature extraction model in the second-level decision model can be used to process the image data and audio data respectively to determine whether the scene to be identified is a violation scene, thus obtaining the second-level decision result. If the second-level decision result is a hit, the scene to be identified can be determined as the target violation scene.

[0047] S140. If the secondary decision result is a miss or the violation feature is not obvious, input the image data, audio data, primary decision result and secondary decision result into the tertiary decision model to obtain the tertiary decision result corresponding to the scene to be identified.

[0048] The three-level decision model includes a multimodal large language model, which is used to make a final judgment by integrating information from multiple aspects; the three-level decision result can be that the scene to be identified is a target violation scene, or that the scene to be identified is not a violation scene.

[0049] Understandably, if the secondary decision result is also a miss or the violation characteristics are not obvious—for example, if the secondary decision model fails to extract effective features from the image and audio data, or if the extracted features are insufficient to determine whether the scene to be identified is a violation—then the image data, audio data, primary decision result, and secondary decision result can be input into the tertiary decision model. The reasoning capabilities of the multimodal large language model can then be used to obtain the tertiary decision result.

[0050] S150. Determine whether the scene to be identified is a target violation scene based on the three-level decision results.

[0051] Among them, the target violation scenario refers to a predefined scenario of violation type that needs to be identified, such as a gambling scenario;

[0052] Specifically, the three-level decision-making results can determine whether the scene to be identified is a target violation scene. The advantages of this approach are: a lightweight first layer combined with an efficient second-layer model can handle most requests, resulting in a short average response time and a significant reduction in the number of large model calls; through multimodal cross-validation and fine-grained feature extraction and fusion of the second-layer decision model, the accuracy of violation scene identification is high, the false positive rate is low, and the precise interception capability is enhanced; the hierarchical strategy ensures that computing resources are allocated on demand, requiring only ordinary computing resources for deployment, resulting in low hardware costs and significant resource optimization; the multi-level verification mechanism effectively improves the system's judgment ability and anti-interference capability in complex environments, enhancing system robustness.

[0053] The technical solution of this invention involves acquiring image and audio data of the scene to be identified; processing the audio data based on a first-level decision model to obtain a first-level decision result corresponding to the scene to be identified; if the first-level decision result is a miss or the violation characteristics are not obvious, processing the image and audio data based on a second-level decision model to obtain a second-level decision result corresponding to the scene to be identified; if the second-level decision result is a miss or the violation characteristics are not obvious, inputting the image data, audio data, first-level decision result, and second-level decision result into a third-level decision model to obtain a third-level decision result corresponding to the scene to be identified; and determining whether the scene to be identified is a target violation scene based on the third-level decision result. This technical solution solves the problems of single-modal limitations, resource and efficiency bottlenecks, and rigid and unadaptable strategies in existing violation scene identification technologies. This invention can quickly screen using lightweight audio events and keywords, and then combine them with a mid-level dual-tower model to handle difficult cases. It only calls the large model for highly uncertain scenes, significantly reducing resource consumption and latency, and constructing an efficient hierarchical detection system. By designing cross-validation rules for multiple modalities such as sound, text, and images, the reliability of system judgment is improved, achieving multi-modal collaboration and conflict resolution. It also introduces a dynamic resource allocation mechanism to automatically adjust the detection strategy according to the system load, achieving an intelligent balance between real-time performance and accuracy.

[0054] Figure 2 This is a flowchart of another hierarchical decision-making method for identifying violation scenarios provided by an embodiment of the present invention. Based on the above embodiments, this embodiment further optimizes the decision-making process of the two-level and three-level decision-making models. Specific implementation methods can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 2As shown, the method specifically includes the following steps:

[0055] S210. Acquire image data and audio data of the scene to be identified.

[0056] S220. Based on the first-level decision model, the audio data is processed to obtain the first-level decision result corresponding to the scene to be identified.

[0057] S230. When the first-level decision result is a miss or the violation feature is not obvious, determine the text to be processed corresponding to the audio data, perform hierarchical encoding on the text to be processed based on the text feature extraction model, and output the text feature vector corresponding to the text to be processed through the pooling layer.

[0058] S240. Based on the image feature extraction model, perform layer-by-layer feature extraction on the image data, and obtain the image feature vector corresponding to the image data through global average pooling.

[0059] S250. The text feature vector and the image feature vector are fused to obtain a fused feature vector, and the secondary decision result is determined based on the fused feature vector.

[0060] The text to be processed can be the text obtained from the audio data through speech recognition that requires feature extraction; preferably, in order to improve the recognition efficiency of violation identification, the text data corresponding to the audio data output by the speech recognition model can be directly used as the text to be processed, without the need for the secondary decision model to perform the speech recognition process again.

[0061] Text feature extraction models can be based on the Transformer Encoder architecture, which encodes text hierarchically and outputs fixed-dimensional text feature vectors through pooling layers, such as BERT-base; image feature extraction models use ResNet50 deep convolutional networks or lightweight improved models like MobileNetV3, which can extract features from images layer by layer and output image feature vectors through global average pooling.

[0062] Specifically, a text feature extraction model can be used to hierarchically encode the text to be processed, and then a pooling layer can be used to output the corresponding text feature vector. Simultaneously, an image feature extraction model can be used to extract features from the image data layer by layer, and then global average pooling can be used to output the corresponding image feature vector. Finally, the text feature vector and the image feature vector are fused to obtain a fused feature vector, and the secondary decision result is determined based on this fused feature vector. The advantage of this approach is that extracting and fusing deep features from text and images allows for a more accurate analysis of whether a scene violates regulations.

[0063] For example, a dual-tower model architecture can be constructed, employing a dual-path parallel feature processing architecture, to achieve deep semantic mining and collaborative reasoning of text and image modalities:

[0064] Text Tower: A pre-trained language model based on the Transformer Encoder architecture, such as BERT-base, which performs hierarchical encoding on the complete text transformed by ASR, including contextual semantics, and outputs fixed-dimensional text feature vectors through pooling layers, such as 768-dimensional, to capture deep semantic information such as "gambling slang, veiled expressions, and dialogue intent".

[0065] Image Tower: Employing a ResNet50 deep convolutional network or a lightweight improved model such as MobileNetV3, it extracts features layer by layer from video keyframes or static images, and outputs image feature vectors, such as 2048-dimensional vectors, through global average pooling, focusing on capturing key visual elements such as "card table, chips, dice, cash stacks".

[0066] In some optional embodiments, text feature vectors and image feature vectors are fused to obtain a fused feature vector, and secondary decision results are determined based on the fused feature vector, including: fusing text feature vectors and image feature vectors through attention-weighted fusion or bilinear fusion to obtain a fused feature vector; determining the secondary classification result and secondary classification confidence corresponding to the fused feature vector based on the fused feature vector and a logistic regression model; and determining the secondary decision result based on the secondary classification result and secondary classification confidence.

[0067] Attention-weighted fusion can be understood as a dynamic weight allocation method for text and image feature vectors, enhancing the fusion of key correlation weights. Bilinear fusion can be a method of fusing text and image feature vectors through bilinear transformation. A logistic regression model is used to classify based on the fused feature vectors and output classification confidence. The secondary classification result is a determination of whether the scene is a target violation scene, and the secondary classification confidence is an indicator of the reliability of this determination result.

[0068] Specifically, text feature vectors and image feature vectors are fused using attention-weighted fusion or bilinear fusion to obtain a fused feature vector. Then, based on the fused feature vector and a logistic regression model, the secondary classification result and secondary classification confidence corresponding to the fused feature vector are determined. Finally, the secondary decision result is determined based on the secondary classification result and secondary classification confidence.

[0069] For example, dynamic weight allocation is performed on the text and image feature vectors. For instance, the association weight between the "chip image" and the "betting" text is enhanced to obtain a fused feature vector X=(X1,X2,...,Xn). The probability that the model predicts a positive class result (e.g., a gambling scenario, denoted as y=1) is:

[0070] ;

[0071] Here, w refers to the weight vector, which is learned by the model through training and is used to measure the influence of each dimension in the fused feature X on "whether to gamble"; b is the bias term.

[0072] S260. When the secondary decision result is a no-hit or the violation characteristics are not obvious, construct an evidence chain based on image data, audio data, primary decision result and secondary decision result, and construct prompt words based on the evidence chain;

[0073] The chain of evidence can be understood as an information chain that integrates image data, audio data, primary decision results, and secondary decision results to assist in judgment.

[0074] Specifically, an evidence chain is constructed based on image data, audio data, first-level decision results, and second-level decision results, and then prompt words are constructed based on the evidence chain. These prompt words are then input into a third-level decision model to obtain third-level decision results. The advantage of this approach is that it leverages the understanding capabilities of a multimodal large language model to synthesize information from multiple sources and make a more accurate final judgment.

[0075] S270. Input the prompt words into the three-level decision model to obtain the three-level decision results. Based on the three-level decision results, determine whether the scene to be identified is the target violation scene.

[0076] For example, Figure 3 This is a flowchart illustrating a preferred hierarchical decision-making method for identifying violation scenarios, provided by an embodiment of the present invention. Figure 3 As shown, the detection results from the first two levels can be integrated with the raw data into a structured input to form a large model prompt word, specifically including:

[0077] First-level decision results: audio event type (e.g., the sound of shuffling mahjong tiles) and corresponding confidence level (e.g., 0.65); text keyword matching results (e.g., medium-risk words: win or lose (appeared 3 times), low-risk words: entertainment (appeared 1 time));

[0078] Secondary decision results: binary classification output of the dual-tower model (e.g., in a suspected gambling scenario, confidence level 0.68) and baseline model recognition accuracy (e.g., historical accuracy 82%).

[0079] Raw multimodal data: keyframe images (such as frames in a video containing the card table and chips), and complete dialogue text converted from ASR (with timestamps).

[0080] Organize the above information into prompts (e.g., "Please determine if this is a gambling scenario? Evidence: 1. Audio detected mahjong sound (confidence 0.65); 2. Text contains medium-risk words related to winning and losing; 3. Image shows 4 people sitting around a table; 4. Two-tower model determines confidence 0.68... Please combine cross-modal information for reasoning, prioritizing contradictions"), and input this information into a multimodal large model. After performing cross-modal reasoning based on the complete evidence chain, the large model outputs:

[0081] Final judgment result (whether it is a gambling scenario) and overall confidence level (e.g., 0.92);

[0082] Key evidence annotation: Select the core judgment area in the image (such as a pile of chips or playing cards), annotate the decisive statement in the text (such as "I lost 500 this time"), and associate it with the time point of the audio event (such as "high frequency chip collision sound appears at 00:02:15").

[0083] Explanation of reasoning basis: The judgment logic is explained using natural language (e.g., although the word 'entertainment' appears, combined with the high-frequency sound of chips, the statement of losing 500, and the cash arrangement in the image, it is determined to be a gambling scene), so that the results are traceable and explainable.

[0084] The technical solution of this invention firstly constructs a primary decision model using audio event detection and speech recognition. This model can quickly perform preliminary screening of audio data in a scene, identifying potential non-voice events related to violations. Simultaneously, it converts speech into text and matches it with a keyword database. This effectively solves the problems of existing pure text keyword filtering being easily bypassed by coded language, homophones, and secret codes, and the difficulty in distinguishing background noise from gambling sounds in pure audio analysis. This provides a preliminary and efficient basis for subsequent decision-making, representing a crucial step forward in detection efficiency. The lightweight primary model can handle most routine requests, significantly reducing the number of calls to subsequent large models and shortening the average response time. Then, a secondary decision model composed of a text feature extraction model and an image feature extraction model is used. The text converted from audio is hierarchically encoded and pooled to obtain text feature vectors. Image data is extracted layer by layer and globally averaged to obtain image feature vectors. Finally, the two are fused using attention weighting or bilinear fusion methods. This technology overcomes the single-modal limitation of pure image detection in recognizing voice-based gambling behavior. Through multimodal cross-validation and fine-grained feature extraction and fusion, it significantly improves the accuracy of gambling scene recognition, reduces the false positive rate, and further enhances the ability to accurately intercept. Finally, a three-level decision model is constructed using a multimodal large language model. When the results of the first and second levels of decision are not clear enough, image data, audio data, and the results of the first two levels of decision are integrated to construct an evidence chain and prompt words, which are then input into the large model for comprehensive judgment. This solves the resource and efficiency bottlenecks caused by directly calling a large multimodal model to process all data, such as high response latency and high computational cost, as well as the problems of high false interception rate caused by fixed threshold judgment mechanism and lack of multimodal evidence collaboration and conflict handling mechanism.

[0085] In this embodiment of the invention, the hierarchical strategy ensures that computing resources are allocated on demand, requiring only ordinary computing resources for deployment, thus reducing hardware costs. At the same time, the multi-level verification mechanism effectively improves the system's judgment and anti-interference capabilities in complex environments, enhances system robustness, and enables it to better adapt to complex and ever-changing real-world scenarios, avoiding misjudgments in normal scenarios such as family mahjong entertainment.

[0086] Figure 4 This is a schematic diagram of a hierarchical decision-making violation scene recognition device provided in an embodiment of the present invention. Figure 4 As shown, the device includes:

[0087] Data acquisition module 410 is used to acquire image data and audio data of the scene to be identified;

[0088] The first-level decision module 420 is used to process audio data based on the first-level decision model to obtain the first-level decision result corresponding to the scene to be identified. The first-level decision model includes an audio event detection model, a speech recognition model, and a keyword hierarchical filtering module.

[0089] The secondary decision module 430 is used to process the image data and audio data based on the secondary decision model when the primary decision result is a miss or the violation feature is not obvious, to obtain the secondary decision result corresponding to the scene to be identified. The secondary decision model includes a text feature extraction model and an image feature extraction model.

[0090] The third-level decision module 440 is used to input the image data, the audio data, the first-level decision result and the second-level decision result into the third-level decision model when the second-level decision result is a miss or the violation feature is not obvious, so as to obtain the third-level decision result corresponding to the scene to be identified. The third-level decision model includes a multimodal large language model.

[0091] The scene judgment module 450 is used to determine whether the scene to be identified is a target violation scene based on the three-level decision results.

[0092] In some alternative embodiments, the device is further configured to:

[0093] If the first-level decision result is a hit, the scene to be identified is determined to be the target violation scene;

[0094] Alternatively, if the secondary decision result indicates a hit, the scene to be identified is determined to be a target violation scene.

[0095] In some optional embodiments, the first-level decision module 420 is specifically used for:

[0096] The audio data is input into the pre-trained audio event detection model to obtain the non-speech event detection result corresponding to the audio data;

[0097] The audio data is input into the speech recognition model to obtain the text data corresponding to the audio data, and the text data is matched with the keyword database through the keyword hierarchical filtering module to obtain the keyword matching result.

[0098] Based on the non-voice event detection results and the keyword matching results, the first-level decision result is determined.

[0099] In some alternative embodiments, the secondary decision module 430 includes:

[0100] The text feature extraction submodule is used to determine the text to be processed corresponding to the audio data, perform hierarchical encoding on the text to be processed based on the text feature extraction model, and output the text feature vector corresponding to the text to be processed through a pooling layer.

[0101] The image feature extraction submodule is used to extract features from the image data layer by layer based on the image feature extraction model, and to obtain the image feature vector corresponding to the image data through global average pooling.

[0102] The feature fusion submodule is used to fuse the text feature vector and the image feature vector to obtain a fused feature vector, and to determine the secondary decision result based on the fused feature vector.

[0103] In some optional embodiments, the feature fusion submodule is specifically used for:

[0104] The text feature vector and the image feature vector are fused by attention-weighted fusion or bilinear fusion to obtain a fused feature vector;

[0105] Based on the fused feature vector and the logistic regression model, the secondary classification result and the secondary classification confidence level corresponding to the fused feature vector are determined, and the secondary decision result is determined based on the secondary classification result and the secondary classification confidence level.

[0106] In some optional embodiments, the three-level decision module 440 is specifically used for:

[0107] An evidence chain is constructed based on the image data, the audio data, the first-level decision result, and the second-level decision result, and a prompt word is constructed based on the evidence chain;

[0108] The prompt words are input into the three-level decision model to obtain the three-level decision results.

[0109] The hierarchical decision-making violation scene identification device provided in this embodiment of the invention can execute the hierarchical decision-making violation scene identification method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0110] Figure 5 This is a schematic diagram of the structure of an electronic device for implementing the hierarchical decision-making violation scene identification method of this invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0111] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0112] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0113] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a violation scene recognition method for hierarchical decision-making.

[0114] In some embodiments, the hierarchical decision-making violation scene identification method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the hierarchical decision-making violation scene identification method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the hierarchical decision-making violation scene identification method by any other suitable means (e.g., by means of firmware).

[0115] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0117] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0119] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0120] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0121] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0122] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A hierarchical decision-making method for identifying violation scenarios, characterized in that, include: Acquire image and audio data of the scene to be identified; The audio data is processed based on a first-level decision model to obtain the first-level decision result corresponding to the scene to be identified. The first-level decision model includes an audio event detection model, a speech recognition model, and a keyword hierarchical screening module. If the first-level decision result is a miss or the violation feature is not obvious, the image data and audio data are processed based on the second-level decision model to obtain the second-level decision result corresponding to the scene to be identified. The second-level decision model includes a text feature extraction model and an image feature extraction model. If the secondary decision result is a miss or the violation feature is not obvious, the image data, the audio data, the primary decision result and the secondary decision result are input into the tertiary decision model to obtain the tertiary decision result corresponding to the scene to be identified. The tertiary decision model includes a multimodal large language model. Based on the results of the three-level decision-making process, it is determined whether the scene to be identified is a target violation scene; The process of processing the audio data based on the first-level decision model to obtain the first-level decision result corresponding to the scene to be identified includes: The audio data is input into the pre-trained audio event detection model to obtain the non-speech event detection result corresponding to the audio data; The audio data is input into the speech recognition model to obtain the text data corresponding to the audio data, and the text data is matched with the keyword database through the keyword hierarchical filtering module to obtain the keyword matching result. Based on the non-voice event detection results and the keyword matching results, the first-level decision result is determined; The process of processing the image and audio data based on the two-level decision model to obtain the two-level decision result corresponding to the scene to be identified includes: The text to be processed corresponding to the audio data is determined, the text to be processed is hierarchically encoded based on the text feature extraction model, and the text feature vector corresponding to the text to be processed is output through the pooling layer. Based on the image feature extraction model, the image data is extracted layer by layer, and the image feature vector corresponding to the image data is obtained by global average pooling. The text feature vector and the image feature vector are fused to obtain a fused feature vector, and the secondary decision result is determined based on the fused feature vector; The step of inputting the image data, the audio data, the first-level decision result, and the second-level decision result into the third-level decision model to obtain the third-level decision result corresponding to the scene to be identified includes: An evidence chain is constructed based on the image data, the audio data, the first-level decision result, and the second-level decision result, and a prompt word is constructed based on the evidence chain; The prompt words are input into the three-level decision model to obtain the three-level decision results.

2. The method according to claim 1, characterized in that, The method further includes: If the first-level decision result is a hit, the scene to be identified is determined to be the target violation scene; Alternatively, if the secondary decision result indicates a hit, the scene to be identified is determined to be a target violation scene.

3. The method according to claim 1, characterized in that, The step of fusing the text feature vector and the image feature vector to obtain a fused feature vector, and determining the secondary decision result based on the fused feature vector, includes: The text feature vector and the image feature vector are fused by attention-weighted fusion or bilinear fusion to obtain a fused feature vector; Based on the fused feature vector and the logistic regression model, the secondary classification result and the secondary classification confidence level corresponding to the fused feature vector are determined, and the secondary decision result is determined based on the secondary classification result and the secondary classification confidence level.

4. A hierarchical decision-making device for identifying violation scenarios, characterized in that, include: The data acquisition module is used to acquire image and audio data of the scene to be identified; The first-level decision module is used to process the audio data based on the first-level decision model to obtain the first-level decision result corresponding to the scene to be identified. The first-level decision model includes an audio event detection model, a speech recognition model, and a keyword hierarchical filtering module. The secondary decision module is used to process the image data and audio data based on the secondary decision model when the primary decision result is a miss or the violation feature is not obvious, so as to obtain the secondary decision result corresponding to the scene to be identified. The secondary decision model includes a text feature extraction model and an image feature extraction model. The third-level decision module is used to input the image data, the audio data, the first-level decision result, and the second-level decision result into the third-level decision model when the second-level decision result is a miss or the violation feature is not obvious, so as to obtain the third-level decision result corresponding to the scene to be identified. The third-level decision model includes a multimodal large language model. The scene judgment module is used to determine whether the scene to be identified is a target violation scene based on the three-level decision results; The primary decision-making module is specifically used for: The audio data is input into the pre-trained audio event detection model to obtain the non-speech event detection result corresponding to the audio data; The audio data is input into the speech recognition model to obtain the text data corresponding to the audio data, and the text data is matched with the keyword database through the keyword hierarchical filtering module to obtain the keyword matching result. Based on the non-voice event detection results and the keyword matching results, the first-level decision result is determined; The secondary decision-making module includes: The text feature extraction submodule is used to determine the text to be processed corresponding to the audio data, perform hierarchical encoding on the text to be processed based on the text feature extraction model, and output the text feature vector corresponding to the text to be processed through a pooling layer. The image feature extraction submodule is used to extract features from the image data layer by layer based on the image feature extraction model, and to obtain the image feature vector corresponding to the image data through global average pooling. The feature fusion submodule is used to fuse the text feature vector and the image feature vector to obtain a fused feature vector, and to determine the secondary decision result based on the fused feature vector; The three-level decision-making module is specifically used for: An evidence chain is constructed based on the image data, the audio data, the first-level decision result, and the second-level decision result, and a prompt word is constructed based on the evidence chain; The prompt words are input into the three-level decision model to obtain the three-level decision results.

5. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the violation scene identification method of any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the violation scene identification method of any one of claims 1-3 for hierarchical decision-making.

7. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the violation scene identification method for hierarchical decision-making according to any one of claims 1-3.

Images