Method and device for identifying downhole violations, electronic device and readable medium

Abstract

The application relates to a downhole violation behavior identification method and device, electronic equipment and readable medium, wherein the method comprises the following steps: acquiring a monitoring image of a downhole operation scene to be identified; performing feature extraction on the monitoring image to obtain an image feature vector, and performing scene identification on the monitoring image to generate a scene feature vector; fusing the image feature vector and the scene feature vector to generate an image representation vector; acquiring a pre-constructed violation behavior description set, and performing semantic-level feature extraction on each text description in the violation behavior description set to generate a text feature vector; calculating the semantic similarity between the image representation vector and each text feature vector, and determining a behavior identification result matched with the monitoring image according to the semantic similarity. The problem of reduced identification accuracy caused by missing scene semantics or single semantic judgment is solved.

Description

Technical Field

[0001] This application relates to the field of behavior detection technology, and in particular to a method, device, electronic device, and readable medium for identifying illegal behavior in underground mines. Background Technology

[0002] The underground working environment in coal mines is complex, with diverse work scenarios and overlapping personnel operations and equipment operating status. Violations are characterized by their high degree of concealment, high frequency of occurrence, and serious risks and consequences. To ensure safety in underground operations, monitoring equipment is typically deployed in key working areas to collect real-time video of the work process and to identify and issue warnings for violations through manual or intelligent methods.

[0003] Existing underground violation recognition technologies mostly employ rule-based or classification-based methods to detect and judge predefined violation types. These technologies typically require separate model training for different operational scenarios and often rely on fixed sets of violation labels, making them ill-suited to the demands of frequently changing underground operational scenarios and diverse violation manifestations. Furthermore, while some methods incorporate the correlation between image features and textual descriptions, they often focus on single semantic judgments of image content, failing to fully consider the impact of the operational scenario on the semantic understanding of violations. This can easily lead to inaccurate or inconsistent recognition results for the same behavior in different scenarios.

[0004] Therefore, it is urgent to solve the problem of reduced recognition accuracy in existing technologies due to missing scene semantics or only performing single semantic judgments.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a method, device, electronic device, and readable medium for identifying illegal activities in underground mines, in order to solve the aforementioned technical problem of "reduced recognition accuracy due to missing scene semantics or only performing single semantic judgment".

[0007] According to one aspect of the embodiments of this application, this application provides a method for identifying underground violations, including: acquiring a monitoring image of an underground operation scene to be identified; extracting features from the monitoring image to obtain an image feature vector, and performing scene recognition on the monitoring image to generate a scene feature vector; fusing the image feature vector and the scene feature vector to generate an image representation vector; acquiring a pre-constructed set of violation descriptions, and extracting semantic features from each text description in the set of violation descriptions to generate a text feature vector; calculating the semantic similarity between the image representation vector and each text feature vector, and determining the behavior recognition result matching the monitoring image based on the semantic similarity.

[0008] Optionally, scene recognition is performed on the surveillance image to generate a scene feature vector, including: inputting the surveillance image into a preset scene classification model; determining the scene category to which the surveillance image belongs based on the output of the preset scene classification model; and encoding the scene category into an embeddable vector to obtain the scene feature vector.

[0009] Optionally, the image feature vector and the scene feature vector are fused to generate an image representation vector, including: inputting the scene feature vector into a mapping network and mapping it to the semantic feature space to which the image feature vector belongs, to obtain an aligned scene feature vector; fusing the image feature vector and the scene feature vector to generate a fused feature vector; and normalizing the fused feature vector to obtain an image representation vector.

[0010] Optionally, before obtaining the pre-constructed set of violation descriptions, the method further includes constructing the set of violation descriptions in the following manner: for the target job scenario to be trained, the job type and job environment range of the target job scenario are used as limiting conditions; under the limiting conditions, each violation description and at least one non-violation description in the target job scenario are enumerated and combined to obtain multiple first violation descriptions; semantic text annotations are generated for each first violation description, and the first violation descriptions and semantic text annotations are stored in a structured form, wherein the semantic text annotations include scene labels, violation subjects, and violation action information; second violation descriptions that require subjective judgment are selected from each first violation description, and the second violation descriptions are subjected to fine-grained semantic division to obtain third violation descriptions; the first violation descriptions and third violation descriptions are integrated to obtain a set of violation descriptions.

[0011] Optionally, the semantic similarity between the image representation vector and each text feature vector is calculated, including calculating the semantic similarity between the image representation vector and the current text feature vector in the following manner: performing semantic parsing on the current text feature vector to determine whether the current text feature vector belongs to a fine-grained violation category; if the current text feature vector does not belong to a fine-grained violation category, then calculating the overall similarity between the image representation vector and the current text feature vector; and using the overall similarity as the semantic similarity between the image representation vector and the current text feature vector.

[0012] Optionally, after determining whether the current text feature vector belongs to the fine-grained violation category, the method further includes: if the current text feature vector belongs to the fine-grained violation category, extracting local region features corresponding to the current text description from the monitoring image; calculating the local similarity between the local region features and the current text feature vector; calculating the overall similarity between the image representation vector and the current text feature vector; and weightedly fusing the overall similarity and local similarity to obtain the semantic similarity.

[0013] Optionally, the method further includes dynamically updating the set of violation descriptions in the following manner: establishing a semantic knowledge graph corresponding to underground violations, wherein the semantic knowledge graph includes multiple category nodes, violation behavior nodes, and semantic relationships between nodes; when a new violation description appears, retrieving at least one existing violation behavior node matching the new violation behavior description from the semantic knowledge graph; generating a new text feature vector corresponding to the new violation behavior description through transfer learning based on the existing text feature vectors corresponding to the existing violation behavior nodes; obtaining feedback data corresponding to the new violation behavior description and the new text feature vector, and optimizing the new violation behavior description and the new text feature vector based on the feedback data; adding the optimized new violation behavior description and the new text feature vector to the set of violation descriptions to complete the dynamic update of the set of violation descriptions.

[0014] According to another aspect of the embodiments of this application, this application provides an identification device for underground violations, comprising: an acquisition module for acquiring a monitoring image of an underground operation scene to be identified; a first extraction module for extracting features from the monitoring image to obtain an image feature vector, and performing scene recognition on the monitoring image to generate a scene feature vector; a fusion module for fusing the image feature vector and the scene feature vector to generate an image representation vector; a second extraction module for acquiring a pre-constructed set of violation descriptions, and performing semantic-level feature extraction on each text description in the set of violation descriptions to generate a text feature vector; and a calculation module for calculating the semantic similarity between the image representation vector and each text feature vector, and determining the behavior identification result matching the monitoring image based on the semantic similarity.

[0015] According to another aspect of the embodiments of this application, this application provides an electronic device, including a memory, a processor, a communication interface and a communication bus. The memory stores a computer program that can run on the processor. The memory and the processor communicate with each other through the communication bus and the communication interface. When the processor executes the computer program, it implements the steps of the above method.

[0016] According to another aspect of the embodiments of this application, this application also provides a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the above-described method.

[0017] Compared with related technologies, the technical solutions provided in this application have the following advantages: This application provides a method for identifying underground violations, comprising: acquiring a monitoring image of the underground work scene to be identified; extracting features from the monitoring image to obtain an image feature vector, and performing scene recognition on the monitoring image to generate a scene feature vector; fusing the image feature vector and the scene feature vector to generate an image representation vector; acquiring a pre-constructed set of violation descriptions, and extracting semantic features from each text description in the set to generate a text feature vector; calculating the semantic similarity between the image representation vector and each text feature vector, and determining the behavior identification result matching the monitoring image based on the semantic similarity. By explicitly introducing the semantics of the work scene into the violation identification process, the same image behavior can obtain differentiated semantic understanding in different work scenes, and the violation determination is completed by combining the semantic matching of image and text, thereby improving the accuracy and adaptability of underground violation identification and solving the problem of reduced identification accuracy due to missing scene semantics or only performing single semantic judgment. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the hardware environment for an optional method for identifying downhole violations according to an embodiment of this application; Figure 2 A flowchart illustrating an optional method for identifying downhole violations according to an embodiment of this application; Figure 3 This is a flowchart illustrating the training process of an optional image-text matching model provided according to an embodiment of this application. Figure 4 This is a flowchart illustrating an optional reasoning process for identifying downhole violations according to an embodiment of this application. Figure 5 This is a block diagram of an optional downhole violation identification device provided according to an embodiment of this application; Figure 6 This is a schematic diagram of an optional electronic device structure provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustration and has no specific meaning in itself. Therefore, "module" and "part" may be used interchangeably.

[0023] The underground working environment in coal mines is complex, with diverse work scenarios and overlapping personnel operations and equipment operating status. Violations are characterized by their high degree of concealment, high frequency of occurrence, and serious risks and consequences. To ensure safety in underground operations, monitoring equipment is typically deployed in key working areas to collect real-time video of the work process and to identify and issue warnings for violations through manual or intelligent methods.

[0024] Existing underground violation recognition technologies mostly employ rule-based or classification-based methods to detect and judge predefined violation types. These technologies typically require separate model training for different operational scenarios and often rely on fixed sets of violation labels, making them ill-suited to the demands of frequently changing underground operational scenarios and diverse violation manifestations. Furthermore, while some methods incorporate the correlation between image features and textual descriptions, they often focus on single semantic judgments of image content, failing to fully consider the impact of the operational scenario on the semantic understanding of violations. This can easily lead to inaccurate or inconsistent recognition results for the same behavior in different scenarios.

[0025] Therefore, it is urgent to solve the problem of reduced recognition accuracy in existing technologies due to missing scene semantics or only performing single semantic judgments.

[0026] To address the problems mentioned in the background art, according to one aspect of the embodiments of this application, an embodiment of a method for identifying downhole violations is provided.

[0027] Optionally, in the embodiments of this application, the above-described method for identifying downhole violations can be applied to, for example... Figure 1 The hardware environment shown consists of terminal 101 and server 103. Figure 1As shown, server 103 is connected to terminal 101 via a network and can be used to provide services to the terminal or clients installed on the terminal. Database 105 can be set up on the server or independently of the server to provide data storage services for server 103. The network mentioned above includes, but is not limited to, wide area network, metropolitan area network or local area network. Terminal 101 includes, but is not limited to, PC, mobile phone, tablet computer, etc.

[0028] The method for identifying downhole violations in this application embodiment can be executed by server 103, or it can be jointly executed by server 103 and terminal 101, such as... Figure 2 As shown, it includes: Step 201: Obtain monitoring images of the downhole operation scene to be identified; Step 202: Extract features from the monitoring image to obtain image feature vectors, and perform scene recognition on the monitoring image to generate scene feature vectors; Step 203: Fuse the image feature vector with the scene feature vector to generate an image representation vector; Step 204: Obtain a pre-constructed set of descriptions of violations, and extract semantic features from each text description in the set of descriptions of violations to generate a text feature vector; Step 205: Calculate the semantic similarity between the image representation vector and each text feature vector, and determine the behavior recognition result that matches the monitoring image based on the semantic similarity.

[0029] This application can be applied to industrial safety production fields where there are strict requirements for the operation environment and employee behavior and high-risk control, and is especially suitable for non-standardized, complex and changeable on-site environments.

[0030] The system acquires monitoring images of a designated area (such as the work area) collected by monitoring equipment. By extracting features from the monitoring images and combining them with scene recognition results, the image features and scene features are fused to generate an image representation vector containing semantic information of the work scene. Semantic features are extracted from the descriptions of violations in a pre-constructed set of violation descriptions to generate text feature vectors that are in the same semantic space as the image representation vector. By calculating the semantic similarity between the image representation vector and each text feature vector, semantic matching and recognition of underground violations are achieved.

[0031] As an optional embodiment, scene recognition is performed on the surveillance image to generate a scene feature vector, including: inputting the surveillance image into a preset scene classification model; determining the scene category to which the surveillance image belongs based on the output of the preset scene classification model; and encoding the scene category into an embeddable vector to obtain the scene feature vector.

[0032] Adjust the surveillance image to be identified (e.g., resolution 1920*1080) to the input size required by the preset scene classification model (e.g., 224*224), and perform normalization and other standardization processing.

[0033] The preprocessed monitoring image is input into a pre-trained preset scene classification model. The preset scene classification model provided in this application is a lightweight convolutional neural network (CNN) or vision transformer (ViT) model, specifically designed to identify typical underground operation scenarios, such as "coal mining," "tunneling," "drilling rig operation," and "maintenance." The model output is the probability distribution of the image belonging to each scene category.

[0034] From the output probability distribution of the scene classification model, the category with the highest probability value is selected as the scene category to which the monitoring image belongs. For example, if the model determines that the current monitoring image has a 90% probability of belonging to the "drilling rig operation" scene, then the scene category is determined to be "drilling rig operation".

[0035] Discrete category labels are converted into continuous numerical vectors that can be fused by the network. Specifically, the determined scene categories are encoded using a learnable embedding lookup table. This lookup table is essentially a matrix, where each row corresponds to a predefined scene category, and each row is a fixed-length vector (e.g., 128-dimensional or 256-dimensional). By querying the row containing the corresponding category, a scene feature vector containing scene semantics can be obtained.

[0036] Since simple category labels (such as "drilling rig operation") cannot be directly utilized by neural networks, this application uses embedding encoding to map categories into an optimizable feature vector. During training, this vector is jointly optimized with the entire image-text matching task, enabling the model to learn that when the scene feature vector of "drilling rig operation" appears, the patterns related to "entering the rotating region" in the image features should be given higher weights, while the weights of patterns related to "not wearing a safety helmet" remain relatively fixed. This achieves dynamic modulation of scene information on visual feature perception.

[0037] As an optional embodiment, the image feature vector and the scene feature vector are fused to generate an image representation vector, including: inputting the scene feature vector into a mapping network and mapping it to the semantic feature space to which the image feature vector belongs, to obtain an aligned scene feature vector; fusing the image feature vector and the scene feature vector to generate a fused feature vector; and normalizing the fused feature vector to obtain an image representation vector.

[0038] To accurately identify violations in underground mining operations, image feature vectors extracted from monitoring images are fused with scene feature vectors representing the semantics of the work scenario to generate an image representation vector that simultaneously reflects both image content information and work scenario semantic information.

[0039] The generated scene feature vectors are input into a pre-defined mapping network, which maps them to a feature space consistent with the semantic feature space of the image feature vectors, resulting in aligned scene feature vectors. The mapping network eliminates the differences in feature distribution and semantic representation between scene feature vectors and image feature vectors, enabling effective fusion of the two types of features within the same semantic space.

[0040] After obtaining the aligned scene feature vector, the image feature vector and the scene feature vector are fused to generate a fused feature vector. Feature fusion combines the visual information contained in the image content with the semantic information corresponding to the work scene, so that the fused feature vector has both image-level detailed features and scene-level prior information, thereby more comprehensively representing the actual work status reflected by the monitoring image.

[0041] After generating the fused feature vector, the fused feature vector is normalized to obtain the final image representation vector. Normalization ensures that the image representation vector has uniform scale characteristics in the vector space, facilitating subsequent semantic similarity calculations with the feature vectors of the violation description text, thus guaranteeing the stability and comparability of similarity calculation results between different samples.

[0042] By setting up a mapping network to map scene feature vectors to the semantic feature space of image feature vectors, semantic alignment between image features and scene features is achieved. Based on this semantic alignment, the two types of features are fused, enabling the generated image representation vector to simultaneously reflect the visual content information of the surveillance image and the corresponding semantic information of the operational scene.

[0043] For example, this application first uses a lightweight CNN / ViT to identify scene categories (such as coal mining, tunneling, and drilling operations), then concatenates and fuses the scene feature vectors with image features, and finally performs normalization processing to obtain the image representation vector. This allows the same action to have different risk levels in different scenarios; for example, the semantics of "standing in front of the drilling rig" differs when the drilling rig is stopped and when it is running.

[0044] Since the risk meanings corresponding to the same behavior differ in different downhole operation scenarios, introducing scene semantics into image representation can help improve the accuracy of the model's semantic understanding of violations.

[0045] As an optional embodiment, before obtaining the pre-constructed set of violation descriptions, the method further includes constructing the set of violation descriptions in the following manner: for the target job scenario to be trained, the job type and job environment range of the target job scenario are used as limiting conditions; under the limiting conditions, each violation description and at least one non-violation description in the target job scenario are enumerated and combined to obtain multiple first violation descriptions; semantic text annotations are generated for each first violation description, and the first violation descriptions and semantic text annotations are stored in a structured form, wherein the semantic text annotations include scene tags, violation subjects, and violation action information; second violation descriptions that require subjective judgment are selected from each first violation description, and the second violation descriptions are subjected to fine-grained semantic division to obtain third violation descriptions; the first violation descriptions and third violation descriptions are integrated to obtain a set of violation descriptions.

[0046] For the target operation scenario to be trained, the operation type and operation environment range corresponding to the target operation scenario are used as limiting conditions. The operation type is used to limit the specific category of downhole operation, and the operation environment range is used to limit the reasonable boundaries of personnel, equipment and space distribution under the operation type, so as to avoid introducing descriptions of violations in irrelevant scenarios into the current target operation scenario.

[0047] Under certain conditions, enumerate all possible descriptions of violations and at least one description of non-violations in the target operation scenario, and combine the descriptions of violations and non-violations to obtain multiple descriptions of first violations.

[0048] By introducing non-violation descriptions into the descriptions of violations, the set of violation descriptions can cover both violation and normal states, thus providing the model with a complete discriminative semantic space.

[0049] For each description of the first violation, a corresponding semantic text annotation is generated, and the description of the first violation and the semantic text annotation are stored in a structured form. The semantic text annotation includes at least a scene tag, the violating subject, and information about the violating action, which is used to clearly describe the work scene in which the violation occurred, the object performing the action, and the specific violating action, thereby improving the semantic standardization and consistency of the violation description.

[0050] For example, the semantic text annotation for the first violation is as follows: Scene tag: "tunneling operation"; Violation subject: such as "personnel", "equipment", "environment"; Violation action: such as "not wearing", "entering", "not set". The description, annotation, and possible corresponding image tags are stored in a structured format such as JSON.

[0051] After completing the structured annotation of the first violation descriptions, the second violation descriptions that require human experience or subjective judgment are selected from the various first violation descriptions. The second violation descriptions usually correspond to violation types with differences in degree or ambiguous judgment boundaries, and it is difficult to accurately express their risk level or violation degree by directly using a single description.

[0052] For the selected second violation descriptions, fine-grained semantic segmentation is performed to obtain third violation descriptions. For example, the segmentation is usually based on a "violation category - violation subcategory - violation severity" system. For example, "improper wearing of safety helmet" (second violation description) is segmented into "safety helmet strap not fastened" (third violation description, which can be labeled: subcategory = improper wearing, severity = mild) and "safety helmet severely tilted" (third violation description, which can be labeled: subcategory = improper wearing, severity = moderate); "improper personnel positioning" (second violation description) is segmented into "personnel less than 3 meters from the rear of the tunneling machine" (third violation description, which can be labeled: subcategory = insufficient safe distance, severity = specific value / moderate).

[0053] By performing fine-grained semantic segmentation on the description of the second violation, the same violation can correspond to different semantic descriptions under different degrees of violation or different manifestations, thereby forming a more refined system for expressing violations.

[0054] The initial, relatively coarse-grained description of the first violation is integrated with the more precise description of the third violation obtained after fine-grained segmentation. After deduplication, the final set of violation descriptions for the target operation scenario is formed. This set includes textual descriptions of both objective and subjective violations at different granularities.

[0055] In this embodiment, by setting clear job types and environmental constraints for the target work scenario, it is ensured that the descriptions of violations only cover behaviors that may actually occur in specific scenarios. Based on this, structured semantic text annotation unifies the expression of violation descriptions, and further fine-grained semantic decomposition is performed on violations requiring subjective judgment, thereby constructing a set of violation descriptions with clear semantic hierarchy and controllable granularity.

[0056] The method for constructing a set of violation descriptions provided in this embodiment systematically enumerates, structurally annotates, and performs fine-grained semantic segmentation on the descriptions of violations under specific operational scenario constraints. This results in a set of violation descriptions that is semantically standardized, scenario-specific, and highly scalable. This reduces recognition errors caused by ambiguity or inconsistent granularity in violation descriptions and improves the accuracy and practicality of identifying downhole violations.

[0057] In addition, when constructing the set of violation descriptions, it is also necessary to construct a set of difficult examples with a certain proportion (such as 30%) and add them to the set of violation descriptions. The purpose is to enhance the robustness and stability of the model in real-world complex downhole environments.

[0058] For example, when extracting frames from the underground video stream, samples with the following features are selected, accounting for 30% of the total dataset: abnormal lighting, including underground light obstruction, backlighting, and low illumination (brightness value < 50); severe occlusion, including personnel being obscured by equipment ≥ 50%, and safety helmets being obscured by miner's lamps / clothing; abnormal angles, including camera tilt angle > 60°, elevation angle > 45°, and side view obscuring key parts; motion blur, including frame blur caused by personnel / equipment movement (blurrity > 0.6); and violations by small targets, including the violating subject occupying < 10% of the frame (e.g., personnel in the distance not wearing safety helmets). Difficult sample samples are additionally labeled with "difficulty type" (e.g., "insufficient lighting - low illumination" or "occlusion - equipment obscuring the head"), and the weights of this type of sample are specifically increased during training.

[0059] As an optional embodiment, calculating the semantic similarity between the image representation vector and each text feature vector includes calculating the semantic similarity between the image representation vector and the current text feature vector in the following manner: performing semantic parsing on the current text feature vector to determine whether the current text feature vector belongs to a fine-grained violation category; if the current text feature vector does not belong to a fine-grained violation category, then calculating the overall similarity between the image representation vector and the current text feature vector; and using the overall similarity as the semantic similarity between the image representation vector and the current text feature vector.

[0060] Before calculating the similarity between the image representation vector and the current text feature vector, the current text feature vector or its corresponding original text description is first analyzed at the semantic level to determine whether the violation description corresponding to the current text feature vector belongs to a predefined fine-grained violation category.

[0061] Fine-grained violation categories are a predefined set that typically includes violations identified based on subjective judgment, subtle object states, or precise spatial relationships, such as "helmet strap not fastened," "personnel less than 3 meters from equipment," and "helmet tilted." The parsing process can be based on pre-labeled category tags in a violation knowledge graph, or by using a lightweight text classifier to analyze the features of the current text feature vector.

[0062] If the analysis result determines that it does not belong to the fine-grained violation category, it means that the current text description corresponds to a coarse-grained violation or no violation. For example, "person not wearing a safety helmet" or "no violation". For such descriptions, the violation determination mainly depends on the presence or absence of the target (such as a safety helmet or person) in the image or the overall behavior of entering a specific area, without the need to analyze local details (such as the chin or hat strap) or precise distances.

[0063] Specifically, the cosine similarity between the image representation vector and the current text feature vector is calculated as the overall similarity. The image representation vector is a global image feature representation that incorporates scene priors, while the current text feature vector is the semantic representation of the corresponding overall text. The dot product of the two (after normalization) directly reflects the degree of matching between the overall semantics of the image and the overall description of the text. The overall similarity is directly used as the final semantic similarity between the image representation vector and the current text feature vector.

[0064] For coarse-grained descriptions, their semantics are highly correlated with the overall scene, the presence of the subject, and the approximate actions of the image. In this embodiment, the image representation vector used for calculation is a global vector that has already incorporated scene features, precisely containing this holistic information. Therefore, directly calculating the similarity between the image representation vector and text features is the most direct and effective way to align global visual semantics with overall textual semantics. This avoids introducing irrelevant local feature noise, making the matching process more focused and efficient.

[0065] For most simple and clear violations (such as not wearing a safety helmet), this embodiment bypasses the computationally expensive steps of local feature extraction, human pose estimation, local similarity calculation, and weighted fusion. Since such coarse-grained violation descriptions account for a high proportion in actual underground scenarios, this diversion mechanism can significantly reduce the average computation time required to match a single image with the entire text set, enabling the system to have stronger real-time performance in high-throughput video stream analysis.

[0066] This application, while ensuring high recognition accuracy for core, high-frequency coarse-grained violations, significantly reduces the system's average inference latency and computational resource consumption through intelligent computational path selection, thereby improving the system's overall efficiency and practicality.

[0067] As another optional embodiment, this application provides an image-text matching model, which is trained through dual-tower contrastive learning and is used to calculate the semantic similarity between images and text. Figure 3 The training steps for the image-text matching model provided in this application, as shown in the figure, include the following four parts: 1. Prepare a dataset for "image-violation text" pairing (corresponding to "preparing image-text pairing dataset"), ensuring that each image has a corresponding violation description (e.g., "not wearing a safety helmet"). First, use "lightweight CNN / ViT" to complete scene recognition (e.g., tunneling, coal mining scenes). The principle is to use a lightweight model to extract scene features, providing prior knowledge for subsequently distinguishing "risk differences of the same action in different scenarios" (e.g., the semantic difference of "standing in front of the drilling rig" in drilling rig operating / shutdown scenarios).

[0068] 2. Image feature encoding: Taking “image batch (32, 3, image width and height)” as input, ViT (visual transformer) is used to extract the global features of the image (output (32, 768)), and then compressed into 512-dimensional features through linear projection (output (32, 512)). ViT captures global features through the self-attention of image blocks, adapting to feature extraction of complex downhole images. Text feature encoding: Taking “text batch (32, text length)” as input, the Transformer extracts the semantic features of the violation text (output (32, 768)), and then projects it linearly into 512-dimensional features (output (32, 512)). The Transformer captures the semantic associations of the text through self-attention. Feature alignment: concatenates and compares image features with text features to correct them, so that the image and text features are in the same vector space.

[0069] 3. Calculate the cosine similarity between image features and text features by computing the similarity matrix S(32, 32). Positive sample pairs (image-text matching, such as an image of "not wearing a helmet" corresponding to the text) have high similarity; negative sample pairs (image-text mismatch, such as an image of "not wearing a helmet" corresponding to the text "not fastening the chin strap") have low similarity. This allows the model to learn the pattern that correct image-text pairs have high similarity and incorrect pairs have low similarity.

[0070] 4. When the training iterations are reached or the training set loss function stops decreasing, the model converges. If the conditions are not met, the ViT and Transformer parameters are updated, and the model is optimized through gradient descent to make the image-text matching more accurate. If false alarms are found, the data is corrected and the dataset is updated through manual review and feedback. Specifically, the model bias is corrected by using human experience to form a closed loop of "training-feedback-retraining" to improve the model's practicality. After the criteria are met, the final model is output for inference. At this time, the model can receive new underground images and output the corresponding violation text descriptions.

[0071] The loss function provided in this application is: Where N represents the number of "image-text" pairs in the batch, and S ij S represents the cosine similarity between the i-th image and the j-th text segment. iiLet represent the cosine similarity between the i-th image and the corresponding correct text.

[0072] The core of the loss function is to guide the model to strengthen the matching degree of correct image-text pairs and weaken the matching degree of incorrect pairs, thereby enabling the model to learn accurate semantic alignment of images and text. For example, in a batch (N) of "image-text" pairs, the images and texts are encoded into unit vectors of the same dimension by ViT and BERT respectively; the cosine similarity between the images and texts is scaled using a temperature coefficient α (used to adjust the discriminative power of similarity) to obtain the similarity matrix S. ij (representing the matching degree between the i-th image and the j-th text); the loss is composed of the negative logarithm of "the similarity between the current image and the correct text" divided by "the sum of the similarities between the current image and all candidate texts". If the similarity (S) of the correct image-text pair... ii The number of incorrect pairs is much higher than the number of incorrect pairs, and the S value in the denominator is much higher. ii A higher similarity ratio results in a smaller overall loss; conversely, if the similarity of incorrect pairs is also high, the denominator will be increased, leading to a larger loss. In short, the core of this loss function is to guide the model to strengthen the matching degree of correct image-text pairs and weaken the matching degree of incorrect pairs, thereby enabling the model to learn accurate image-text semantic alignment.

[0073] This embodiment can achieve image-text matching and recognition of underground violations by aligning dual-tower image and text features, improving semantic discriminability by combining scene priors, and improving recognition accuracy by closed-loop optimization.

[0074] The image-text matching model can be used for inference in identifying and recognizing illegal activities in underground mines through image-text matching. Figure 4 The reasoning flowchart for identifying downhole violations provided in this application is shown in the figure, and includes the following implementation details: 1. Input and scene prior loading.

[0075] The input image (underground monitoring image) is loaded along with descriptions of all known violations (e.g., text such as "not wearing a safety helmet" or "helmet not fastened" in a tunneling scenario). First, a lightweight CNN / ViT scene recognition method is used to determine the scene to which the current image belongs (e.g., tunneling / coal mining). Then, the lightweight model is used to quickly extract scene features.

[0076] 2. Image and text feature encoding and alignment.

[0077] Image feature encoding: Taking "image batch (32, 3, image width and height)" as input, the global features of the image are extracted by the image encoder ViT (output (32, 768)), and then compressed into 512-dimensional features through linear projection (output (32, 512)). ViT captures key features (such as personnel heads and equipment areas) of complex downhole images through the self-attention of image blocks. Text feature encoding: Taking "text batch (32, text length)" (i.e., a description of a known violation scenario) as input, the text encoder Transformer extracts semantic features (output (32, 768)), and then linearly projects them into 512-dimensional features (output (32, 512)). The Transformer is used to capture the semantic relationships within the text (e.g., the semantic features of "not wearing a helmet"). Feature alignment: After concatenating image features and text features, a comparison is performed to ensure that the image and text features are in the same vector space.

[0078] 3. Similarity matching and violation description filtering.

[0079] The cosine similarity between image features and each violation description text feature is calculated by calculating the similarity matrix S(32, 32): if it is a "positive sample pair (image and text matching)" (for example, the input image is "not wearing a safety helmet" and the corresponding text is also the description), the similarity is high, and the violation description and the corresponding image will be recorded; if it is a "negative sample pair (image and text mismatch)" (for example, the input image is "not wearing a safety helmet" and the corresponding text is "standing in front of the drilling rig"), the similarity is low, and the violation description will be "abandoned".

[0080] 4. Results output and report generation.

[0081] All successfully matched violation descriptions are statistically analyzed and compiled into a report containing both violation descriptions and images, thus completing the structured output of the inference results and transforming the recognition results into an archiveable format that meets practical needs.

[0082] By first identifying the scene to which the image belongs, and then using image-text feature alignment and similarity matching, content that matches the input image is filtered out from known violation descriptions, and finally a structured violation report is generated.

[0083] As an optional embodiment, after determining whether the current text feature vector belongs to the fine-grained violation category, the method further includes: if the current text feature vector belongs to the fine-grained violation category, extracting local region features corresponding to the current text description from the monitoring image; calculating the local similarity between the local region features and the current text feature vector; calculating the overall similarity between the image representation vector and the current text feature vector; and weightedly fusing the overall similarity and local similarity to obtain the semantic similarity.

[0084] Once the current text feature vector is determined to belong to a fine-grained violation category, it is necessary to introduce fine-grained matching of local regions to enhance the ability to identify violations that rely on subtle visual features or subjective judgment.

[0085] The system locates and extracts key local regions directly related to the semantics of the current fine-grained text description from the original surveillance images, including: rule-based guidance: locating based on the anatomical regions (e.g., "hat strap not fastened" associated with "mandibular region"), equipment parts, or spatial orientations associated with the violation type in a pre-built violation semantic knowledge base; model-based detection: locating relevant subjects (e.g., people) and key points (e.g., head key points) using pre-built lightweight computer vision models (e.g., human pose estimation models, object detection models), and further determining more precise regions of interest (e.g., regions centered on mandibular key points) based on text semantics.

[0086] After acquiring the local region image, a feature extraction network is used to generate the local region feature vector of the local region image.

[0087] This application uses a lightweight version of Open Pose (or other pose estimation algorithms) to estimate human pose and locate the violation-related parts (head). The similarity between the feature vector of this region and the fine-grained semantic word vector (such as the jaw region feature corresponding to "hat strap not fastened") is calculated separately, and then weighted and fused with the overall image and text similarity to improve the fine-grained recognition accuracy.

[0088] The semantic similarity between the feature vector of a local region and the feature vector of the current text is calculated to obtain the local similarity. This value directly reflects the degree of semantic detail matching between the visual content of a key local region in the image and the fine-grained text description. For example, for the description "hatband not fastened," the local similarity measures the semantic consistency between "features of the jaw region of a person in the image" and the text description "hatband not fastened."

[0089] The above explanation illustrates how the overall similarity between the image representation vector and the current text feature vector is calculated. The overall similarity value represents the degree of matching between the semantics expressed by the overall surveillance image (including scene, subject, and general behavior) and the overall text description, providing a macro-contextual basis for violation determination.

[0090] The calculated overall similarity and local similarity are weighted and fused to obtain the final semantic similarity. A linear weighted fusion method can be used: Semantic Similarity = Overall Weight × Overall Similarity + Local Weight × Local Similarity, where the sum of the overall weight and local weight is 1. The weights can be preset fixed values ​​(e.g., each 0.5), dynamically adjusted parameters based on the violation category, or adaptive weights learned through model learning.

[0091] This application introduces a feature alignment mechanism for local regions and constructs a hierarchical image-text similarity calculation framework, thereby significantly improving the accuracy and reliability of identifying complex violations in downhole operations that rely on local conditions, subtle features, or subjective judgment.

[0092] As an optional embodiment, the method further includes dynamically updating the set of violation descriptions in the following manner: establishing a semantic knowledge graph corresponding to underground violations, wherein the semantic knowledge graph includes multiple category nodes, violation behavior nodes, and semantic relationships between nodes; when a new violation description appears, retrieving at least one existing violation behavior node matching the new violation behavior description from the semantic knowledge graph; generating a new text feature vector corresponding to the new violation behavior description through transfer learning based on the existing text feature vectors corresponding to the existing violation behavior nodes; obtaining feedback data corresponding to the new violation behavior description and the new text feature vector, and optimizing the new violation behavior description and the new text feature vector based on the feedback data; adding the optimized new violation behavior description and the new text feature vector to the set of violation descriptions to complete the dynamic update of the set of violation descriptions.

[0093] A semantic knowledge graph for the field of underground violations is pre-built. This graph is a structured semantic network in which nodes include category nodes representing major and subcategories of violations (e.g., "personal protective equipment" and "violation of helmet wearing"), and violation behavior nodes representing specific violations (e.g., "not wearing a helmet" and "helmet strap not fastened"). Edges (semantic relationships) connect the nodes and represent the relationships between them, such as "belongs to" (a behavior belongs to a certain category), "synonymous / near-synonymous," "partially" (e.g., "helmet strap not fastened" is a type of "improper helmet wearing"), and "often occurs together." Each edge can have a quantified semantic similarity weight.

[0094] When a new violation description requiring system identification appears (e.g., "improper use of respirator" added according to new safety procedures), the semantic knowledge graph is used to retrieve one or more existing violation nodes that are semantically most relevant to the new description through semantic similarity calculation (comparing text embedding vectors) or keyword matching. For example, the node "not wearing a dust mask" might be matched because both fall under the category of "respiratory protection" and are semantically similar.

[0095] Based on the matching results, feature transfer is performed. Specifically, pre-trained text feature vectors (i.e., the feature representation of the existing description in the model) associated with the matched existing violation nodes are obtained. Using these existing vectors as references, new text feature vectors for the description of the new violation are generated through transfer learning (which can be achieved through a lightweight neural network transformation layer or linear interpolation).

[0096] The newly added descriptions of violations and their corresponding text feature vectors are added to the system for trial recognition. The resulting recognition results and feedback data from manual review (positive samples confirming correct violations and negative samples indicating incorrect recognition) are collected. Using this feedback data, the new descriptions themselves (their textual wording can be adjusted) and their corresponding text feature vectors are optimized through online learning or small-batch fine-tuning.

[0097] Once the optimized new text feature vectors reach a certain level of stability or performance, the optimized new violation descriptions and the new text feature vectors are formally added to the violation description set. At this point, without modifying the main parameters of the image encoder and text encoder, the identification of these new violations can begin, completing the dynamic updating of the violation description set.

[0098] For new descriptions of violations, there's no need to collect and label large amounts of training images, nor is it necessary to retrain massive deep learning models. Only the new text description is required, and through knowledge graphs and feature transfer, preliminary recognition capabilities can be achieved within minutes or hours, reducing expansion costs.

[0099] This application provides a method for identifying underground violations, comprising: acquiring a monitoring image of the underground work scene to be identified; extracting features from the monitoring image to obtain an image feature vector, and performing scene recognition on the monitoring image to generate a scene feature vector; fusing the image feature vector and the scene feature vector to generate an image representation vector; acquiring a pre-constructed set of violation descriptions, and extracting semantic features from each text description in the set to generate a text feature vector; calculating the semantic similarity between the image representation vector and each text feature vector, and determining the behavior identification result matching the monitoring image based on the semantic similarity. By explicitly introducing the semantics of the work scene into the violation identification process, the same image behavior can obtain differentiated semantic understanding in different work scenes, and the violation determination is completed by combining the semantic matching of image and text, thereby improving the accuracy and adaptability of underground violation identification and solving the problem of reduced identification accuracy due to missing scene semantics or only performing single semantic judgment.

[0100] According to another aspect of the embodiments of this application, this application provides a device for identifying downhole violations, such as... Figure 5 As shown, it includes: The acquisition module 501 is used to acquire monitoring images of the downhole operation scene to be identified; The first extraction module 502 is used to extract features from the monitoring image to obtain an image feature vector, and to perform scene recognition on the monitoring image to generate a scene feature vector. The fusion module 503 is used to fuse the image feature vector with the scene feature vector to generate an image representation vector; The second extraction module 504 is used to obtain a pre-constructed set of descriptions of violations, and to extract semantic features from each text description in the set of descriptions of violations to generate a text feature vector. The calculation module 505 is used to calculate the semantic similarity between the image representation vector and each text feature vector, and to determine the behavior recognition result that matches the monitoring image based on the semantic similarity.

[0101] It should be noted that the acquisition module 501 in this embodiment can be used to execute step 201 in this application embodiment, the first extraction module 502 in this embodiment can be used to execute step 202 in this application embodiment, the fusion module 503 in this embodiment can be used to execute step 203 in this application embodiment, the second extraction module 504 in this embodiment can be used to execute step 204 in this application embodiment, and the calculation module 505 in this embodiment can be used to execute step 205 in this application embodiment.

[0102] Optionally, scene recognition is performed on the surveillance image to generate a scene feature vector, including: inputting the surveillance image into a preset scene classification model; determining the scene category to which the surveillance image belongs based on the output of the preset scene classification model; and encoding the scene category into an embeddable vector to obtain the scene feature vector.

[0103] Optionally, the image feature vector and the scene feature vector are fused to generate an image representation vector, including: inputting the scene feature vector into a mapping network and mapping it to the semantic feature space to which the image feature vector belongs, to obtain an aligned scene feature vector; fusing the image feature vector and the scene feature vector to generate a fused feature vector; and normalizing the fused feature vector to obtain an image representation vector.

[0104] Optionally, before obtaining the pre-constructed set of violation descriptions, the method further includes constructing the set of violation descriptions in the following manner: for the target job scenario to be trained, the job type and job environment range of the target job scenario are used as limiting conditions; under the limiting conditions, each violation description and at least one non-violation description in the target job scenario are enumerated and combined to obtain multiple first violation descriptions; semantic text annotations are generated for each first violation description, and the first violation descriptions and semantic text annotations are stored in a structured form, wherein the semantic text annotations include scene labels, violation subjects, and violation action information; second violation descriptions that require subjective judgment are selected from each first violation description, and the second violation descriptions are subjected to fine-grained semantic division to obtain third violation descriptions; the first violation descriptions and third violation descriptions are integrated to obtain a set of violation descriptions.

[0105] Optionally, the semantic similarity between the image representation vector and each text feature vector is calculated, including calculating the semantic similarity between the image representation vector and the current text feature vector in the following manner: performing semantic parsing on the current text feature vector to determine whether the current text feature vector belongs to a fine-grained violation category; if the current text feature vector does not belong to a fine-grained violation category, then calculating the overall similarity between the image representation vector and the current text feature vector; and using the overall similarity as the semantic similarity between the image representation vector and the current text feature vector.

[0106] Optionally, after determining whether the current text feature vector belongs to the fine-grained violation category, the method further includes: if the current text feature vector belongs to the fine-grained violation category, extracting local region features corresponding to the current text description from the monitoring image; calculating the local similarity between the local region features and the current text feature vector; calculating the overall similarity between the image representation vector and the current text feature vector; and weightedly fusing the overall similarity and local similarity to obtain the semantic similarity.

[0107] Optionally, the method further includes dynamically updating the set of violation descriptions in the following manner: establishing a semantic knowledge graph corresponding to underground violations, wherein the semantic knowledge graph includes multiple category nodes, violation behavior nodes, and semantic relationships between nodes; when a new violation description appears, retrieving at least one existing violation behavior node matching the new violation behavior description from the semantic knowledge graph; generating a new text feature vector corresponding to the new violation behavior description through transfer learning based on the existing text feature vectors corresponding to the existing violation behavior nodes; obtaining feedback data corresponding to the new violation behavior description and the new text feature vector, and optimizing the new violation behavior description and the new text feature vector based on the feedback data; adding the optimized new violation behavior description and the new text feature vector to the set of violation descriptions to complete the dynamic update of the set of violation descriptions.

[0108] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown.

[0109] According to another aspect of the embodiments of this application, this application provides an electronic device, such as... Figure 6 As shown, the system includes a memory 601, a processor 603, a communication interface 605, and a communication bus 607. The memory 601 stores a computer program that can run on the processor 603. The memory 601 and the processor 603 communicate through the communication interface 605 and the communication bus 607. When the processor 603 executes the computer program, it implements the steps of the above method.

[0110] The memory and processor in the aforementioned electronic devices communicate with each other via a communication bus and a communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0111] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0112] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0113] According to another aspect of the embodiments of this application, a computer-readable medium having processor-executable non-volatile program code is also provided.

[0114] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0115] In specific implementation, the embodiments of this application can be referred to the above embodiments and have corresponding technical effects.

[0116] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0117] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0118] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0120] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0121] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0123] If the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks. It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0124] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for identifying illegal activities in underground mines, characterized in that, include: Acquire monitoring images of the downhole operation scene to be identified; Feature extraction is performed on the monitoring image to obtain an image feature vector, and scene recognition is performed on the monitoring image to generate a scene feature vector; The image feature vector is fused with the scene feature vector to generate an image representation vector; Obtain a pre-constructed set of descriptions of violations, and extract semantic features from each text description in the set of descriptions of violations to generate a text feature vector; Calculate the semantic similarity between the image representation vector and each of the text feature vectors, and determine the behavior recognition result that matches the monitoring image based on the semantic similarity.

2. The method according to claim 1, characterized in that, The step of performing scene recognition on the monitored image and generating a scene feature vector includes: The surveillance image is input into a preset scene classification model; The scene category to which the surveillance image belongs is determined based on the output of the preset scene classification model; The scene category is encoded into an embeddable vector to obtain the scene feature vector.

3. The method according to claim 1, characterized in that, The step of fusing the image feature vector with the scene feature vector to generate an image representation vector includes: The scene feature vector is input into a mapping network and mapped to the semantic feature space to which the image feature vector belongs, to obtain the aligned scene feature vector; The image feature vector and the scene feature vector are fused to generate a fused feature vector; The fused feature vector is normalized to obtain the image representation vector.

4. The method according to claim 1, characterized in that, Before obtaining the pre-built set of violation descriptions, the method further includes constructing the set of violation descriptions in the following manner: For the target job scenario to be trained, the job type and job environment range of the target job scenario are used as limiting conditions; Under the given constraints, each description of violation and at least one description of non-violation in the target work scenario are enumerated and combined to obtain multiple descriptions of first violation. Semantic text annotations are generated for each of the first violation descriptions, and the first violation descriptions and the semantic text annotations are stored in a structured form, wherein the semantic text annotations include scene tags, violation subjects, and violation action information; From each of the first violation descriptions, the second violation descriptions that require subjective judgment are selected, and the second violation descriptions are further divided into fine-grained semantic segments to obtain the third violation descriptions; The first violation description and the third violation description are integrated to obtain the violation description set.

5. The method according to claim 4, characterized in that, The step of calculating the semantic similarity between the image representation vector and each of the text feature vectors includes calculating the semantic similarity between the image representation vector and the current text feature vector in the following manner: Perform semantic parsing on the current text feature vector to determine whether the current text feature vector belongs to a fine-grained violation category; If the current text feature vector does not belong to the fine-grained violation category, then calculate the overall similarity between the image representation vector and the current text feature vector; The overall similarity is used as the semantic similarity between the image representation vector and the current text feature vector.

6. The method according to claim 5, characterized in that, After determining whether the current text feature vector belongs to the fine-grained violation category, the method further includes: If the current text feature vector belongs to the fine-grained violation category, then extract the local region features corresponding to the current text description from the monitoring image; Calculate the local similarity between the local region features and the current text feature vector; Calculate the overall similarity between the image representation vector and the current text feature vector; The semantic similarity is obtained by weighted fusion of the overall similarity and the local similarity.

7. The method according to claim 4, characterized in that, The method further includes dynamically updating the set of violation descriptions in the following manner: Establish a semantic knowledge graph corresponding to underground violations, wherein the semantic knowledge graph includes multiple category nodes, violation behavior nodes, and semantic relationships between nodes; When a new violation description appears, at least one existing violation node that matches the new violation description is retrieved from the semantic knowledge graph; Based on the existing text feature vectors corresponding to the existing violation nodes, new text feature vectors corresponding to the descriptions of the new violations are generated through transfer learning. Obtain feedback data corresponding to the description of the newly added violation and the feature vector of the newly added text, and optimize the description of the newly added violation and the feature vector of the newly added text based on the feedback data; The optimized descriptions of the new violations and the new text feature vectors are added to the set of violation descriptions to complete the dynamic update of the set of violation descriptions.

8. A device for identifying illegal activities in underground mines, characterized in that, include: The acquisition module is used to acquire monitoring images of the downhole operation scene to be identified; The first extraction module is used to extract features from the monitoring image to obtain an image feature vector, and to perform scene recognition on the monitoring image to generate a scene feature vector. The fusion module is used to fuse the image feature vector with the scene feature vector to generate an image representation vector; The second extraction module is used to obtain a pre-constructed set of descriptions of violations, and to extract semantic features from each text description in the set of descriptions of violations to generate a text feature vector. The calculation module is used to calculate the semantic similarity between the image representation vector and each of the text feature vectors, and to determine the behavior recognition result that matches the monitoring image based on the semantic similarity.

9. An electronic device comprising a memory, a processor, a communication interface, and a communication bus, wherein the memory stores a computer program executable on the processor, and the memory and the processor communicate via the communication bus and the communication interface, characterized in that... When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the method of any one of claims 1 to 7.

Images