Weakly supervised human-object interaction detection method for bad scene recognition

Abstract

The application discloses a weakly supervised human-object interaction detection method for bad scene recognition, step 1, collecting bad scene pictures and constructing a data set; step 2, target detection, obtaining the preselected frame and feature information of the target person and object; step 3, relationship prediction, the human and object feature information obtained through step 2 is completed through the relationship prediction module to judge the action category; step 4, finally, according to the human and object and relationship prediction result, the bad scene picture is identified and verified; solve the local focusing problem in target detection when the supervision information is insufficient in the prior art, and the human-object interaction isolation problem in weak supervision relationship prediction, and improve the recognition effect of the bad scene.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and image recognition technology, and relates to a weakly supervised human-object interaction detection method for identifying adverse scenes. Background Technology

[0002] Inappropriate scene identification is a crucial part of content security review, aiming to detect whether images contain inappropriate information such as smoking or drinking. Existing inappropriate scene identification methods often rely on image classification or object detection methods for global identification, neglecting the complex interactions between multiple people and objects commonly present in images. Considering that inappropriate scene images often involve interactions between people and objects, such as the interaction between a person and the cigarette in a smoking scene, this invention proposes using Human-Object Interaction (HOI) detection for inappropriate scene image identification. An inappropriate scene is only identified when a person is detected interacting with a target object, rather than simply based on the detection of a target object in the image background, thus ensuring the accuracy of inappropriate scene identification. To overcome the difficulties caused by sparse labeled data, weakly supervised human-object interaction detection has gradually attracted more attention from researchers. However, it still faces two main problems: 1) Local focus problem in object detection when supervision information is insufficient. In the field of weakly supervised object detection, due to the weak supervision information, existing methods often tend to select the more discriminative parts of the object, frequently leading to the local focus problem. This results in incomplete detection of objects, and incomplete objects further affect the extraction of subsequent interaction relationships. 2) Complex backgrounds in adverse scene image data make it difficult to effectively extract human-object interaction relationships. Existing methods often search for interaction relationships by analyzing the spatial positions of human and objects pairwise. However, when multiple interaction relationships exist in an image, the potential connections between multiple interaction relationships are easily overlooked, leading to the interaction isolation problem. Summary of the Invention

[0003] The purpose of this invention is to provide a weakly supervised human-object interaction detection method for adverse scene recognition, which solves the problem of local focusing in target detection when supervision information is insufficient, and the problem of isolated human-object interactions in weakly supervised relationship prediction.

[0004] The technical solution adopted in this invention is a weakly supervised human-object interaction detection method for identifying adverse scenes, which is implemented according to the following steps: Step 1: Collect images of problematic scenes and construct a dataset; Step 2, target detection, obtaining pre-selected bounding boxes and feature information of target people and objects; Step 3, Relationship Prediction: The relationship prediction module uses the person and object feature information obtained in Step 2 to determine the action category. Step 4: Finally, identify and verify the problematic scene images.

[0005] The invention is further characterized by: Step 1 specifically involves: The dataset was constructed by crawling image search results from search engines to collect images of inappropriate scenes, including smoking and drinking. Multiple search engine websites were selected, a keyword crawling list was formulated, and the keywords were sent to different search engines through a crawling system to search and download the required images. At the same time, a normal image dataset was constructed to obtain the total dataset, which was then divided into training and test sets in an 8:2 ratio. The target detection in step 2 specifically involves: Step 2.1: Generate preselection boxes and extract features from the input image to obtain a set of preselection boxes. and its feature map ; Step 2.2: The attention masking method is used to find the most discriminative locations in the image and perform structured masking, forcing the network to learn target information from other features, thus obtaining the masked feature map. ; Step 2.3: Use the multi-instance feature classification method to classify and distinguish several pre-selected boxes in the given image, and correct the scores of the pre-selected boxes through self-training refinement supervision to obtain more accurate localization results; Step 2.4: A consistency constraint scheme is used to assist the multi-instance classifier in learning, and the feature maps before masking are... and the feature map after masking Simultaneously, the data is fed into the multi-example classification head, and a consistency constraint is applied to the classification results of both, thereby improving the classifier's ability to discriminate features. Step 2.1, which involves generating the preselection box and extracting features, is implemented as follows: Step 2.1.1: Use a selective search method to process the image to generate several preselected boxes, thus obtaining a set of preselected boxes. ; Step 2.1.2: Extract features from the region containing the preselected box using VGG-16 and ResNet-50 backbone networks. Step 2.1.3: After obtaining the preselected bounding box and image features, region feature maps are extracted using region pooling. ,in These represent the number of channels, width, and height of the convolutional feature map, respectively. The attention blocking in step 2.2 is implemented as follows: Step 2.2.1, transfer the feature map use The attention module extracts salient regions from the convolutional feature maps: (1) In the formula, the hard mask image Gumbel-Softmax represents the most discriminative region in the feature map and converts the probability map obtained after the Sigmoid operation into a hard mask map. Step 2.2.2: Use the Dropblock module to structurally mask the salient regions in the convolutional network, thus removing the salient regions. Feature map after inversion and before shielding Multiplying them yields a feature map with salient regions removed. ; (2); Step 2.3, the multi-instance feature classification, is implemented as follows: Step 2.3.1: Using the WSDDN feature classification framework, the features are fed into the classification stream and the detection stream respectively. The scores of the classification features and the detection features are aggregated to obtain the category score of the entire preselected box. Then, by summarizing the scores of all preselected boxes in the image on each category, the image-level prediction score is obtained, and supervision is performed based on the image-level label information. (3) In the formula, For the image-level loss of the feature classification part, For category Image level score, for a given image level label ,in Indicates whether the image contains a category. The goal; Step 2.3.2: Employ an improved OICR self-training framework, adding [a feature] to the existing example classifier. Each example is categorized into refined branches, and each refined branch is used to correct the preselection box score of the previous branch. The preselection box score is corrected through multi-level refined supervision to obtain more accurate positioning results. (4) In the formula, To refine the branches of the classification The preselected bounding box classification loss, Indicates the number of preselected boxes. The weight, For branches Middle Preselection Box In category Pseudo-label information on For branches Middle Preselection Box In category The scores on the list, of which the first Dimension is the background class; Step 2.3.3: Construct the loss function for the multi-instance classification head. Image-level loss from feature classification It consists of the pre-selected box classification loss of each category refinement branch; (5); The consistency constraint in step 2.4 is implemented in the following steps: Step 2.4.1, transform the feature map before shielding. and the feature map after masking Simultaneously send multiple example classification heads Since the target of the feature maps before and after masking is the same object, their classification results should maintain the same data distribution. Therefore, a consistency constraint loss is applied to the classification results of the two to improve the classifier's ability to discriminate features. (6) In the formula, For consistency constraint loss, A metric function for measuring the similarity between distributions; Step 2.4.2: For the weakly supervised target detection stage, construct its overall loss function. And conduct training; (7) In the formula, and The multi-instance classification head loss is defined before and after masking, respectively, and its form is the same as that in step 2.3.3. The same represents the classifier's ability to distinguish features of the preselected boxes. These are hyperparameters used to balance classification loss and consistency constraint loss; The relationship prediction in step 3 specifically involves: Step 3.1: An attention mechanism is used to encode and classify human and object features along with global features to obtain the appearance relationship prediction results, specifically: Step 3.1.1, for the region feature map of the given preselected box Human features were extracted using RoI pooling. Convolution and global average pooling are performed on the sample to obtain human features at the example level. ; (8) Step 3.1.2: Use dilated convolutional (CSD) modules of different scales to process the feature map. Global features are obtained by stacking multiple times. Combined with attention calculation Vector sum Feature transformation yields context features ; (9) In the formula, For LayerNorm normalization, For activation functions; Step 3.1.3, Example-level features and context features The context-encoded features are concatenated and fed into the final relation classifier to obtain the score of the preselected box. ; (10) Step 3.1.4: If there are multiple preselected boxes in the image, assign each preselected box to only one relationship category, and then assign all preselected boxes to the relationship category... The scores on each image are summed to obtain its image-level score. (11) In the formula, Represents relation categories Image-level score, Indicates the number of preselected boxes; Step 3.1.5, for a given image-level label Construct a classification loss function for the human feature stream. And then train it; similarly, the classification score of the object feature flow can be obtained. and image level score Classification loss of the feature flow of the construct : (12) (13) In the formula, Indicates whether the image contains a relationship category. The goal; Step 3.2: Construct human-centered and object-centered interaction relationship subgraphs. Extract interaction node features based on the graph structure and classify them to obtain spatial relationship prediction results. Then, fuse the scores of corresponding categories in the two subgraphs to obtain the final spatial relationship prediction result, as detailed below: Step 3.2.1: Extract initial spatial relationship representation. In order to encode the spatial relationship between people and objects, HO-RCNN is first used to represent the relative positions of the bounding boxes of people and objects, and at the same time, example-level feature information of people and objects is introduced to supplement it. (14) In the formula, Indicates the first Individual and the Initial spatial relationship characteristics between objects These are the bounding box features extracted by HO-RCNN. , These are example-level feature information for people and objects, respectively. Step 3.2.2: Construct an interaction relationship graph; For the detected people and objects, construct subgraphs centered on people and objects respectively, and insert interaction nodes into the subgraphs to construct two interaction relationship subgraphs; Step 3.2.3: Extract and classify interaction node features based on the graph structure; use the JKMLP module to extract interaction node features from the two interaction relationship subgraphs. The JKMLP module consists of two GMLP submodules and a fully connected layer, obtaining the spatial features of the two relationship subgraphs. and The data is then fed into a relation classifier to obtain a classification score for the spatial relations. and ; (15) (16) In the formula, This represents the characteristics of a human-centered relational subgraph space. The spatial characteristics of a relational subgraph centered on objects; Step 3.2.4: Construct a human-centered spatial flow classification loss function. and object-centric spatial flow classification loss function The system is then trained, and the scores of corresponding categories in the two subgraphs are fused to obtain the final spatial relationship prediction result. (17) (18) In the formula, and They are respectively for and Matrix in the preselection box set Interaction relationships obtained by accumulating across dimensions Image-level score; Step 3.3: Combine the appearance relationship prediction results with the spatial relationship prediction results to achieve the final relationship prediction and realize the discrimination of action relationships; Step 3.3 specifically involves: The action relationship discrimination model combines the appearance relationship prediction results with the spatial relationship prediction results, and constructs an overall loss function for the weakly supervised human-object relationship prediction stage. And through training, the final action relationship discrimination can be achieved: (19) In the formula, and The hyperparameters are used to balance the classification loss of human and object appearance feature flow and the classification loss of spatial feature flow. Step 4 specifically involves: The adverse scene image recognition method inputs the images to be recognized in the test set into a trained two-stage recognition model. Based on the predicted results of people, objects and relationships, i.e., the triplet of people, actions and objects, it determines whether the image is an adverse scene.

[0006] The beneficial effects of this invention are This invention presents a weakly supervised human-object interaction detection method for adverse scene identification. With lower annotation costs, it identifies adverse scenes only when a target interaction between a person and a target object is detected, rather than simply relying on the detection of a target object in the image background. This ensures the accuracy of adverse scene identification. This invention utilizes only image-level label information, achieving adverse scene identification through two stages: target detection and relationship prediction. It primarily addresses the localized focusing problem in target detection when supervision information is insufficient, and the isolated human-object interaction problem in weakly supervised relationship prediction, thus improving the effectiveness of adverse scene identification. Attached Figure Description

[0007] Figure 1 This is a diagram illustrating the overall model framework of the weakly supervised human-object interaction detection method for identifying adverse scenes according to the present invention. Figure 2 This is a framework diagram of the weakly supervised target detection method based on consistency constraints in step 2 of the weakly supervised human-object interaction detection method for adverse scene recognition in this invention; Figure 3 This is a structural diagram of the attention shielding module in step 2.2 of the weakly supervised human-object interaction detection method for adverse scene recognition in this invention. Figure 4 This is a structural diagram of the multi-instance feature classification module in step 2.3 of the weakly supervised target detection stage of the weakly supervised human-object interaction detection method for adverse scene recognition in this invention; Figure 5 This is a method framework diagram of the weakly supervised target detection stage in step 3 of the weakly supervised human-object interaction detection method for adverse scene recognition in this invention; Figure 6This is an example diagram of the interaction relationship construction in the weakly supervised relationship prediction stage of the weakly supervised human-object interaction detection method for adverse scene recognition in this invention; Figure 7 This is a structural diagram of the spatial feature encoder JKMLP module in step 3.2.3 of the weakly supervised human-object interaction detection method for identifying adverse scenes in this invention. Detailed Implementation

[0008] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0009] This invention provides a weakly supervised human-object interaction detection method for identifying adverse scenes, as shown in the following embodiments: This invention proposes a weakly supervised human-object interaction detection method for adverse scene recognition. Utilizing only image-level labeling information, it identifies adverse scenes through two stages: target detection and relationship prediction. In the target detection stage, an attention masking module is first used to find the most distinctive parts of the target and mask their features. The feature maps before and after masking are then fed into a multi-example classification head for feature classification. Consistency constraints are applied to the classification results, forcing the network to learn target information from other features, ensuring the completeness of target detection. In the relationship prediction stage, human and object features are first encoded and classified to obtain predicted relationships. Then, human-centered and object-centered interaction relationship graphs are constructed, and graph aggregation methods are used for feature extraction and classification to obtain spatial relationship prediction results, effectively improving the relationship prediction capability. Finally, adverse scene judgment is performed based on the obtained human, object, and relationship prediction results, improving the recognition effect of adverse scenes.

[0010] A weakly supervised human-object interaction detection method for identifying adverse scenes is implemented according to the following steps: Step 1, Construction of the Inappropriate Scene Image Dataset: In this embodiment, the inappropriate scene dataset is constructed by web crawling the three major search engine websites: Baidu, Google, and Bing. Inappropriate scenes include smoking and drinking. By defining a keyword crawling list, the crawling system sends the keywords to different search engines to search for and download the required images. The collected data is cleaned, including deduplication and deletion of irrelevant images. At the same time, a normal image dataset is constructed to obtain the total dataset, which is then divided into training and test sets in an 8:2 ratio. Step 2, Weakly Supervised Target Detection Stage: A weakly supervised target detection method based on consistency constraints is used to obtain pre-selected bounding boxes and feature information for target people and objects; the framework of this method is as follows: Figure 2 As shown, the specific steps include: Step 2.1: Generate preselection boxes and extract features from the input image to obtain a set of preselection boxes. and its feature map ; Step 2.1.1: Use the Selective Search (SS) method to process the image to generate several preselected boxes, thus obtaining a set of preselected boxes. ; Step 2.1.2: Extract features from the region containing the preselected box using VGG-16 and ResNet-50 backbone networks. Step 2.1.3: After obtaining the preselected bounding box and image features, region feature maps are extracted using Region of Interest Pooling (RoI Pooling). ,in These represent the number of channels, width, and height of the convolutional feature map, respectively. Step 2.2: The Attention Dropblock method is used to identify the most discriminative locations in the image and perform structured masking, forcing the network to learn target information from other features, resulting in a masked feature map. ,like Figure 3 As shown, please follow these steps: Step 2.2.1, transfer the feature map use The attention module extracts salient regions from the convolutional feature maps: (1) In the formula, the hard mask image Gumbel-Softmax represents the most discriminative region in the feature map and converts the probability map obtained after the Sigmoid operation into a hard mask map. Step 2.2.2: Use the Dropblock module to structurally mask the salient regions in the convolutional network, thus removing the salient regions. Feature map after inversion and before shielding Multiplying them yields a feature map with salient regions removed. ; (2); Step 2.3 employs Multiple Instance Learning (MIL) to classify and discriminate several pre-selected bounding boxes in the given image, and refines the scores of the pre-selected boxes through self-training and supervision to obtain more accurate localization results, such as... Figure 4 As shown, please follow these steps: Step 2.3.1: Using the WSDDN feature classification framework, the features are fed into the classification stream and the detection stream respectively to obtain the classification matrix. With detection matrix Then, softmax is applied to the two feature matrices along different dimensions to obtain the classification scores of the preselected boxes. and test score ; (20) (twenty one) In the formula, Indicates the preselection box Category The probability, Indicates the preselection box Category The score for the preselected box is determined by the degree of contribution. can be and The element-wise dot product yields: (twenty two) Image-level prediction scores can be obtained by summing the scores of all pre-selected boxes in the image for each category. Specifically, for Matrix in The categories are obtained by accumulating the dimensions. Image level score : (twenty three) Image-level loss for constructing the feature classification part And conduct training; (3) In the formula, For category Image level score, for a given image level label ,in Indicates whether the image contains a category. The goal; Step 2.3.2: Employ an improved OICR self-training framework, adding [a feature] to the existing example classifier. Each example is categorized into refined branches, and each refined branch is used to correct the preselection box score of the previous branch. The preselection box score is corrected through multi-level refined supervision to obtain more accurate positioning results. (4) In the formula, To refine the branches of the classification The preselected bounding box classification loss, Indicates the number of preselected boxes. The weight, For branches Middle Preselection Box In category Pseudo-label information on For branches Middle Preselection Box In category The scores on the list, of which the first Dimension is the background class; Step 2.3.3: Construct the loss function for the multi-instance classification head. Image-level loss from feature classification It consists of the pre-selected box classification loss of each category refinement branch; (5); Step 2.4: A consistency constraint scheme is used to assist the multi-instance classifier in learning, and the feature maps before masking are... and the feature map after masking Simultaneously, the data is fed into the multi-example classification head, and a consistency constraint is applied to the classification results of both, thereby improving the classifier's ability to discriminate features. This is implemented through the following steps: Step 2.4.1, transform the feature map before shielding. and the feature map after masking Simultaneously send multiple example classification heads Since the target of the feature maps before and after masking is the same object, their classification results should maintain the same data distribution. Therefore, a consistency constraint loss is applied to the classification results of the two to improve the classifier's ability to discriminate features. (6) In the formula, For consistency constraint loss, The measurement function for the similarity between distributions can be the mean squared error loss or the KL divergence loss, etc. Step 2.4.2: For the weakly supervised target detection stage, construct its overall loss function. And conduct training; (7) In the formula, and The multi-instance classification head loss is defined before and after masking, respectively, and its form is the same as that in step 2.3.3. The same represents the classifier's ability to distinguish features of the preselected boxes. These are hyperparameters used to balance classification loss and consistency constraint loss; Step 3, Weakly Supervised Target Detection Stage: The person and object feature information obtained in the target detection stage is sent to the weakly supervised relationship prediction module, and the action category is determined using a weakly supervised person-object relationship prediction method based on interaction graphs; for example... Figure 5 As shown, please follow these steps: Step 3.1: When people and objects are in different action relationships, they often exhibit different appearances. Therefore, an attention mechanism is used to combine human and object features with global features for encoding and classification to obtain appearance relationship prediction results. The specific process is as follows: Step 3.1.1, for the region feature map of the given preselected box Human features were extracted using RoI pooling. Convolutional and Global Average Pooling (GAP) operations are performed on the sample to obtain human features at the example level. ; (8) Step 3.1.2: Use dilated convolutional CSD modules (Concat Stack Dconv) of different scales to process the feature maps. Global features are obtained by stacking multiple times. That is, using three different scales of dilated convolution to stitch together the intermediate results and then perform convolution fusion; (twenty four) In the formula, A dilated convolution with a dilation rate of 1. A dilated convolution with a dilation rate of 3. A dilated convolution with a dilation rate of 6 is used; global features can be obtained by stacking CSD modules multiple times. : (25) In the formula, This refers to the number of times the module is stacked, combined with the attention calculation. Vector sum Feature transformation yields context features : (9) In the formula, For LayerNorm normalization, For activation functions; Step 3.1.3, Example-level features and context features The context-encoded features are concatenated and fed into the final relation classifier to obtain the score of the preselected box. ; (10) Step 3.1.4: If there are multiple preselected boxes in the image, assign each preselected box to only one relationship category, and then assign all preselected boxes to the relationship category... The scores on each image are summed to obtain its image-level score. (11) In the formula, Represents relation categories Image-level score, Indicates the number of preselected boxes; Step 3.1.5, for a given image-level label Construct a classification loss function for the human feature stream. And then train it; similarly, the classification score of the object feature flow can be obtained. and image level score Classification loss of the feature flow of the construct : (12) (13) In the formula, Indicates whether the image contains a relationship category. The goal; Step 3.2: When people and objects interact, different actions usually result in different spatial relationships. By constructing interaction relationship subgraphs centered on the person and the object, the interaction node features are extracted and classified according to the graph structure to obtain spatial relationship prediction results. The scores of corresponding categories in the two subgraphs are then fused to obtain the final spatial relationship prediction result, as detailed below: Step 3.2.1: Extract initial spatial relationship representation. In order to encode the spatial relationship between people and objects, HO-RCNN is first used to represent the relative positions of the bounding boxes of people and objects, and at the same time, example-level feature information of people and objects is introduced to supplement it. (14) In the formula, Indicates the first Individual and the Initial spatial relationship characteristics between objects These are the bounding box features extracted by HO-RCNN. , These are example-level feature information for people and objects, respectively. Step 3.2.2: Construct an interaction relationship graph; For the detected people and objects, construct subgraphs centered on people and objects respectively, and insert interaction nodes into the subgraphs to construct two interaction relationship subgraphs; The specific construction process of the interaction diagram is as follows: Figure 6 As shown, let The set of people obtained through target detection. For each individual node, the set is a collection of objects. to Connect all nodes to establish a network. Similarly, for each object node, the interaction subgraph centered on the object node is also... , and Connect all nodes to establish a network. This is the central interaction subgraph; each node in the subgraph represents a detected object instance. To explicitly express the interaction relationships, each node... and object nodes Insert a relation node , For the first Individual and the first By analyzing the spatial semantic features between objects and connecting all the relation nodes, we can obtain a human-centered relation subgraph and an object-centered relation subgraph. Step 3.2.3: Extract and classify interaction node features based on the graph structure; use the JKMLP module to extract interaction node features from the two interaction relationship subgraphs. The structure diagram of the JKMLP module is shown below. Figure 7 As shown, the JKMLP module consists of two GMLP sub-modules and a fully connected layer. The first GMLP sub-module extracts initial neighborhood information. The initial spatial features and the output of the first GMLP sub-module are concatenated, and the result is transformed by the fully connected layer before being fed into the second GMLP sub-module to obtain the final graph node convolutional features. At this point, the graph node features have obtained dependencies between different interaction pairs and contain rich contextual information. Specifically, the first... Individual and the first Spatial semantic feature nodes between objects Obtained from JKMLP It can be represented as: (26) Similarly, the spatial features of the two relational subgraphs can be obtained. and The data is then fed into a relation classifier to obtain a classification score for the spatial relations. and ; (15) (16) In the formula, This represents the characteristics of a human-centered relational subgraph space. For object-centric relational subgraph space features, Matrix and In the preselection set By accumulating dimensions, we can obtain the interaction relationships. Image level score and : (27) (28) Step 3.2.4: Construct a human-centered spatial flow classification loss function. and object-centric spatial flow classification loss function The system is then trained, and the scores of corresponding categories in the two subgraphs are fused to obtain the final spatial relationship prediction result. (17) (18) In the formula, and They are respectively for and Matrix in the preselection box set Interaction relationships obtained by accumulating across dimensions Image-level score; Step 3.3 combines the appearance relationship prediction results with the spatial relationship prediction results to achieve the final relationship prediction, by constructing an overall loss function for the weakly supervised human-object relationship prediction stage. And train them to discern the relationships between movements: (19) In the formula, and The hyperparameters are used to balance the loss for classifying the flow of people and objects and the loss for classifying the flow of spatial features. Step 4: Load the trained recognition model and perform adverse scene recognition verification. Input images from the test set of the dataset into the model. Based on the predicted results of people, objects, and relationships (i.e., the triplet of person, action, and object), determine whether the image is an adverse scene. For example, for detecting a smoking scene, if the recognition result is (person, hold, smoke), it is determined to be an image of an adverse smoking scene, rather than simply judging it as an adverse scene based on the detection of a cigarette object in the background. The weakly supervised human-object interaction detection method of this invention, with low annotation costs, only judges it as an adverse scene when a target interaction action between a person and a target object is detected, ensuring the accuracy of the detection results.

Claims

1. A weakly supervised human-object interaction detection method for identifying adverse scenes, characterized in that, The specific steps are as follows: Step 1: Collect images of problematic scenes and construct a dataset; Step 2, target detection, obtaining pre-selected bounding boxes and feature information of target people and objects; The target detection in step 2 specifically involves: Step 2.1: Generate preselection boxes and extract features from the input image to obtain a set of preselection boxes. and its feature map ; Step 2.2: The attention masking method is used to find the most discriminative locations in the image and perform structured masking, forcing the network to learn target information from other features, thus obtaining the masked feature map. ; Step 2.3: Use the multi-instance feature classification method to classify and distinguish several pre-selected boxes in the given image, and correct the scores of the pre-selected boxes through self-training refinement supervision to obtain more accurate localization results; Step 2.4: A consistency constraint scheme is used to assist the multi-instance classifier in learning, and the feature maps before masking are... and the feature map after masking Simultaneously, the data is fed into the multi-example classification head, and a consistency constraint is applied to the classification results of both, thereby improving the classifier's ability to discriminate features. The consistency constraint is implemented in the following steps: Step 2.4.1, transform the feature map before shielding. and the feature map after masking Simultaneously send multiple example classification heads Since the target of the feature maps before and after masking is the same object, their classification results should maintain the same data distribution. Therefore, a consistency constraint loss is applied to the classification results of the two to improve the classifier's ability to discriminate features. （6） In the formula, For consistency constraint loss, A metric function for measuring the similarity between distributions; Step 2.4.2: For the weakly supervised target detection stage, construct its overall loss function. And conduct training; （7） In the formula, and These are the multi-instance classification head losses before and after masking, representing the classifier's ability to distinguish features of the preselected boxes. These are hyperparameters used to balance classification loss and consistency constraint loss; Step 3, Relationship Prediction: The relationship prediction module uses the person and object feature information obtained in Step 2 to determine the action category. Step 4: Finally, identify and verify the problematic scene images.

2. The weakly supervised human-object interaction detection method for adverse scene recognition according to claim 1, characterized in that, Step 1 specifically involves: The dataset was constructed by crawling image search results from search engines to collect images of inappropriate scenes, including smoking and excessive drinking. Multiple search engine websites were selected, a keyword crawling list was defined, and the crawling system was used to send the keywords to different search engines to search and download the required images. At the same time, a normal image dataset was constructed to obtain the total dataset, which was then divided into training and test sets in an 8:2 ratio.

3. The weakly supervised human-object interaction detection method for adverse scene recognition according to claim 1, characterized in that, The preselection box generation and feature extraction in step 2.1 are implemented as follows: Step 2.1.1: Use a selective search method to process the image to generate several preselected boxes, thus obtaining a set of preselected boxes. ; Step 2.1.2: Extract features from the region containing the preselected box using VGG-16 and ResNet-50 backbone networks. Step 2.1.3: After obtaining the preselected bounding box and image features, region feature maps are extracted using region pooling. ,in These represent the number of channels, width, and height of the convolutional feature map, respectively.

4. The weakly supervised human-object interaction detection method for adverse scene recognition according to claim 3, characterized in that, The attention blocking in step 2.2 is implemented in the following steps: Step 2.2.1, transfer the feature map use The attention module extracts salient regions from the convolutional feature maps: （1） In the formula, the hard mask image Gumbel-Softmax represents the most discriminative region in the feature map and converts the probability map obtained after the Sigmoid operation into a hard mask map. Step 2.2.2: Use the Dropblock module to structurally mask the salient regions in the convolutional network, thus removing the salient regions. Feature map after inversion and before shielding Multiplying them yields a feature map with salient regions removed. ; （2）。 5. The weakly supervised human-object interaction detection method for adverse scene recognition according to claim 4, characterized in that, The multi-instance feature classification in step 2.3 is implemented in the following steps: Step 2.3.1: Using the WSDDN feature classification framework, the features are fed into the classification stream and the detection stream respectively. The scores of the classification features and the detection features are aggregated to obtain the category score of the entire preselected box. Then, by summarizing the scores of all preselected boxes in the image on each category, the image-level prediction score is obtained, and supervision is performed based on the image-level label information. （3） In the formula, For the image-level loss of the feature classification part, For category Image level score, for a given image level label ,in Indicates whether the image contains a category. The goal; Step 2.3.2: Employ an improved OICR self-training framework, adding [a feature] to the existing example classifier. Each example is categorized into refined branches, and each refined branch is used to correct the preselection box score of the previous branch. The preselection box score is corrected through multi-level refined supervision to obtain more accurate positioning results. （4） In the formula, To refine the branches for classification The preselected bounding box classification loss, Indicates the number of preselected boxes. The weight, For branches Middle Preselection Box In category Pseudo-label information on For branches Middle Preselection Box In category The scores on the list, of which the first Dimension is the background class; Step 2.3.3: Construct the loss function for the multi-instance classification head. Image-level loss from feature classification It consists of the pre-selected box classification loss of each category refinement branch; （5）。 6. The weakly supervised human-object interaction detection method for adverse scene recognition according to claim 3, characterized in that, The relationship prediction in step 3 specifically involves: Step 3.1: An attention mechanism is used to encode and classify human and object features along with global features to obtain the appearance relationship prediction results, specifically: Step 3.1.1, for the region feature map of the given preselected box Human features were extracted using RoI pooling. Convolution and global average pooling are performed on the sample to obtain human features at the example level. ; （8） Step 3.1.2: Use dilated convolutional (CSD) modules of different scales to process the feature map. Global features are obtained by stacking multiple times. Combined with attention calculation Vector sum Feature transformation yields context features ; （9） In the formula, For LayerNorm normalization, For activation functions; Step 3.1.3, Example-level features and context features The context-encoded features are concatenated and fed into the final relation classifier to obtain the score of the preselected box. ; （10） Step 3.1.4: If there are multiple preselected boxes in the image, assign each preselected box to only one relationship category, and then assign all preselected boxes to the relationship category... The scores on each image are summed to obtain its image-level score. （11） In the formula, Represents relation categories Image-level score, Indicates the number of preselected boxes; Step 3.1.5, for a given image-level label Construct a classification loss function for the human feature stream. And then train it; similarly, the classification score of the object feature flow can be obtained. and image level score Classification loss of the feature flow of the construct : （12） （13） In the formula, Indicates whether the image contains a relationship category. The goal; Step 3.2: Construct human-centered and object-centered interaction relationship subgraphs. Extract interaction node features based on the graph structure and classify them to obtain spatial relationship prediction results. Then, fuse the scores of corresponding categories in the two subgraphs to obtain the final spatial relationship prediction result, as detailed below: Step 3.2.1: Extract initial spatial relationship representation. In order to encode the spatial relationship between people and objects, HO-RCNN is first used to represent the relative positions of the bounding boxes of people and objects, and at the same time, example-level feature information of people and objects is introduced to supplement it. （14） In the formula, Indicates the first Individual and the Initial spatial relationship characteristics between objects These are the bounding box features extracted by HO-RCNN. , These are example-level feature information for people and objects, respectively. Step 3.2.2: Construct an interaction relationship graph; For the detected people and objects, construct subgraphs centered on people and objects respectively, and insert interaction nodes into the subgraphs to construct two interaction relationship subgraphs; Step 3.2.3: Extract and classify interaction node features based on the graph structure; use the JKMLP module to extract interaction node features from the two interaction relationship subgraphs. The JKMLP module consists of two GMLP submodules and a fully connected layer, obtaining the spatial features of the two relationship subgraphs. and The data is then fed into a relation classifier to obtain a classification score for the spatial relations. and ; （15） （16） In the formula, This represents the characteristics of a human-centered relational subgraph space. The spatial characteristics of a relational subgraph centered on objects; Step 3.2.4: Construct a human-centered spatial flow classification loss function. and object-centric spatial flow classification loss function The system is then trained, and the scores of corresponding categories in the two subgraphs are fused to obtain the final spatial relationship prediction result. （17） （18） In the formula, and They are respectively for and Matrix in the preselection box set Interaction relationships obtained by accumulating across dimensions Image-level score; Step 3.3: Combine the appearance relationship prediction results with the spatial relationship prediction results to achieve the final relationship prediction and realize the discrimination of action relationships.

7. The weakly supervised human-object interaction detection method for adverse scene recognition according to claim 6, characterized in that, Step 3.3 specifically involves: The action relationship discrimination model combines the appearance relationship prediction results with the spatial relationship prediction results, and constructs an overall loss function for the weakly supervised human-object relationship prediction stage. And through training, the final action relationship discrimination can be achieved: （19） In the formula, and The hyperparameter is used to balance the classification loss of appearance features of people and objects and the classification loss of spatial features.

8. The weakly supervised human-object interaction detection method for adverse scene recognition according to claim 2, characterized in that, Step 4 specifically involves: The adverse scene image recognition method inputs the images to be recognized in the test set into a trained two-stage recognition model. Based on the predicted results of people, objects and relationships, i.e., the triplet of people, actions and objects, it determines whether the image is an adverse scene.

Images