A method for detecting illegal use of a mobile phone, a monitoring method and an electronic device

Abstract

The application discloses a method for detecting illegal mobile phone use behavior, a monitoring method and electronic equipment, and belongs to the technical field of image processing. Three branches are used respectively for human body, object and interaction relationship detection. Each branch decodes features related to respective tasks from image features through a corresponding decoder i H , f i I , f i O In this process, spatial prior is provided by introducing anchor boxes, and instance distribution of a training set is learned to guide query decoding. The decoding features of each branch are fused with mutual relationship features f i H , f i I , f i O and image features, so that rich multiple context information is transmitted between decoders to realize feature enhancement, and the corresponding detection task is accurately realized. The relationship branch constructs an interaction area according to human body and object anchor boxes to obtain relationship information, and ensures that the attention area of the relationship branch is aligned with the human and object branches. Based on this, the application can efficiently and accurately detect illegal mobile phone use behavior at a low cost.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to a method for detecting, monitoring, and using electronic equipment for detecting unauthorized mobile phone use. Background Technology

[0002] In the production process of enterprises, especially thermal power plants, a large number of high-temperature and high-pressure equipment are involved. Any operational errors or monitoring negligence can lead to serious safety accidents. The work quality of the control room staff directly affects the safety and operational efficiency of the entire power plant. The control room is the command center of the power plant, responsible for monitoring and controlling all production links to ensure the continuity and safety of power production. Control room work requires a high degree of concentration and timely response. Unauthorized use of mobile phones not only reduces focus but can also lead to slow reaction times and an inability to quickly handle emergencies. Detecting and preventing unauthorized mobile phone use ensures that staff work efficiently and improves overall operational efficiency.

[0003] To address this issue, businesses typically rely on manual inspections and monitoring. However, this traditional method is not only resource-intensive but also inefficient. Therefore, how to utilize modern technology to automate the monitoring of unauthorized mobile phone use has become an urgent problem to solve.

[0004] The use of computer vision and deep learning methods to detect unauthorized mobile phone use in work areas has attracted widespread attention across various industries. These technologies can significantly improve the efficiency and accuracy of monitoring, reduce labor costs, and enhance overall security management. However, existing technologies often define mobile phone use detection as either a mobile phone target detection task or a human behavior classification task. The former assumes mobile phone use upon detection of a phone; for example, some methods extract the area surrounding a person in an image, determine if the person is holding a phone or if there are phones around them, calculate the direction of the face, and then determine if the person is "using a phone." However, in many cases, workers simply place their phones on a table without actually using them, making accurate detection impossible. The latter method only considers human movement to determine if mobile phone use has occurred; for example, some methods extract the human skeleton and use the YOLO algorithm to detect if there are phones near the hands of a person looking down, thus determining if a phone is being used. However, in scenarios like duty rooms, actions such as writing are similar to mobile phone use, easily generating numerous false alarms and also failing to achieve accurate detection. Based on this, some scholars have proposed sensor detection methods. Although sensor detection methods can make up for the shortcomings of visual detection, their equipment costs are high, deployment is difficult, and they are not suitable for large-scale applications. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method for detecting, monitoring and electronic devices for detecting unauthorized use of mobile phones, the purpose of which is to detect unauthorized use of mobile phones efficiently and accurately at a low cost.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for detecting unauthorized mobile phone use, comprising:

[0007] The scene image to be detected is input into the scene detection model to obtain the classification results of objects in the scene image and the classification results of the interaction relationship between objects and human body; when there is an object in the scene image that is classified as a mobile phone, and the classification result of the interaction relationship between human body and the object is a preset type of illegal mobile phone use behavior, then it is determined that there is illegal mobile phone use behavior.

[0008] The methods for constructing scene detection models include:

[0009] Construct a scene analysis model including a feature extraction module and first to third branches; and use the pre-defined prior anchor boxes for each human body. H Prior anchor frames b for each object O Combine them to form P unique binary anchor frames {b} H ,b O} i i = 1, 2, ..., P; P is a positive integer;

[0010] The scene images in the training set are input into the scene analysis model. The feature extraction module extracts the image features M of the input scene images, which are then input into the first to third branches respectively. In the first branch, the first decoder within it is based on {b H ,b O} i b H Obtain human features f in M i H and f i H Feature F after feature enhancement i H The first fully connected layer within it is mapped to the corresponding adjustment parameters to adjust {b}. H ,b O} i b H Adjustments are made to obtain the human body position detection result; in the second branch, the second decoder within it is based on {b H ,b O} i b H With b O The interactive region is used to obtain the interactive features f in M. iI and f i I Feature F after feature enhancement i I The second fully connected layer within it maps the interaction relationship between objects and the human body to a classification result; in the third branch, the third decoder within it is based on {b H ,b O} i b O Obtain object features f in M i O and f i O Feature F after feature enhancement i O The F is mapped to the object classification result through its third fully connected layer. i O The fourth fully connected layer within it is mapped to the corresponding adjustment parameters to adjust {b}. H ,b O} i b O Adjustments are made to obtain the object position detection result; where F i H F i I F i O To make f i H f i I f i O f respectively i H f i I f i O The characteristics of their interrelationships and the characteristics following M;

[0011] The scene analysis model is trained by minimizing the difference loss between the obtained human position detection results, interaction relationship classification results, object classification results, object position detection results and corresponding labels;

[0012] After training, a scene detection model is constructed, including a feature extraction module, a second branch, and a third branch.

[0013] More preferably, the scene analysis model also includes methods for obtaining F i H F i I F i OThe feature enhancement module is used to obtain F. i H F i I F i O .

[0014] The feature enhancement module is used to learn f i H f i I and f i O The characteristic of the pairwise binary interrelationship f i HO f i OI and f i HI , and f i H f i I and f i O The ternary interrelationship characteristics among the three elements f i HOI Based on the self-attention mechanism for f i H f i I and f i O Feature extraction is performed to obtain feature U. i Based on the cross-attention mechanism, feature U i Embedded into ternary mutual relationship features f i HOI In the middle, we obtain feature f i HOI′ Based on the self-attention mechanism for f i HO f i OI and f i HI Feature extraction is performed to obtain feature P. i Based on the cross-attention mechanism, feature P i Embedded into ternary mutual relationship features f i HOI In the middle, we obtain feature f i HOI″ ; will feature f i HOI′ with f i HOI″ Add them together to obtain the feature f. i HOI″′ Based on the cross-attention mechanism, feature f iHOI″′ Embedded into M, we obtain the multi-relation context features m. i ; respectively f i H f i I f i O Contextual features of multiple relationships m i By fusion, we obtain F. i H F i I F i O .

[0015] More preferably, in the feature enhancement module, an attention fusion module is used to respectively... i H f i I f i O Contextual features of multiple relationships m i By fusion, we obtain F. i H F i I F i O .

[0016] More preferably, in the feature enhancement module, a multilayer perceptron is used to learn f. i H f i I and f i O The characteristic of the pairwise binary interrelationship f i HO f i OI and f i HI , and f i H f i I and f i O The ternary interrelationship characteristics among the three elements f i HOI .

[0017] More preferably, the human body prior anchor frame b is recorded. H =(x H ,y H ,w H ,h H ), object prior anchor frame b O =(xO ,y O ,w O ,h O );(x H ,y H ) and (x O ,y O ) represent the center coordinates of the human body prior anchor frame and the object prior anchor frame, respectively; w H and h H These represent the width and height of the prior anchor frame for the human body, respectively; w O and h O These represent the width and height of the object's prior anchor frame, respectively;

[0018] b H With b O The interactive area is in,

[0019]

[0020] More preferably, the first branch is paired with b H After each adjustment is completed, the adjusted b will be re-evaluated. H The data is re-entered into the first branch for adjustment. After the first preset round of adjustment, the human body position detection result is obtained.

[0021] The third branch is for b O After each adjustment is completed, the adjusted b will be re-evaluated. O Re-enter the data into the third branch for adjustment until the third preset round of adjustment is completed, and obtain the object position detection result;

[0022] After each round of adjustments to the first and third branches, the adjusted b will also be... H and b O Re-enter into the second branch;

[0023] The first branch, the second branch, and the third branch execute each round of operations in parallel.

[0024] More preferably, the feature extraction module includes: a Swing Transformer and a deformable attention feature encoder;

[0025] The Swing Transformer is used to extract multi-scale features from the scene image, and the multi-scale features are concatenated. The concatenated features are then subjected to embedding position encoding and hierarchical encoding, and the encoded features are input into the deformable attention feature encoder to obtain image features M.

[0026] Secondly, the present invention provides a monitoring method, comprising: acquiring a monitoring video of a person to be monitored within a preset time period; using the detection method for illegal use of mobile phones provided in the first aspect of the present invention to detect each frame of the scene image of the monitoring video; counting the frequency of detected illegal use of mobile phones; and determining that the person to be monitored has ultimately engaged in illegal use of mobile phones when the frequency is greater than a preset frequency.

[0027] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method provided in the first or second aspect of the present invention.

[0028] Fourthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device in which the storage medium is located to perform the method provided in the first or second aspect of the present invention.

[0029] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0030] 1. This invention provides a method for detecting unauthorized mobile phone use. In the feature decoding stage, a first branch, a second branch, and a third branch are used for human detection, interaction detection, and object detection, respectively. Each branch decodes features f related to its respective task from image features using a corresponding decoder. i H f i I f i O In this process, anchor boxes are introduced to provide spatial priors, and the query decoding is guided by learning the distribution of binary anchor boxes in the training set; the decoding features of each branch are fused with f. i H f i I f i O The system utilizes the interrelationship features and image features to transmit rich, multi-contextual information between decoders, thereby enhancing features and accurately achieving the corresponding detection tasks. Specifically, the relationship branch constructs interaction regions based on human and object anchor boxes to obtain relationship information, ensuring that the region of interest of the relationship branch is aligned with the human and object branches. The entire process does not rely on sensors, enabling efficient and accurate detection of unauthorized mobile phone use at a low cost.

[0031] 2. Furthermore, the method for detecting unauthorized mobile phone use provided by this invention employs a feature enhancement module to learn fi H f i I and f i O The characteristics of pairwise binary interrelationships and f i H f i I and f i O The ternary interrelationship features among the three elements are then fused with the binary interrelationship features, ternary interrelationship features, and image features to analyze f. i H f i I f i O Enhancement is performed; among which, the ternary interrelationship features provide information about {b} H ,b O} i A holistic description of F, while the context of unary and binary relations provides more granular information, thus enabling the acquisition of F. i H F i I F i O It can transmit rich, multi-dimensional contextual features in the decoder, further improving the accuracy of detection.

[0032] 3. Furthermore, the method for detecting unauthorized mobile phone use provided by this invention employs a multilayer perceptron to learn f i H f i I and f i O The characteristic of the pairwise binary interrelationship f i HO f i OI and f i HI , and f i H f i I and f i O The ternary interrelationship characteristics among the three elements f i HOI Multilayer perceptron (MLP) is a non-linear approach that uses MLP to obtain binary and ternary cross-relation features, resulting in richer feature sets.

[0033] 4. Furthermore, the method for detecting unauthorized mobile phone use provided by this invention selects the center of the interaction area as the center of the line connecting the human body and the object frame. This causes the interaction area frame to shift towards the smaller target in the human body frame, and adaptively adjusts its width and height according to the distance between the human body and the object frame. When the distance between the human body and the object is greater, the size of the interaction area also increases accordingly. Through the above design, areas of the human body far from the interactive object are discarded, as this information has little effect on determining the interaction behavior. Instead, by adding some scene information around the object, the interaction behavior can be more accurately determined, improving the accuracy of detection.

[0034] 5. Furthermore, in the method for detecting unauthorized mobile phone use provided by this invention, the first branch targets b. H After each adjustment is completed, the adjusted b will be re-evaluated. H Re-enter the information into the first branch for adjustment until the first preset round of adjustment is completed; the third branch adjusts b. O After each adjustment is completed, the adjusted b will be re-evaluated. O Re-enter the data into the third branch for adjustment, until the third preset round of adjustment is completed; after each round of adjustment in the first and third branches, the adjusted value of b will also be updated. H and b O The data is then re-entered into the second branch; the first, second, and third branches execute each round of operations in parallel. By updating the coordinate information of the learned anchor boxes multiple times, the accuracy of detection is further improved.

[0035] 6. Furthermore, the method for detecting unauthorized mobile phone use provided by the present invention includes a feature extraction module comprising: a Swing Transformer and a deformable attention feature encoder; multi-scale features are extracted using the Swing Transformer model and feature encoding is performed through a deformable attention mechanism, which significantly improves the accuracy and efficiency of small target detection in complex industrial scenarios. Attached Figure Description

[0036] Figure 1 A schematic diagram illustrating the method for constructing a scene detection model according to an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the scene detection model is provided for embodiments of the present invention;

[0038] Figure 3 This is a schematic diagram of the feature enhancement module provided in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0040] To achieve the above objectives, in a first aspect, the present invention provides a method for detecting unauthorized mobile phone use, comprising:

[0041] The scene image to be detected is input into the scene detection model to obtain the classification results of objects in the scene image and the classification results of the interaction relationship between objects and human body; when there is an object in the scene image that is classified as a mobile phone, and the classification result of the interaction relationship between human body and the object is a preset type of illegal mobile phone use behavior, then it is determined that there is illegal mobile phone use behavior.

[0042] Among them, such as Figure 1 As shown, the methods for constructing the scene detection model include:

[0043] Construct a scene analysis model including a feature extraction module and first to third branches; and use the pre-defined prior anchor boxes for each human body. H Prior anchor frames b for each object O Combine them to form P unique binary anchor frames {b} H ,b O} i i = 1, 2, ..., P; P is a positive integer, and in one optional implementation, P is 64; here, the first branch corresponds to the human body branch, which performs the human body detection subtask; the second branch corresponds to the relationship branch, which performs the human interaction detection subtask; and the third branch corresponds to the object branch, which performs the object detection subtask.

[0044] The prepared training set includes: scene images with human bodies and objects, and the corresponding labels include: location labels of human bodies and objects, object category labels, and interaction relationship labels between objects and human bodies.

[0045] The scene images in the training set are input into the scene analysis model. The feature extraction module extracts the image features M of the input scene images, which are then input into the first to third branches respectively. In the first branch, the first decoder within it is based on {b H ,b O} i b H Obtain human features f in M i H and f i H Feature F after feature enhancement iH The first fully connected layer within it is mapped to the corresponding adjustment parameters to adjust {b}. H ,b O} i b H Adjustments are made to obtain the human body position detection result; in the second branch, the second decoder within it is based on {b H ,b O} i b H With b O The interactive region is used to obtain the interactive features f in M. i I and f i I Feature F after feature enhancement i I The second fully connected layer within it maps the interaction relationship between objects and the human body to a classification result; in the third branch, the third decoder within it is based on {b H ,b O} i b O Obtain object features f in M i O and f i O Feature F after feature enhancement i O The F is mapped to the object classification result through its third fully connected layer. i O The fourth fully connected layer within it is mapped to the corresponding adjustment parameters to adjust {b}. H ,b O} i b O Adjustments are made to obtain the object position detection result; where F i H F i I F i O To make f i H f i I f i O f respectively i H f i I f i O The characteristics of their interrelationships and the characteristics following M;

[0046] The scene analysis model is trained by minimizing the difference loss between the obtained human position detection results, interaction relationship classification results, object classification results, object position detection results and corresponding labels;

[0047] After training, a scene detection model is constructed, including a feature extraction module, a second branch, and a third branch.

[0048] In one alternative implementation, the first branch corresponds to b. H After each adjustment is completed, the adjusted b will be re-evaluated. H The data is re-entered into the first branch for adjustment. After the first preset round of adjustment (in one optional implementation, the value is 3), the human body position detection result is obtained.

[0049] The third branch is for b O After each adjustment is completed, the adjusted b will be re-evaluated. O Re-enter the data into the third branch for adjustment until the third preset round of adjustment (in one optional implementation, the value is 3) is completed, and the object position detection result is obtained;

[0050] After each round of adjustments to the first and third branches, the adjusted b will also be... H and b O Re-enter into the second branch;

[0051] The first branch, the second branch, and the third branch execute each round of operations in parallel.

[0052] In one optional implementation, the scene analysis model further includes a feature enhancement module; correspondingly, the scene detection model also includes a feature enhancement module.

[0053] The feature enhancement module is used to learn f i H f i I and f i O The characteristic of the pairwise binary interrelationship f i HO f i OI and f i HI , and f i H f i I and f i O The ternary interrelationship characteristics among the three elements f i HOI Based on the self-attention mechanism for f i H fi I and f i O Feature extraction is performed to obtain feature U. i Based on the cross-attention mechanism, feature U i Embedded into ternary mutual relationship features f i HOI In the middle, we obtain feature f i HOI′ Based on the self-attention mechanism for f i HO f i OI and f i HI Feature extraction is performed to obtain feature P. i Based on the cross-attention mechanism, feature P i Embedded into ternary mutual relationship features f i HOI In the process, we obtain feature f. i HOI″ ; will feature f i HOI′ with f i HOI″ Add them together to obtain the feature f. i HOI″′ Based on the cross-attention mechanism, feature f i HOI″′ Embedded into M, we obtain the multi-relation context features m. i ; respectively f i H f i I f i O Contextual features of multiple relationships m i By fusion, we obtain F. i H F i I F i O .

[0054] Preferably, in the feature enhancement module, an attention fusion module is used to respectively... i H f i I f i O Contextual features of multiple relationships m i By fusion, we obtain F. i H F i I F iO .

[0055] Preferably, in the feature enhancement module, a multilayer perceptron is used to learn f. i H f i I and f i O The characteristic of the pairwise binary interrelationship f i HO f i OI and f i HI , and f i H f i I and f i O The ternary interrelationship characteristics among the three elements f i HOI .

[0056] It should be noted that the Multilayer Perceptron (MLP) does not acquire binary cross-relation features f. i HO f i OI f i HI Based on the characteristics of the ternary interrelationship f i HOI The only other way is to obtain it by splicing, such as f i H and f i I splicing to obtain f i HI , will f i H f i I and f i O The three parts are spliced ​​together to obtain f. i HOI However, splicing is a linear method, while multilayer perceptron (MLP) is a non-linear method. By using MLP to obtain binary and ternary cross-relation features, the features are richer. Therefore, MLP is preferred for obtaining binary and ternary cross-relation features.

[0057] In one alternative implementation, the prior human anchor frame b is recorded. H =(x H ,y H ,w H ,h H ), object prior anchor frame bO =(x O ,y O ,w O ,h O );(x H ,y H ) and (x O ,y O ) represent the center coordinates of the human body prior anchor frame and the object prior anchor frame, respectively; w H and h H These represent the width and height of the prior anchor frame for the human body, respectively; w O and h O These represent the width and height of the object's prior anchor frame, respectively;

[0058] b H With b O The interactive area is in,

[0059] It should be noted that the anchor frame can be represented in more than one way; it can also be composed of the coordinates of the top-left and bottom-right corners. Similarly, the interaction area can be obtained in more than one way; it can also be represented by a minimum bounding matrix that includes both the human body and the object. However, this design only focuses on the information of the object and human body regions, ignoring the surrounding scene information; and there are errors between the prior human body bounding box and the object bounding box, especially for smaller objects, where the error between the prior anchor frame and the actual bounding box is usually larger. Therefore, this invention preferably uses the above method to determine the interaction area, selecting the center of the interaction area as the center of the line connecting the human body and the object bounding box, causing the interaction area bounding box to shift towards the smaller target in the human body, and adaptively adjusting its width and height according to the distance between the human body and the object bounding box. When the distance between the human body and the object is greater, the size of the interaction area also increases accordingly. Through this design, the areas of the human body far from the interactive object are discarded, as this information has little effect on determining the interaction behavior. Conversely, by adding some scene information around the object, the interaction behavior can be determined more accurately.

[0060] In one alternative implementation, the feature extraction module includes: a Swing Transformer and a deformable attention feature encoder;

[0061] The SwinTransformer is used to extract multi-scale features from the scene image, and the multi-scale features are concatenated. The concatenated features are then subjected to embedding position encoding and hierarchical encoding, and the encoded features are input into the deformable attention feature encoder to obtain image features M.

[0062] It should be noted that the above feature extraction module is only one preferred feature extraction module. It can also be a traditional CNN feature extraction module, such as ResNet, DenseNet, EfficientNet, MobileNet, etc.

[0063] In summary, the proposed method for detecting unauthorized mobile phone use employs three parallel decoders in its feature decoding stage for human detection, object detection, and interaction relationship detection, respectively. Each branch uses randomly initialized learnable content queries, and the decoder decodes features relevant to its respective task from visual memory. Simultaneously, anchor boxes are introduced to provide spatial priors, and the distribution of HOI instances in the training set is learned to guide query decoding. Specifically, the human and object branches use randomly initialized learnable anchor boxes, which are updated after the decoder. The relationship branch obtains relationship information based on the interaction regions constructed from the human and object anchor boxes, ensuring that the region of interest for the relationship branch aligns with that of the human branch, effectively guiding the decoding of the relationship branch and reducing the learning difficulty. Based on this, after the decoder, the features from the three branches are fused together. i H f i I f i O The interrelationship features and image features extracted by the feature extraction module are used for feature enhancement, thereby transmitting rich multi-contextual information between decoders and accurately achieving the corresponding detection task.

[0064] Secondly, the present invention provides a monitoring method, comprising: acquiring a monitoring video of a person to be monitored within a preset time period; using the detection method for illegal use of mobile phones provided in the first aspect of the present invention to detect each frame of the scene image of the monitoring video; counting the frequency of detected illegal use of mobile phones; and determining that the person to be monitored has ultimately engaged in illegal use of mobile phones when the frequency is greater than a preset frequency.

[0065] In one alternative implementation, an alarm is triggered when it is determined that the person being monitored has engaged in unauthorized use of a mobile phone.

[0066] The relevant technical solutions are the same as the detection method for unauthorized use of mobile phones provided in the first aspect of this invention, and will not be described in detail here.

[0067] To further illustrate the detection and monitoring method for unauthorized mobile phone use provided by this invention, a specific embodiment is described in detail below:

[0068] like Figure 2 The diagram shown is a schematic representation of the scene detection model in this embodiment:

[0069] The feature extraction module includes a Swin Transformer and a deformable attention feature encoder. The Swin Transformer is used as the backbone network to extract multi-scale features from the image, constructing a four-layer feature map. These layers are then concatenated and subsequently embedded with positional and hierarchical encodings. The deformable attention encoder is used to extract visual memory in a multi-scale manner. The formula for deformable attention is shown below:

[0070]

[0071] To clarify the decoding objective and reduce the learning difficulty of deformable attention sampling points, the feature decoding stage uses three parallel decoders for human detection, object detection, and interaction category detection, respectively. Each branch uses randomly initialized learnable content queries and decodes features relevant to its respective task from visual memory through a multi-layer deformable attention decoder. Simultaneously, four-dimensional anchor boxes are introduced into the decoders to provide spatial priors by learning the instance distribution of the dataset, guiding query decoding. The human and object branches each use randomly initialized learnable anchor boxes, which are iteratively updated after each decoder layer. The relationship branch uses interaction regions constructed based on the human and object anchor boxes as anchor boxes. This ensures that the attention region of the relationship branch is aligned with the human branch, and the manually set prior interaction region anchor boxes effectively guide the decoding of the relationship branch, reducing the learning difficulty.

[0072] Given a character frame b H =(x H ,y H ,w H ,h H ) and object frame b O =(x O ,y O ,w O ,h O ), where (x H ,y H ),(x O ,y O The center of the character frame and the center of the object frame are represented by ), then the interactive area frame is defined as:

[0073]

[0074]

[0075] Furthermore, to address the global feature loss issue caused by deformable attention, after each layer of the decoder, the features from the three branches are passed through a feature enhancement module to transmit rich, multi-layered contextual information between decoders. The structure of the feature enhancement module is as follows: Figure 3 As shown:

[0076] The feature enhancement module takes the features output from the three decoders as input, given the i-th tuple anchor box {b H ,b O} i (Dedicated to the HOI instance) The three features corresponding to the query are respectively

[0077] In this embodiment, the method for obtaining the corresponding features in image features M based on anchor boxes is as follows: the encoder queries with learnable content. Four-dimensional prior anchor frame and image features obtained from the feature extraction module As input, the human body prior anchor boxes and object prior anchor boxes are randomly initialized by the model and learned from the training set. For a given i-th anchor box... Location lookup:

[0078] P i =MLP(PE(x) i ); PE(y i ); PE(w i ); PE(h i ));

[0079] Where PE is a sinusoidal positional encoding function, used to map floating-point coordinates into a vector of dimension D / 2. The symbol represents a concatenation operation, concatenating the positional encoding of the coordinate values ​​into a 2×D vector, which is then passed through a multilayer perceptron (MLP) to convert its dimension to the same dimension D as the content query. Standard self-attention is performed on the positional query P and the content query C to obtain C'. In the cross-attention stage, key features are decoded from the image features M. Preferably, the cross-attention stage uses a deformable attention mechanism, specifically, f i =MSDefAttn(C i ',A i M).

[0080] First, each unary feature is concatenated, and then a multilayer perceptron is used to construct the ternary context features corresponding to the i-th HOI instance.

[0081] f i HOI =MLP([f i H ;f i O ;f i I ])

[0082] Here, [;] represents a concatenation operation. Ternary relation Considering the overall understanding of each subtask, it provides holistic contextual information about the HOI instance. On the other hand, unary and binary relations consider a fine-grained understanding of each subtask, providing fine-grained contextual information about the HOI instance. To leverage both holistic and fine-grained contextual information, It will aggregate key information that is effective in understanding the overall task from both unary and binary contexts.

[0083] A set of unary features corresponding to queries for i HOI instances. Perform self-attention operations, and then use cross-attention operations to embed the unary relation context into the ternary relation context:

[0084] U i =SelfAttn(f i H ,f i O ,f i I )

[0085] f i HOI′ =CrossAttn(f i HOI U i )

[0086] For simplicity, SelfAttn(·) is denoted as the self-attention operation, and CrossAttn(q,kv) is denoted as the cross-attention operation.

[0087] To embed a binary relation context into a ternary relation context, a binary context is first generated using unary feature sets, and then extracted... These correspond to binary contexts for human-object, human-interaction, and object-interaction, respectively.

[0088] f i HO =MLP([f i H ;f i O ])

[0089] f i OI =MLP([f i O ;f i I ])

[0090] f i HI =MLP([f i H ;fi I ])

[0091] Similar to the process described above, self-attention is applied to a set of binary features, and cross-attention is used to embed the binary context into the ternary relation context:

[0092] P i =selfAttn(f i HO ,f i OI ,f i HI )

[0093] f i HOI″ =crossAttn(f i HOI ,P i )

[0094] f after univariate feature enhancement i HOI′ and the enhanced binary features of f i HOI″ Adding them together gives f i HOI″′ .

[0095] f i HOI″′ =f i HOI′ +f i HOI″

[0096] At this time f i HOI″′ It also aggregates the unary, binary, and ternary contexts of HOI instances, providing a complete understanding of HOI tasks, and then uses f... i HOI″′ Generate a multi-relational context m from visual memory M. i This can be achieved based on a deformable attention mechanism:

[0097] m i =MSDefAttn(f i HOI″′ ,p i M)

[0098] Wherein, reference point p iThe midpoint is set as the line connecting the center points of the human body and the object bounding box. MSDefAttn is a deformable attention formula. It's worth noting that a nonlinear transformation, i.e., a multilayer perceptron (MLP), is used when constructing higher-order features (ternary and binary context features), rather than reducing them to a linear summation of multiple lower-order features. Such higher-order feature functions can simultaneously consider all inputs and learn structural relationships between multiple inputs. For example, a cubic function f(a,b,c) with three coordinates as input can yield the angular features between ab and ac, which cannot be obtained through univariate functions g(a), g(b), g(c) and their linear combinations. Similarly, ternary context features can effectively capture structural relationships that are not easily combined from univariate and binary feature functions.

[0099] Finally, the extracted multi-relation context features m need to be processed. i The information is integrated into three branches. Considering that each subtask requires different contextual information during relational reasoning, an attention fusion module (AFM) is introduced. This module can filter out the necessary information for each subtask from multiple relational contexts and merge it into their respective branches, thereby ensuring that each decoder can efficiently complete the decoding work based on a full understanding of the task requirements. The feature fusion calculation process is as follows:

[0100] k = sigmoid(MLP([m i ,f i ]))

[0101] F i =f i +k×MLP([m i ,f i ])

[0102] A weight value k is obtained through a sigmoid function and an MLP network, where k represents m i For feature f i The importance of m is determined, and then weighted by k. i Converging to f i The AFM module is applied to all three branches simultaneously, ultimately yielding the output of the feature enhancement module.

[0103] Finally, using the FFN fully connected layer, the human bounding box, object bounding box and category, and interaction relationship category are obtained from the human body branch, object branch and relationship branch, respectively.

[0104] During the training phase, a binary matching process is performed between the prediction set and the real set, and the model is trained using a quinary loss function. During the inference phase, a HOI-aware triplet nonmaximum suppression strategy is used to filter out duplicate character interaction prediction instances.

[0105] It should be noted that the preset types of unauthorized mobile phone use behaviors are set according to specific scenarios and task requirements. For example, in a factory workshop, operating a mobile phone under a table or watching videos on the center console while driving a vehicle are considered unauthorized mobile phone use behaviors. In this embodiment, through analysis and statistics of personnel behavior categories in a real duty room scenario, three detection categories with typical characteristics were designed: staff making phone calls, operating mobile phones, and staring at mobile phones. Making phone calls refers to a person holding a mobile phone and making a call. Operating mobile phones refers to a person using their fingers to operate the mobile phone, such as typing or swiping. Staring at mobile phones refers to a person placing the mobile phone on a table, without the hand directly touching the mobile phone, but with their face staring at the mobile phone for a long time, which is usually a person watching dramas or reading novels. In order to enable the model to detect personnel behavior from various angles and postures, when defining these three categories and collecting data, category examples from different angles facing the monitoring camera were included.

[0106] The process for determining whether a person has violated regulations by using a mobile phone is as follows:

[0107] If an object in the scene image is classified as a mobile phone, and the interaction between the human body and the object is classified as a pre-defined type of unauthorized mobile phone use, then unauthorized mobile phone use is determined to exist. For example, in a certain scene, if the interaction between the human body and the mobile phone is classified as "operation" or "gazing", then unauthorized mobile phone use is determined to exist.

[0108] Furthermore, this embodiment also provides a monitoring method. Upon detecting a potential violation of mobile phone usage rules, this embodiment does not immediately trigger an alarm. Instead, it enters a timing processing module to continue tracking the individual and analyzes video frames over a subsequent period. The data is then accumulated, recording the detection results for each frame within a certain timeframe. Frequency statistics are then performed to calculate the frequency of detected violations of mobile phone usage rules during this period. A threshold is set (e.g., 50%); only when the detection frequency exceeds the threshold is it determined to be an actual violation of mobile phone usage rules.

[0109] Alarm Trigger: When the comprehensive analysis results exceed the set threshold, the system will trigger an alarm and record relevant information (such as time, location, and duration) for subsequent processing.

[0110] By following the steps above, occasional false alarms can be effectively filtered out, improving the accuracy of identifying unauthorized mobile phone use. Experimental results show that this method performs excellently in terms of precision, recall, and F1-measure, making it suitable for industrial monitoring environments.

[0111] In a second aspect, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method provided in the first or second aspect of the present invention.

[0112] The relevant technical solutions are the same as the detection method for illegal use of mobile phones provided in the first aspect of this invention and the monitoring method provided in the second aspect, and will not be described in detail here.

[0113] Thirdly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device in which the storage medium is located to perform the method provided in the first or second aspect of the present invention.

[0114] The relevant technical solutions are the same as the detection method for illegal use of mobile phones provided in the first aspect of this invention and the monitoring method provided in the second aspect, and will not be described in detail here.

[0115] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting unauthorized mobile phone use, characterized in that, include: The scene image to be detected is input into the scene detection model to obtain the classification results of objects in the scene image and the classification results of the interaction relationship between objects and the human body; When there is an object in the scene image that is classified as a mobile phone, and the interaction between the human body and the object is classified as a preset type of illegal mobile phone use, then it is determined that there is illegal mobile phone use. The method for constructing the scene detection model includes: Construct a scene analysis model including a feature extraction module and first to third branches; and use the pre-defined prior anchor boxes for each human body. H Prior anchor frames b for each object O Combine them to form P unique binary anchor frames {b} H ,b O } i i = 1, 2, ..., P; P is a positive integer; The scene images in the training set are input into the scene analysis model. The feature extraction module extracts the image features M of the input scene images and inputs them into the first to third branches respectively. In the first branch, the first decoder within it is based on {b H ,b O } i b H Obtain human features f in M i H and f i H Feature F after feature enhancement i H The first fully connected layer within it is mapped to the corresponding adjustment parameters to adjust {b}. H ,b O } i b H Adjustments are made to obtain the human body position detection result; in the second branch, the second decoder within it is based on {b H ,b O } i b H With b O The interactive region is used to obtain the interactive features f in M. i I and f i I Feature F after feature enhancement i I The second fully connected layer within it maps the interaction relationship between objects and the human body to a classification result; in the third branch, the third decoder within it is based on {b H ,b O } i b O Obtain object features f in M i O and f i O Feature F after feature enhancement i O The F is mapped to the object classification result through its third fully connected layer. i O The fourth fully connected layer within it is mapped to the corresponding adjustment parameters to adjust {b}. H ,b O } i b O Adjustments are made to obtain the object position detection result; where F i H F i I F i O To make f i H f i I f i O f respectively i H f i I f i O The characteristics of their interrelationships and the characteristics following M; The scene analysis model is trained by minimizing the difference loss between the obtained human position detection results, interaction relationship classification results, object classification results, object position detection results and corresponding labels. After training, the scene detection model, including a feature extraction module, a second branch, and a third branch, is constructed.

2. The method for detecting unauthorized mobile phone use according to claim 1, characterized in that, The scenario analysis model also includes methods for obtaining F. i H F i I F i O Feature enhancement module; The feature enhancement module is used to learn f i H f i I and f i O The characteristic of the pairwise binary interrelationship f i HO f i OI and f i HI , and f i H f i I and f i O The ternary interrelationship characteristics among the three elements f i HOI Based on the self-attention mechanism for f i H f i I and f i O Feature extraction is performed to obtain feature U. i Based on the cross-attention mechanism, feature U i Embedded into ternary mutual relationship features f i HOI In the middle, we obtain feature f i HOI′ ; Based on the self-attention mechanism, f i HO f i OI and f i HI Feature extraction is performed to obtain feature P. i Based on the cross-attention mechanism, feature P i Embedded into ternary mutual relationship features f i HOI In the process, we obtain feature f. i HOI″ ; will feature f i HOI′ with f i HOI″ Add them together to obtain the feature f. i HOI″′ Based on the cross-attention mechanism, feature f i HOI″′ Embedded into M, we obtain the multi-relation context features m. i ; respectively f i H f i I f i O Contextual features of multiple relationships m i By fusion, we obtain F. i H F i I F i O .

3. The method for detecting unauthorized mobile phone use according to claim 2, characterized in that, In the feature enhancement module, an attention fusion module is used to respectively... i H f i I f i O Contextual features of multiple relationships m i By fusion, we obtain F. i H F i I F i O .

4. The method for detecting unauthorized mobile phone use according to claim 2, characterized in that, In the feature enhancement module, a multilayer perceptron is used to learn f. i H f i I and f i O The characteristic of the pairwise binary interrelationship f i HO f i OI and f i HI , and f i H f i I and f i O The ternary interrelationship characteristics among the three elements f i HOI .

5. The method for detecting unauthorized mobile phone use according to any one of claims 1-4, characterized in that, Describe the prior anchor frame b of the human body H =(x H ,y H ,w H ,h H ), object prior anchor frame b O =(x O ,y O ,w O ,h O );(x H ,y H ) and (x O ,y O ) represent the center coordinates of the human body prior anchor frame and the object prior anchor frame, respectively; w H and h H These represent the width and height of the prior anchor frame for the human body, respectively; w O and h O These represent the width and height of the object's prior anchor frame, respectively; b H With b O The interactive area is in, 6. The method for detecting unauthorized mobile phone use according to any one of claims 1-4, characterized in that, The first branch is b H After each adjustment is completed, the adjusted b will be re-evaluated. H The data is re-entered into the first branch for adjustment. After the first preset round of adjustment, the human body position detection result is obtained. The third branch is for b O After each adjustment is completed, the adjusted b will be re-evaluated. O Re-enter the data into the third branch for adjustment until the third preset round of adjustment is completed, and obtain the object position detection result; After each round of adjustments to the first and third branches, the adjusted b will also be... H and b O Re-enter into the second branch; The first branch, the second branch, and the third branch execute each round of operations in parallel.

7. The method for detecting unauthorized mobile phone use according to any one of claims 1-4, characterized in that, The feature extraction module includes: Swing Transformer and Deformable Attention Feature Encoder; The Swin Transformer is used to extract multi-scale features from the scene image, and the multi-scale features are concatenated. The concatenated features are then subjected to embedding position encoding and hierarchical encoding, and the encoded features are input into the deformable attention feature encoder to obtain the image feature M.

8. A monitoring method, characterized in that, include: The monitoring video of a person to be monitored within a preset time period is acquired. The method for detecting unauthorized mobile phone use as described in any one of claims 1-7 is used to detect each frame of the scene image in the monitoring video, and the frequency of detected unauthorized mobile phone use is counted. When the frequency is greater than a preset frequency, it is determined that the person to be monitored has ultimately engaged in unauthorized mobile phone use.

9. An electronic device, characterized in that, include: A memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the method according to any one of claims 1-8.

10. A computer-readable storage medium comprising a stored computer program, wherein, When the computer program is run by the processor, it controls the device containing the storage medium to perform the method described in any one of claims 1-8.

Images