Artificial intelligence character interaction detection method and device based on self-attention mechanism

Abstract

The invention provides an artificial intelligence character interaction detection method and device based on a self-attention mechanism. A detection model is pre-trained based on a preset sample data set; performing training fine tuning on the pre-trained detection model by using an actually used data set; inputting a to-be-detected image into the detection model, and obtaining all triples output by the detection model; and filtering out triads which do not meet a preset condition, and taking the reserved triads as detection results. According to the detection model used by the invention, the human body decoder is adopted to independently detect human body position information, the object decoder is adopted to independently detect object position information and object types, and the action decoder is adopted to independently detect action types, so that human body detection, object detection and action detection do not interfere with one another; and moreover, the actions can be better classified, and the accuracy of target detection is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular to an artificial intelligence human interaction detection method and device based on a self-attention mechanism. BACKGROUND

[0002] In actual engineering applications, target detection algorithms based on neural networks can effectively identify and locate potential safety hazards, such as detecting suspicious persons carrying knives, thereby triggering an alarm mechanism to reduce safety risks. However, in some complex scenarios, simple human and object detection alone cannot meet the actual needs. Taking the detection of the behavior of "playing a mobile phone" as an example, simply detecting the presence of a person and a mobile phone cannot accurately determine the behavior of "playing a mobile phone", which requires meeting three conditions simultaneously: detecting a person, detecting a mobile phone, and establishing a correlation between the two, and also accurately identifying the hand action features of the person.

[0003] To address this challenge, human-object interaction (HOI) detection algorithms have emerged. However, current mainstream algorithms such as QPIC (Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information) still have obvious limitations in performance. The main difficulty lies in the fact that such algorithms need to simultaneously complete the tasks of person detection, object detection, and action recognition in a single inference process, and the person and object detection may interfere with each other, and the action recognition is also easily disturbed by the person and object detection, resulting in poor detection accuracy. For example, the existing QPIC algorithm in detecting the behavior of "playing a mobile phone" may cause confusion between "holding a mobile phone" and "looking at a mobile phone" by directly splicing person / object features. This technical bottleneck is directly reflected in the evaluation results on the public dataset HICO-DET - the average precision (mAP) of the QPIC algorithm is only 0.29, indicating that existing methods still have a large room for improvement in complex interaction behavior recognition. SUMMARY

[0004] The present application provides an artificial intelligence human interaction detection method and device based on a self-attention mechanism, to solve the defect in the prior art that the tasks of person detection, object detection, and action recognition need to be completed simultaneously in a single inference process, which easily causes mutual interference and results in poor detection accuracy, and to achieve more accurate person detection, object detection, and action recognition.

[0005] The application provides an artificial intelligence character interaction detection method based on a self-attention mechanism, comprising:

[0006] Pre-training a detection model based on a preset sample data set;

[0007] Training and fine-tuning the pre-trained detection model using an actual use data set;

[0008] Inputting a to-be-detected image into the detection model and obtaining all triplets output by the detection model; the detection model comprises a backbone network for extracting image features, an image encoder for constructing reference frames, a human decoder for independently detecting human position information, an object decoder for independently detecting object position information and categories, and an action decoder for independently detecting action categories;

[0009] Filtering out triplets that do not meet preset conditions and retaining the triplets as detection results.

[0010] According to the artificial intelligence character interaction detection method based on the self-attention mechanism, the detection model processes the to-be-detected image, comprising:

[0011] The backbone network performs M times of downsampling on the to-be-detected image to generate M feature maps of different scales, extracts image features at different positions of the M feature maps, and inputs the image features and position information corresponding to the image features into the image encoder;

[0012] The image encoder constructs encoded image features and determines the positions of corresponding reference frames according to the image features and the position information corresponding to the image features, calculates the probabilities that the image features and the corresponding reference frames are foreground, and groups N image features and corresponding reference frames with the highest probabilities to form N queries and input the queries into the human decoder and the object decoder respectively; the reference frame is an 8-dimensional vector, the first 4 dimensions are human reference frames, and the last 4 dimensions are object reference frames;

[0013] The human decoder constructs human feature encoding according to the image features in the N queries, fine-tunes the corresponding human reference frames based on the human feature encoding, and obtains human detection frames;

[0014] The object decoder constructs object feature encoding according to the image features in the N queries, fine-tunes the corresponding object reference frames based on the object feature encoding, and obtains object detection frames, and determines the object categories and the probabilities of the object categories through the fully connected network respectively;

[0015] The action decoder determines an action category corresponding to the joint feature code and the joint reference frame and a probability of the action category according to the joint feature code and the joint reference frame; the joint feature code is obtained by adding the human feature code and the object feature code, and the joint reference frame is obtained by splicing the human reference frame and the object reference frame.

[0016] According to the artificial intelligence human interaction detection method based on the self-attention mechanism, the triplets include the human detection frame, the object detection frame, the object category, and the action category.

[0017] According to the artificial intelligence human interaction detection method based on the self-attention mechanism, the backbone network of the detection model is a swin-large network.

[0018] According to the artificial intelligence human interaction detection method based on the self-attention mechanism, the triplets that do not meet the preset condition are filtered out, including:

[0019] The product of the probability corresponding to the object category and the probability corresponding to the action category in the triplet is taken as a confidence degree, and the triplet with a confidence degree less than a first preset threshold is filtered out.

[0020] According to the artificial intelligence human interaction detection method based on the self-attention mechanism, after the triplets with a confidence degree less than a preset threshold are filtered out, the method further includes:

[0021] The triplet with the highest confidence degree is taken as a reference triplet, the overlap degrees of other triplets and the reference triplet are calculated, and the triplet with an overlap degree greater than a second preset threshold is filtered out.

[0022] According to the artificial intelligence human interaction detection method based on the self-attention mechanism, the preset sample data set includes an object365 data set with pseudo-labels, part of the annotation data in a VisualGenome data set, and a COCO data set.

[0023] The application further provides an artificial intelligence human interaction detection device based on a self-attention mechanism, including:

[0024] A pre-training module is configured to pre-train a detection model based on a preset sample data set;

[0025] A transfer learning module is configured to train and fine-tune the pre-trained detection model by using an actually used data set.

[0026] An image detection module is configured to input a to-be-detected image into the detection model and obtain all triplets output by the detection model; the detection model comprises a backbone network configured to extract image features, an image encoder configured to construct a reference frame, a human decoder configured to independently detect human position information, an object decoder configured to independently detect object position information and object categories, and an action decoder configured to independently detect action categories.

[0027] A post-processing module is configured to filter out triplets that do not meet preset conditions and retain the triplets as detection results.

[0028] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the artificial intelligence human interaction detection method based on a self-attention mechanism according to any one of the above when executing the program.

[0029] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the artificial intelligence human interaction detection method based on a self-attention mechanism according to any one of the above.

[0030] The application provides an artificial intelligence human interaction detection method and device based on a self-attention mechanism, which pre-trains a detection model based on a preset sample data set, fine-tunes the pre-trained detection model using an actual data set, inputs a to-be-detected image into the detection model, and obtains all triplets output by the detection model; the detection model comprises a backbone network configured to extract image features, an image encoder configured to construct a reference frame, a human decoder configured to independently detect human position information, an object decoder configured to independently detect object position information and object categories, and an action decoder configured to independently detect action categories; and triplets that do not meet preset conditions are filtered out, and the retained triplets are taken as detection results. The application independently detects human position information using a human decoder, independently detects object position information and object categories using an object decoder, and independently detects action categories using an action decoder, so that the human, object, and action detection do not interfere with each other, the actions can be better classified, and the accuracy of target detection is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0032] Figure 1 This is a flowchart illustrating the AI-based human interaction detection method based on a self-attention mechanism provided by the present invention.

[0033] Figure 2 This is a schematic diagram of the processing flow of the detection model for the image to be detected in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the structure of the AI ​​human interaction detection device based on the self-attention mechanism provided by the present invention.

[0035] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0037] The following is combined Figures 1 to 2 This invention describes an AI-based human interaction detection method based on a self-attention mechanism.

[0038] like Figure 1 As shown, the AI-based character interaction detection method based on self-attention mechanism provided by this invention includes the following steps:

[0039] S1. Pre-train the detection model based on a pre-set sample dataset.

[0040] The first stage of model training involves pre-training on a large-scale dataset, which is the foundation for building a high-performance object detection model. Public datasets for human interaction detection are relatively few; currently, the most common are HICO-DET, v-coco, and VG datasets, with data volumes ranging from several thousand to tens of thousands (some have over 100,000, but the annotation quality is very poor). Therefore, in an optional embodiment of this invention, the pre-defined sample dataset includes the pseudo-labeled object365 dataset, a portion of the labeled data from the Visual Genome dataset, and the Common Objects in Context dataset. Object365 is an object detection dataset containing approximately 2 million images of objects across 365 categories; the COCO dataset is one of the most commonly used object detection datasets, with the detection portion containing approximately 200,000 images across 80 categories; the Visual Genome dataset, proposed by Stanford in 2016 as a Language and Vision dataset, aims to better understand the world. It annotates a large number of objects and relationships in images, as well as question-and-answer pairs about the images. The annotations can be used for HOI tasks; it contains approximately 110,000 images with numerous action categories, but the annotation quality is relatively low. This invention only uses a portion of the annotations. Pseudo-labeling technology plays a crucial role in this stage. Since the annotation standards and category systems of different datasets may differ, pseudo-labeling technology can help unify the annotation format and improve data utilization.

[0041] S2. Use the actual dataset to train and fine-tune the pre-trained detection model.

[0042] The pre-trained model needs to undergo a transfer learning process to adapt to a specific task or domain. The purpose of this step is to perform transfer learning on the detection model. In an optional embodiment of the present invention, the pre-trained model is trained and fine-tuned on real-world data, such as the public dataset HICO-DET or some actual detection project data. HICO-DET is one of the most commonly used datasets in the HOI domain, containing 47,776 images, 80 object classes, 117 action classes, and 600 verb-noun combinations (objects and actions are paired, but some combinations do not actually exist). Compared to another commonly used dataset, VCOCO, it is much richer in both the number of images and action categories (VCO contains 10,346 images, 80 object classes, and 29 action classes), making it very suitable for fine-tuning triplet detection tasks. Therefore, the present invention chooses the HICO-DET dataset as the main evaluation metric. During the fine-tuning process, the model learns to adapt to the specific feature distribution of the target domain. For example, in real-world application scenarios, lighting conditions, object frequency, and interaction types may differ significantly from the pre-training dataset. Compared to the detection algorithms based on detr commonly used in existing technologies, the approach adopted in this invention uses transfer learning on large-scale datasets to enable the model to adjust its parameters and better adapt to these specific conditions, thus making the model more robust on small datasets.

[0043] S3. Input the image to be detected into the detection model and obtain all the triples output by the detection model.

[0044] After pre-training and fine-tuning, the model can perform triple detection tasks for images. Specifically, such as... Figure 2 As shown, the detection model includes a backbone network for extracting image features, an image encoder for constructing reference boxes, a human body decoder for independently detecting human body position information, an object decoder for independently detecting object position information and class, and an action decoder for independently detecting action classes.

[0045] Specifically, such as Figure 2 As shown, after the image to be detected is input into the detection model, the processing flow of the detection model for the image to be detected is as follows:

[0046] S301: The backbone network performs M downsampling on the image to be detected, generating M feature maps of different scales. Image features are extracted from different locations of the M feature maps, and the image features and their corresponding location information are input into the image encoder.

[0047] Specifically, the backbone network is responsible for extracting features from the image. After the image enters the backbone network, it undergoes multiple downsampling processes, generating multiple feature maps at different scales. In an optional embodiment of the invention, the backbone network performs four stages of progressive downsampling, generating four feature maps at different scales (e.g., ...). Figure 2 The small square blocks in the text should be understood as follows: Figure 2 The number of feature maps in the image is not the actual number; depending on the size of the input image, this step generates thousands or even tens of thousands of feature maps. Features are then extracted from different locations on the four feature maps at different scales and fed into the next-level image encoder. Preserving multi-scale features is beneficial for detecting targets of different sizes and avoids ignoring smaller targets.

[0048] In an optional embodiment of the present invention, the backbone network of the detection model is a swin-large network. The window attention mechanism of the swin-large network can balance computational efficiency and global modeling ability, making it suitable for dense detection tasks. In addition, its hierarchical design can efficiently handle targets of different scales (such as small objects in the distance and large objects nearby), enabling the detection model to have strong detection capabilities for targets of different sizes. It should be understood that the swin-large network is a preferred backbone network of the present invention, but it is not limited to using only this type of backbone network. For example, it can also be replaced with convolutional or transformer neural networks such as ResNet50 and swin-tiny.

[0049] S302: The image encoder constructs encoded image features based on image features and the corresponding position information of the image features, determines the position of the corresponding reference box, calculates the probability that the image features and the corresponding reference box are foreground, and combines the N image features with the highest probability and the corresponding reference boxes into N queries, which are then input into the human body decoder and the object decoder respectively.

[0050] Specifically, the image encoder in this invention is a transformer structure, which includes a deformable self-attention mechanism and has a global receptive field, enabling global contextual relationship modeling. The image encoder in this invention has a multi-layer structure (preferably 6 layers, but this can be modified). After passing through the image encoder, encoded image features (this invention does not limit the specific feature dimensions and can be flexibly selected according to actual conditions, preferably 256 dimensions) and the positions of the corresponding reference boxes, as well as the probability that the image features and reference boxes at that location are foreground elements, can be obtained. The reference box is an 8-dimensional vector, with the first 4 dimensions representing the human body reference box and the last 4 dimensions representing the object reference box. For example, the reference box can be represented as (x1, y1, w1, h1, x2, y2, w2, h2), where the first four dimensions (x1, y1, w1, h1) represent the human body reference box, and the last four dimensions (x2, y2, w2, h2) represent the object reference box, where (x, y) represents the center coordinates of the reference box, and w and h represent the width and height of the reference box, respectively. By selecting the N image features with the highest probability of being foreground and their corresponding reference boxes to form N queries, the interference of irrelevant background in the image on the target to be detected can be reduced.

[0051] In an optional embodiment of the present invention, the 900 most probable image features and their corresponding reference boxes are combined to form 900 queries, which are then input into the human body decoder and the object decoder, respectively.

[0052] S303: The human body decoder constructs human body feature codes based on the image features in N queries, and fine-tunes the corresponding human body reference boxes based on the human body feature codes to obtain human body detection boxes.

[0053] Specifically, the human body decoder used in this invention is also a multi-layer transform structure (preferably 6 layers, but this can be modified). Compared to the image encoder, the human body decoder, in addition to including a deformable self-attention mechanism, also has a cross-attention mechanism module, which can more effectively model the context of the current generation location. In an optional embodiment of this invention, the input of the human body decoder is the image feature encoding and human reference box (x1, y1, w1, h1) from 900 queries. After passing through the human body decoder, the resulting human feature encoding (this invention does not limit the specific feature dimension and can be flexibly selected according to the actual situation, preferably 256 dimensions) and 4-dimensional human reference box are obtained. The reference box is fine-tuned in combination with the human features to finally obtain the human body detection box (i.e., the human body's position information), completing the detection of the human body in the HOI triplet.

[0054] S304: The object decoder constructs object feature codes based on the image features in N queries, fine-tunes the corresponding object reference boxes based on the object feature codes to obtain object detection boxes, and determines the object category and the probability of the object category through a fully connected network.

[0055] Specifically, the object decoder in this invention is structurally identical to the human body decoder. In an optional embodiment of this invention, the input to the object decoder is the image feature encoding and object reference boxes (x2, y2, w2, h2) from 900 queries, where the image feature encoding is consistent with the input to the human body decoder. The object reference boxes are fine-tuned by combining image features to finally obtain the object detection boxes (i.e., the object's location information). Furthermore, the object decoder also includes a fully connected network through which object features are used to classify objects, employing a softmax cross-entropy loss function to obtain the object's category and complete the object detection in the HOI triplet.

[0056] It should be understood that there is no specific order of execution between the above steps S303 and S304. They can be executed simultaneously or sequentially. This invention does not impose any restrictions on this.

[0057] S305: The action decoder determines the action category and the probability of the action category corresponding to the joint feature code and the joint reference box based on the joint feature code and the joint reference box; the joint feature code is obtained by adding the human feature code and the object feature code, and the joint reference box is obtained by concatenating the human reference box and the object reference box.

[0058] In an optional embodiment of the present invention, the action decoder structure is similar to the object decoder structure described above, but the number of layers is reduced from 6 to 3, which improves efficiency. The feature codes of the human body and the object are directly added together to obtain a 256-dimensional joint feature code. The reference boxes of the human body and the object are concatenated to form an 8-dimensional joint reference box. The joint feature code and the joint reference box are used as input to the action decoder. Finally, the 256-dimensional feature code output by the action decoder is passed through a fully connected layer and classified using a sigmoid cross-entropy loss function (the sigmoid classification head supports multi-label action prediction, such as "hold" and "look" can coexist). The action decoder associates human body features with object features through cross-attention and then predicts the category of the interaction action between them. This design allows the model to explicitly model the semantic and spatial relationships between people and objects, rather than simply relying on post-processing to combine independent detection results, making it faster than the two-stage algorithms in the prior art that separate human body detection and action detection into two steps.

[0059] The triplet includes a human detection bounding box, an object detection bounding box and object category, and an action category. Figure 2 For example, Figure 2The main image shows two riders and a horse, with one rider riding the horse and the other leading it. This image is input into a detection model, and after obtaining and filtering all the triples from the input,... Figure 2 It also contains two triples. Triple 1 can be represented as <human frame 1, horse frame, riding>, and triple 2 can be represented as <human frame 2, horse frame, traction>.

[0060] S4. Filter out triplet pairs that do not meet the preset conditions, and use the retained triplet pairs as the detection results.

[0061] Specifically, triples that do not meet the preset conditions are filtered out, including:

[0062] The product of the probability corresponding to the object category and the probability corresponding to the action category in the triplet is used as the confidence level, and triplets with a confidence level less than the first preset threshold are filtered out.

[0063] Taking triple 1 above as an example, assuming the probability of the object category being "horse" is 0.95 and the probability of the action category being "riding" is 0.92, then the confidence level of this triple is 0.95 * 0.92 = 0.874. If the first preset threshold is 0.8, then this triple is retained.

[0064] In an optional embodiment of the present invention, in order to eliminate redundant or overlapping triples and ensure that only the most trustworthy triple is retained for each interaction instance, after filtering out triples with a confidence level less than a preset threshold, the method further includes:

[0065] A triplet-based version of Non-Maximum Suppression (NMS) is employed. The triplet with the highest confidence level is used as the baseline triplet. The Intersection over Union (IoU) between other triplets and the baseline triplet is calculated, and triplets with IoU exceeding a second preset threshold are filtered out. For example, assuming the confidence level of triplet 1 is higher than that of triplet 2, triplet 2 is filtered out when the weighted average of the overlaps between the human bounding box and the corresponding human bounding box and the object bounding box of triplet 2 and triplet 1 exceeds the second preset threshold. Through this confidence-based and NMS-based filtering, accurate detection of complex human-object interactions is achieved.

[0066] The AI-based human interaction detection method provided by this invention employs an independent decoder design. Taking the detection of "playing on a mobile phone" as an example, the human body decoder focuses on human body position information (such as head orientation), the object decoder identifies the mobile phone and its position information, and the action decoder determines the "looking down" action by summing features and concatenating spatial boxes (such as the relative position of the human box and the mobile phone box). Compared with the existing QPIC algorithm, it can detect the "playing on a mobile phone" behavior more accurately.

[0067] As shown in Table 1, the AI-based human interaction detection method based on self-attention mechanism provided by this invention achieves a map of 45.33 on the publicly available HICO-DET dataset, surpassing existing algorithms such as QPIC, FGA-HOI, DiffHOI, and RLIPv2. The rare (rare mAP) and non-rare (non-rare) categories in Table 1 are specific settings for the HICO-DET dataset. Rare represents interaction pairs with fewer than 10 instances in the training set among the 600 action-object pairs, while non-rare represents the remaining interaction pairs among the 600 action-object pairs.

[0068] Table 1 Comparison of Target Detection Accuracy

[0069]

[0070] In practical tasks, on our self-constructed dataset, the F1 score reached 95.8%, significantly outperforming QPIC's 89.7%. The F1 score is a statistical metric used to measure the precision of a binary classification model. It considers both precision and recall. The F1 score can be seen as a harmonic average of precision and recall, with a maximum value of 1 and a minimum value of 0. The formula is:

[0071]

[0072] In summary, the AI-based human interaction detection method based on self-attention mechanism provided by this invention pre-trains the detection model using a preset sample dataset; fine-tunes the pre-trained detection model using a real-world dataset; inputs the image to be detected into the detection model and obtains all triples output by the model; the detection model includes a backbone network for extracting image features, an image encoder for constructing reference boxes, a human decoder for independently detecting human position information, an object decoder for independently detecting object position information and category, and an action decoder for independently detecting action categories; triples that do not meet preset conditions are filtered out, and the retained triples are used as the detection result. This invention uses a win-large network as the base network, giving the detection model stronger feature extraction capabilities, including strong detection capabilities for small targets; transfer learning on large-scale datasets enables the detection model to have good robustness on small datasets; by separating human and object decoding and adding an additional action decoder, the detection model is more accurate in detecting interactive actions, and there is no interference between human, object, and action detection, allowing for better action classification and effectively improving the accuracy of target detection.

[0073] Based on the same inventive concept, the present invention also provides an AI character interaction detection device based on a self-attention mechanism. The AI ​​character interaction detection device based on a self-attention mechanism provided by the present invention will be described below. The AI ​​character interaction detection device based on a self-attention mechanism described below can be referred to in correspondence with the AI ​​character interaction detection method based on a self-attention mechanism described above.

[0074] like Figure 3 As shown, the AI ​​human interaction detection device based on self-attention mechanism provided by the present invention includes a pre-training module 31, a transfer learning module 32, an image detection module 33, and a post-processing module 34.

[0075] The pre-training module 31 is used to pre-train the detection model based on a preset sample dataset.

[0076] The transfer learning module 32 is used to train and fine-tune the pre-trained detection model using the actual dataset.

[0077] The image detection module 33 is used to input the image to be detected into the detection model and obtain all triples output by the detection model. The detection model includes a backbone network for extracting image features, an image encoder for constructing reference boxes, a human decoder for independently detecting human position information, an object decoder for independently detecting object position information and category, and an action decoder for independently detecting action categories.

[0078] The post-processing module 34 is used to filter out the triplets that do not meet the preset conditions and use the retained triplets as the detection result.

[0079] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute the AI ​​human interaction detection method based on the self-attention mechanism provided by the above methods, which includes:

[0080] The detection model is pre-trained based on a pre-set sample dataset;

[0081] The pre-trained detection model is trained and fine-tuned using a real-world dataset.

[0082] The image to be detected is input into the detection model to obtain all triples output by the detection model; the detection model includes a backbone network for extracting image features, an image encoder for constructing reference boxes, a human decoder for independently detecting human position information, an object decoder for independently detecting object position information and category detection, and an action decoder for independently detecting action categories.

[0083] The triplet that does not meet the preset conditions is filtered out, and the retained triplet is used as the detection result.

[0084] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the artificial intelligence human interaction detection method based on the self-attention mechanism provided by the above methods, the method comprising:

[0086] The detection model is pre-trained based on a pre-set sample dataset;

[0087] The pre-trained detection model is trained and fine-tuned using a real-world dataset.

[0088] The image to be detected is input into the detection model to obtain all triples output by the detection model; the detection model includes a backbone network for extracting image features, an image encoder for constructing reference boxes, a human decoder for independently detecting human position information, an object decoder for independently detecting object position information and category detection, and an action decoder for independently detecting action categories.

[0089] The triplet that does not meet the preset conditions is filtered out, and the retained triplet is used as the detection result.

[0090] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the artificial intelligence human interaction detection method based on the self-attention mechanism provided by the above methods, the method comprising:

[0091] The detection model is pre-trained based on a pre-set sample dataset;

[0092] The pre-trained detection model is trained and fine-tuned using a real-world dataset.

[0093] The image to be detected is input into the detection model to obtain all triples output by the detection model; the detection model includes a backbone network for extracting image features, an image encoder for constructing reference boxes, a human decoder for independently detecting human position information, an object decoder for independently detecting object position information and category detection, and an action decoder for independently detecting action categories.

[0094] The triplet that does not meet the preset conditions is filtered out, and the retained triplet is used as the detection result.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting AI-powered character interactions based on a self-attention mechanism, characterized in that, include: The detection model is pre-trained based on a pre-set sample dataset; The pre-trained detection model is trained and fine-tuned using a real-world dataset. The image to be detected is input into the detection model to obtain all triples output by the detection model; the detection model includes a backbone network for extracting image features, an image encoder for constructing reference boxes, a human decoder for independently detecting human position information, an object decoder for independently detecting object position information and category detection, and an action decoder for independently detecting action categories. The triplet that does not meet the preset conditions is filtered out, and the retained triplet is used as the detection result.

2. The AI-based character interaction detection method based on self-attention mechanism according to claim 1, characterized in that, The detection model processes the image to be detected in the following ways: The backbone network performs M downsampling operations on the image to be detected to generate M feature maps of different scales. Image features are extracted from different positions of the M feature maps, and the image features and their corresponding position information are input into the image encoder. The image encoder constructs encoded image features based on image features and the corresponding position information of the image features, determines the position of the corresponding reference box, calculates the probability that the image features and the corresponding reference box are foreground, and forms N queries with the N image features with the highest probability and the corresponding reference boxes, and inputs them into the human body decoder and the object decoder respectively; the reference box is an 8-dimensional vector, with the first 4 dimensions being the human body reference box and the last 4 dimensions being the object reference box; The human body decoder constructs a human body feature code based on the image features in the N queries, and fine-tunes the corresponding human body reference box based on the human body feature code to obtain a human body detection box; The object decoder constructs an object feature code based on the image features in the N queries, fine-tunes the corresponding object reference box based on the object feature code to obtain an object detection box, and determines the object category and the probability of the object category through the fully connected network. The action decoder determines the action category and the probability of the action category corresponding to the joint feature encoding and the joint reference box based on the joint feature encoding and the joint reference box; the joint feature encoding is obtained by adding the human body feature encoding and the object feature encoding, and the joint reference box is obtained by concatenating the human body reference box and the object reference box.

3. The AI ​​character interaction detection method based on self-attention mechanism according to claim 2, characterized in that, The triplet includes the human detection box, the object detection box and the object category, and the action category.

4. The AI ​​character interaction detection method based on self-attention mechanism according to claim 2, characterized in that, The backbone network of the detection model is a swin-large network.

5. The AI ​​character interaction detection method based on self-attention mechanism according to claim 2, characterized in that, Filtering out triples that do not meet preset conditions includes: The product of the probability corresponding to the object category and the probability corresponding to the action category in the triplet is used as the confidence level, and triplets with a confidence level less than a first preset threshold are filtered out.

6. The AI ​​character interaction detection method based on self-attention mechanism according to claim 5, characterized in that, After filtering out the triples with a confidence level less than a preset threshold, the process further includes: The triplet with the highest confidence level is used as the baseline triplet. The overlap between the other triplets and the baseline triplet is calculated, and triplets with an overlap exceeding a second preset threshold are filtered out.

7. The AI-based character interaction detection method based on self-attention mechanism according to any one of claims 1 to 6, characterized in that, The preset sample dataset includes the object365 dataset with pseudo-labels, some labeled data from the VisualGenome dataset, and the COCO dataset.

8. An AI-powered human interaction detection device based on a self-attention mechanism, characterized in that, include: The pre-training module is used to pre-train the detection model based on a pre-set sample dataset; The transfer learning module is used to train and fine-tune the pre-trained detection model using the actual dataset in use; The image detection module is used to input the image to be detected into the detection model and obtain all triples output by the detection model. The detection model includes a backbone network for extracting image features, an image encoder for constructing reference boxes, a human decoder for independently detecting human position information, an object decoder for independently detecting object position information and category, and an action decoder for independently detecting action categories. The post-processing module is used to filter out the triplets that do not meet the preset conditions and use the retained triplets as the detection result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the AI ​​human interaction detection method based on the self-attention mechanism as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the AI ​​human interaction detection method based on the self-attention mechanism as described in any one of claims 1 to 7.

Images