A semi-supervised human-object interaction detection method and device and a storage medium

Abstract

The present application relates to the technical field of image processing and recognition, and particularly relates to a semi-supervised human-object interaction detection method and device and a storage medium, the method of the present application obtains a human-object interaction detection dataset without annotation and with annotation, and performs enhancement processing on the dataset; a Student network and a Teacher network are respectively constructed, and the Student network and the Teacher network are respectively initialized; the Teacher network and the Student network are simultaneously trained, in the training, the output of the Student network is supervised by a preset loss function and a pseudo label output by the Teacher network, the parameters of the Student network are used to update the parameters of the Teacher network, and the Teacher network after parameter updating is obtained; the human-object relationship is detected by using the Teacher network after parameter updating, and the present application can further improve the accuracy of the current human-object interaction detection algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and recognition, and in particular to a semi-supervised human-object interaction detection method, device and storage medium. BACKGROUND

[0002] Human-object interaction detection can detect the position of the human and the object, the category of the object and the category of the interaction relationship in a picture. Human-object interaction detection has very important significance, for example: in an automatic driving system, a detection system can determine the situation of the surrounding road by detecting the relationship between pedestrians and surrounding objects on the road, so as to make safe driving behavior; in a hospital monitoring system, a detection system can determine whether a monitored person is in an emergency situation by detecting the relationship between the monitored person and the surrounding objects, so as to ensure the safety of the monitored person's life and health.

[0003] The main problem of human-object interaction detection at present is that complex annotation information makes it difficult to expand the training set. Since the training of Transformer requires a large amount of data, the performance improvement of the current mainstream human-object interaction detection model based on Transformer gradually tends to be bottlenecked. In the detection field, some methods have introduced semi-supervised learning ideas into the object detection task and achieved good results. However, directly applying the semi-supervised object detection method to human-object interaction detection will encounter the problems of being unable to improve the quality of the human-object interaction pair bounding box pseudo-label and the action category being too long-tailed to obtain effective action category pseudo-labels, which greatly limits the ability of human-object interaction detection models to learn better interaction pair features from unannotated data. SUMMARY

[0004] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a semi-supervised human-object interaction detection method, device and storage medium.

[0005] To achieve the above purpose, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a semi-supervised human-object interaction detection method, characterized in that it comprises the following steps:

[0007] Obtaining a human-object interaction detection dataset with no annotation and with annotation, and performing enhancement processing on the dataset;

[0008] Respectively constructing a Student network and a Teacher network, and respectively initializing the Student network and the Teacher network;

[0009] The Teacher network and the Student network are trained simultaneously, in the training, the output of the Student network is supervised by a preset loss function and pseudo labels output by the Teacher network, parameters of the Student network are used to update parameters of the Teacher network, and the Teacher network with updated parameters is obtained.

[0010] The Teacher network with updated parameters is used to detect human-object interaction.

[0011] In a second aspect, the present application further provides an electronic device, comprising a processor and a memory.

[0012] The memory is used to store a program.

[0013] The processor executes the program to realize the method as described above.

[0014] In a third aspect, the present application further provides a computer readable storage medium, the storage medium stores a program, and the program is executed by a processor to realize the method as described above.

[0015] In a fourth aspect, the present application further provides a computer program product or a computer program, the computer program product or the computer program comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the method as described above.

[0016] Compared with the prior art, the present application has the beneficial effects that the present application uses a coarse-to-fine mechanism to improve the quality of the pseudo label of the human-object interaction pair bounding box predicted by the unlabeled data, and generates an action score distribution of a rare class by a CLIP model to obtain a high-quality action class pseudo label. Thus, the problem that the existing semi-supervised method cannot obtain high-quality pseudo labels when applied to human-object interaction detection is solved, and the ability of the human-object interaction detection model to learn better interaction pair features from unlabeled data is greatly improved. A large amount of additional unlabeled data is used without manual labeling, and the human-object interaction detection performance is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a step flow chart of a semi-supervised human-object interaction detection method in an embodiment of the present application;

[0019] Figure 2 is a basic network structure schematic diagram of a Transformer encoder in an embodiment of the present application;

[0020] Figure 3 is a basic network structure schematic diagram of a Transformer decoder in an embodiment of the present application;

[0021] Figure 4 is a flow example diagram of a Teacher network structure generating pseudo labels in an embodiment of the present application;

[0022] Figure 5 is a basic network structure schematic diagram of a human-object interaction detection module in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0024] Embodiment:

[0025] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] Embodiment 1

[0027] As Figure 1As shown, this embodiment provides a semi-supervised learning method for detecting human-object interactions. This method improves the bounding boxes of interaction pairs predicted from unlabeled data by using a denoising module, and distills the Student network by generating action score distributions of rare categories through the CLIP model. This guides the Student network to more fully utilize unlabeled data to learn interaction pair features, further improving the accuracy of current human-object interaction detection. The method specifically includes the following steps:

[0028] Step 1: Obtain unlabeled and labeled datasets of human-object interaction detection, and perform augmentation processing on the datasets.

[0029] In this step, strong data augmentation is performed on the input labeled images. In this embodiment, strong data augmentation is performed on the training data of the HICO-DET dataset. Specifically, for an input image, it is horizontally flipped with a 50% probability; its brightness, contrast, and saturation are jittered within the range [0.6, 1.4]; it is then scaled with a 50% probability, during which the shortest side is selected with equal probability within the range [480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800], while ensuring that the longest side does not exceed 1333; finally, the image is normalized, with the mean and variance of the normalization being [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225], respectively. Weak data augmentation is performed on the input labeled and unlabeled data, specifically applying only the random horizontal flipping and scaling from the labeled strong data augmentation. Strong data augmentation is performed on the input unlabeled data. Specifically, on the basis of strong data augmentation with labels, grayscale scaling with a probability of 20% and Gaussian kernel blurring with a mean of 0.1 and a variance of 2.0 with a probability of 50% are added.

[0030] Step 2: Construct the Student network and the Teacher network respectively, and initialize the Student network and the Teacher network respectively.

[0031] In this step, taking the construction of the Student network as an example, the construction and initialization of the Student network includes the following steps: constructing and initializing a CNN-based deep neural network; constructing and initializing a Transformer encoder and a Transformer decoder; constructing a human-object relationship detection network, predicting the relationship between people and objects in the test image based on the output of the Transformer decoder, and initializing the human-object relationship detection network.

[0032] Specifically, (1) for a training picture after data enhancement, first get a feature map F through a CNN-based deep neural network.

[0033] The CNN-based deep neural network constructed in this embodiment is a ResNet-50 network, followed by a 1x1 convolution to reduce the number of channels, and the initialization method is to use the parameters of the object detection model based on the Transformer trained on MS-COCO as the initialization parameters.

[0034] (2) After obtaining F, the feature is input into the Transformer encoder.

[0035] In this embodiment, three-dimensional position encoding is required for each pixel of the three-dimensional feature map F, and the position encoding method is as follows:

[0036]

[0037]

[0038] Where pos represents a position of a two-dimensional picture, D is a constant, and in this embodiment, D is 128; j represents the dimension, and for positions with odd channels, the position is encoded using the cos function; for positions with even channels, the position is encoded using the sin function; and finally, the output PE is a three-dimensional position encoding matrix with the same dimension as F.

[0039] In this embodiment, the Transformer encoder is composed of l cascaded encoder layers, and l is set to 6. The encoder structure of this embodiment is shown in Figure 2 Each encoder layer is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a feedforward network, a residual connection network, and a layer normalization processing module. As shown in Figure 2 It can be seen that the query matrix, the key matrix, and the value matrix of the Transformer encoder are respectively:

[0040] Q e =F+PE

[0041] K e =F+PE

[0042] V e =F

[0043] The calculation process of the Transformer encoder is expressed as:

[0044] E=f enc (F,PE)

[0045] where E is the output of the feature after passing through the Transformer encoder, f enc represent the cascaded encoder layers.

[0046] The initialization of the Transformer encoder is to use the parameters of the encoder in the Transformer-based object detection model trained on MS-COCO as the initialization parameters.

[0047] (3) Next, the Transformer encoder feature E and the position encoding PE are input into the Transformer decoder.

[0048] In this embodiment, the Transformer decoder is composed of l cascaded decoder layers, and l is also set to 6, consistent with the number of layers of the encoder. The decoder structure of this embodiment is shown in Figure 3 , each decoder layer is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a mutual attention module, a residual connection network, a layer normalization processing module, a feedforward network, a residual connection network, and a layer normalization processing module; the specific steps of constructing two query matrices of the mutual attention module of the decoder are:

[0049] Step (31): obtain the position feature encoding vector i A set of learnable vectors is constructed:

[0050]

[0051] Step (32): obtain the position feature encoding vector

[0052]

[0053] Step (33): obtain the position feature encoding vector from the position feature encoding vector t :

[0054]

[0055] Step (34): obtain a set of position features of the labeled relationship pair Q t from the set of feature encoding vectors H q , the feedforward network F gt :

[0056] Q gt = tanh(F q (Shift(H t)))

[0057]

[0058] where Q is a set of learnable vectors, Q gt is a set of position features of annotated relation pairs, and the two sets of query vectors are input into the decoder together through a mask to shield the self-attention mechanism between each other; is the number of annotated relation pairs in the picture, H t is a set of position feature encodings; in , the 8 elements respectively represent the coordinates of the center points of the human and object boxes, width and height; F q is composed of 2 fully connected layers, and the activation function between the fully connected layers is ReLU, F q The initial parameters of F

[0059] The key matrix and the value matrix are respectively:

[0060] K d = E + PE

[0061] V d = E

[0062] The calculation process of the Transformer decoder is expressed as:

[0063] D, D gt = f dec (Q, Q gt , E, PE)

[0064] where D is the output of Q after the Transformer decoder, and D gt is the output of Q gt after the Transformer decoder.

[0065] The initialization method of the Transformer decoder is to use the parameters of the decoder in the Transformer-based object detection model trained on MS-COCO as the initialization parameters.

[0066] (4) D passes through the human-object relation detection network to obtain the final prediction result, and the jth prediction result triple includes where is the normalized position of the human and object boxes, where N obj and N act are the number of objects and the number of relations expressed in the data set. The human-object relation detection network contains 4 forward feedback networks, which are as follows:

[0067]

[0068]

[0069]

[0070]

[0071] Where F h and F o It consists of 3 fully connected layers, with ReLU as the activation function between the fully connected layers; F c and F a It is a fully connected layer. h F o and F c The initialization method uses the parameters of the object detection network in the Transformer-based object detection model trained on MS-COCO as the initialization parameters. a Random initialization.

[0072] After obtaining the prediction results from the Student network, the matching method between the prediction results and the labeled relationship pairs is the Hungarian algorithm, where the loss matrix is ​​calculated as follows:

[0073]

[0074]

[0075]

[0076]

[0077]

[0078] in Let Φ represent the relation pair of the i-th labeled image, and let Φ represent the set of indices of the empty set of relation pairs in the image; GIOU is short for generalized IoU; the above calculation yields the index position of the predicted relation pair corresponding to each labeled relation pair.

[0079] D gt Similarly, after passing through the human-object relationship detection network, the final prediction result is obtained. The j-th prediction result triple includes... However, at this point, the Hungarian algorithm matching is not required; Q can be constructed directly. gt The loss function is calculated based on the labeled information.

[0080] The Teacher network is constructed and initialized. It also consists of a CNN-based deep neural network, a Transformer encoder, a Transformer decoder, and a human-object relationship detection network. Its parameters are updated using an exponential moving average based on the Student network's parameters. The differences between the Teacher network and the Student network are as follows:

[0081] (1) In the Student network, the inputs are a strongly augmented view with labeled data, a weakly augmented view with labeled data, and a strongly augmented view without labeled data; in the Teacher network, the input is a weakly augmented view without labeled data.

[0082] (2) In the Student network, the query matrix Q of the mutual attention module in the Transformer decoder. gt It is constructed using labeled data and annotation information; in the Teacher network, it is the query matrix of the mutual attention module in the Transformer decoder. Through the learnable vector Q t The query matrix of the mutual attention module in the Teacher network's Transformer decoder is obtained from the prediction of unlabeled data interaction results. The construction method is as follows:

[0083] Step (21): Use unlabeled weakly enhanced image x u，w A set of learnable vectors Q in a Teacher network t and Teacher Network The first prediction result of the Teacher network was obtained.

[0084]

[0085] Step (22): Use the network's first prediction result And a pseudo-label filtering strategy based on threshold t The first prediction result of the filtered Teacher network is obtained.

[0086]

[0087] Step (23): Make

[0088]

[0089]

[0090] In the formula, xu，w weak augmented views for the unlabeled data input into the Teacher network, denotes the entire Teacher network; denotes that the Teacher network uses a learnable query vector Q t predicted results, denotes a threshold t-based pseudo-label filtering strategy, denotes filtered predicted results; consists of 2 fully connected layers, and the activation function between the fully connected layers is ReLU, The parameters of the Teacher network are obtained by exponential moving average of the parameters of the Student network.

[0091] (3) In the Teacher network, construct The threshold filtering strategy used when is:

[0092]

[0093]

[0094] In the formula, N Q is the number of the set of learnable vectors Q t , which is consistent with Q, and in the embodiment, is 100, t is the threshold value when filtering the reliable pseudo label, in the embodiment, the threshold value t is 0.9, and I is the index set that meets the threshold condition.

[0095] Step 3: simultaneously train the Teacher network and the Student network, in the training, use a preset loss function and use the pseudo label output by the Teacher network to supervise the output of the Student network, use the parameters of the Student network to update the parameters of the Teacher network, and obtain the Teacher network after parameter updating.

[0096] In this step, after obtaining the predicted results of the Student network, the embodiment uses a plurality of loss functions to supervise the predicted results, and the preset loss functions include: L1 loss, generalized IoU loss, cross-entropy loss, focal loss, and quality focal loss.

[0097] Wherein, L1 loss and generalized IoU loss are used to supervise the regression of human and object boxes in labeled data and unlabeled data; cross-entropy loss is used to supervise the classification of object categories in labeled data and unlabeled data; focal loss is used to supervise the classification of relationship categories in labeled data; quality focal loss is used to supervise the classification of relationship categories in unlabeled data.

[0098] The specific relationship detection total loss function is:

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] Wherein, represents the set of subscripts of the non-empty set of labeled relationship pairs in the picture, represents the number; represents the set of subscripts of the empty set of labeled relationship pairs in the picture; is the result of the labeled picture corresponding to the labeled data through Q after the output of the Student network and the matching of the Hungarian algorithm; is the prediction result of the output of the Student network through Q of the labeled data; N gt q ​the number of potential relations for prediction, l f focal loss for each element; the output of the Student network through Q after matching with the pseudo-labels generated by the Teacher network the result of the quality focal loss, lqf and L are the total loss functions of the relation detection of the labeled data and the unlabeled data through the Student network, respectively; the output results of Q and Q gt L1 loss, generalized IoU loss, cross-entropy loss and focal loss, L1 loss, generalized IoU loss, cross-entropy loss and quality focal loss of the output results of Q through the unlabeled data; are all weight coefficients. In the embodiment are 2.5, 1, 1, 1, 2.5, 1, 1 and 2, respectively.

[0114] The method for the Teacher network to output the pseudo-labels is:

[0115] Step 31: using the unlabeled weakly augmented image x u，w the set of position features of a set of labeled relation pairs in the Teacher network and the Teacher network obtain the second prediction result of the Teacher network and the bounding box prediction result of the person, the bounding box prediction result of the object and the prediction result of the object class in the second prediction result are directly used as the final pseudo-labels of the bounding box of the person the bounding box pseudo-labels of the object and the object class pseudo-labels

[0116]

[0117]

[0118] Step 32: using the set of position features of a set of labeled relation pairs in the Teacher network the i-th output feature vector obtained through the Transformer decoder the set of text feature vectors generated by the CLIP text encoder through the interaction classe obtain a feature similarity score s i :

[0119]

[0120] Step 33: obtain a set of position features of the labeled relation pair using the feature similarity score s i , a scaling parameter γ and a correction coefficient m i the i-th output feature vector obtained by the Transformer decoder the interaction category probability distribution p i :

[0121]

[0122] Step 34: obtain an interaction category probability distribution set P using the interaction category probability distribution p i , the action category prediction result in the second prediction result of the Teacher network and a combination strategy to obtain the final action category pseudo label

[0123]

[0124]

[0125] wherein, represents the result predicted by the Teacher network using the position feature vector of the relation pair , represents the i-th output feature vector obtained by the Transformer decoder of the Teacher network, t e represents the text feature vector set of the interaction category generated by the CLIP text encoder; γ is a scaling parameter, which is 1000 in this embodiment, and m i is a correction coefficient to eliminate invalid interaction categories, p i represents the probability distribution of the interaction category; is a combination strategy used to combine the non-rare interaction categories in and the rare interaction categories in P; is the pseudo label set generated by the final Teacher network, which is used to supervise the output of the unlabeled data through the Student network.

[0126] Step 4: detecting the human-object relationship using the Teacher network with updated parameters.

[0127] ​In this step, the parameters of the Teacher network are updated by exponential moving average using the parameters of the Student network in training:

[0128] θ t ←αθ t +(1-α)θ s

[0129] wherein θ t represents the parameters of the Teacher network, θ s represents the parameters of the Student network, and a is a moving average coefficient, which is set to 0.9999 in this embodiment. In testing, the Teacher network updated by exponential moving average is used to obtain the detection result of human-object interaction.

[0130] In this embodiment, the feature of the testing process is that only the Q t through the output of the decoder, without relying on No additional computational load is added in the testing process.

[0131] To verify the effectiveness of the present application, an embodiment was tested on the HICO-DET dataset, as shown in Table 1 below, using the mean Average Precision (mAP) as the measurement standard, which is calculated as follows: the prediction accuracy is calculated for each action category contained in the dataset in all test images, and the average prediction accuracy of all actions is the mAP.

[0132] Table 1 Comparison data of the present application and other methods on HICO-DET

[0133]

[0134]

[0135] In summary, the present embodiment has the following advantages and beneficial effects compared with the prior art:

[0136] (1) The method of the present embodiment solves the problem that the existing semi-supervised method cannot obtain high-quality pseudo labels when applied to the current human-object interaction detection based on Transformer. The quality of the human-object interaction pair bounding box pseudo labels predicted by the unlabeled data is improved by using the coarse-to-fine mechanism, and the high-quality action class pseudo labels are obtained by generating the action score distribution of rare classes through the CLIP model, which greatly improves the performance of the human-object interaction detection algorithm.

[0137] (2) By utilizing unlabeled human-object interaction detection data, the size of the existing training dataset is greatly expanded; more data related to human-object interaction detection is used without relying on manual annotation, thus improving training efficiency; at the same time, the Teacher network is updated through the exponential moving average of the Student network, reducing the space occupied by the GPU memory.

[0138] (3) During the testing process, only the Q-factor of the Teacher network is used. t The output of the decoder is independent of The testing process does not increase the amount of additional computation, thus improving performance without increasing the testing time of the model.

[0139] (4) At the same time, this method is applicable to many Transformer-based human-object interaction detection networks and has wide application value.

[0140] Example 2

[0141] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, the electronic device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to realize the semi-supervised human-object interaction detection method as described above.

[0142] It is understood that the memory may include random access memory (RAM) or read-only memory. In an alternative embodiment, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the data storage area may store data created according to the use of the server, etc.

[0143] The processor can include one or more processing cores. The processor connects various parts within the entire server by various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. In an optional embodiment, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with one or several combinations of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs; the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but be implemented by a separate chip.

[0144] Since the electronic device is the electronic device corresponding to the semi-supervised human-object interaction detection method of the embodiments of the present application, and the principle of solving the problem of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described again.

[0145] Embodiment 3

[0146] Based on the same inventive concept, the embodiments of the present application also provide a computer readable storage medium, the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to realize the semi-supervised human-object interaction detection method as described above.

[0147] Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be instructed by programs to relevant hardware, and the programs can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0148] Since the storage medium is the storage medium corresponding to the semi-supervised human-object interaction detection method of the embodiments of the present application, and the principle of solving the problem of the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0149] Embodiment 4

[0150] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the semi-supervised human-object interaction detection method according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Wherein, the executable computer program codes or "codes" for executing various embodiments can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0151] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0152] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0153] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made in accordance with the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A semi-supervised method for detecting human-object interaction, characterized in that, Includes the following steps: Obtain unlabeled and labeled datasets of human-object interaction detection, and perform augmentation processing on the datasets; Construct a Student network and a Teacher network respectively, and initialize the Student network and the Teacher network respectively; Simultaneously train the Teacher network and the Student network. During training, use a preset loss function and the pseudo-labels output by the Teacher network to supervise the output of the Student network. Use the parameters of the Student network to update the parameters of the Teacher network to obtain the Teacher network with updated parameters. The updated Teacher network is used to detect relationships between people and objects. Both the Student and Teacher networks include a CNN-based deep neural network, a Transformer encoder, a Transformer decoder, and a human-object interaction detection network. The query matrix of the mutual attention module in the Transformer decoder of the Teacher network is also included. Construct according to the following preset formula: In the formula, x u,w To provide a weakly augmented view of the unlabeled data from the Teacher network, Represents the entire Teacher network; This indicates that the Teacher network uses a learnable query vector Q. t The predicted results This represents a pseudo-label filtering strategy based on a threshold t. This indicates the filtered prediction result; It consists of two fully connected layers, with ReLU as the activation function between the fully connected layers. The parameters are obtained from the Student network exponential moving average; Shift indicates the addition of random noise to the location information.

2. The semi-supervised human-object interaction detection method according to claim 1, characterized in that, The Transformer encoder includes a cascaded self-attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network, and a layer normalization processing module. The query matrix, key matrix, and value matrix of the self-attention module are F+PE, F+PE, and F, respectively. The output of the Transformer encoder is E. The Transformer decoder includes a cascaded self-attention module, a residual connection network, a layer normalization processing module, a mutual attention module, a residual connection network, a layer normalization processing module, a feedforward network, a residual connection network, and a layer normalization processing module; the query matrix of the mutual attention module is Q and Q. gt The key matrix and value matrix are E+PE and E, respectively. The output of the Transformer decoder is D; Where F represents the output features of the input image after passing through a CNN-based deep neural network, PE represents the positional encoding of the input image, and the query matrix Q represents a set of learnable vectors. gt This represents a set of location features for a set of labeled relationship pairs.

3. The semi-supervised human-object interaction detection method according to claim 1, characterized in that, The human-object interaction detection network includes four feedforward networks, each consisting of a fully connected layer, a fully connected layer, a fully connected layer, and a fully connected layer. These four feedforward networks are used to predict the human's position, the object's position, the object's category, and the relationship category, respectively.

4. The semi-supervised human-object interaction detection method according to claim 2, characterized in that, The query matrix Q of the mutual attention module in the Transformer decoder of the Student network. gt Construct according to the following preset formula: Q gt =tanh(F q (Shift(H t ))) In the formula, H represents the number of relationship pairs labeled in the image. t A set of location feature codes; in In the text, the eight elements represent the coordinates of the center point of the person and object frames, as well as their width and height; F q It consists of two fully connected layers, with ReLU as the activation function between the fully connected layers. q The initial parameters are initialized randomly; Shift indicates that random noise is added to the position information.

5. The semi-supervised human-object interaction detection method according to claim 2, characterized in that, In the Student network, the inputs are a strongly augmented view with labeled data, a weakly augmented view with labeled data, and a strongly augmented view without labeled data; In the Teacher network, the input is a weakly augmented view with unlabeled data; In the Student network, the query matrix of the mutual attention module of the Transformer decoder is constructed using labeled data annotation information; in the Teacher network, the query matrix of the mutual attention module of the Transformer decoder is obtained by predicting the results of unlabeled data interaction pairs using learnable vectors.

6. The semi-supervised human-object interaction detection method according to claim 2, characterized in that, The pseudo-labels output by the Teacher network are constructed according to the following preset formula: In the formula, This indicates that the Teacher network uses relational features on the location feature vector. Predicting results for unlabeled data, express The i-th output feature vector t obtained through the Transformer decoder of the Teacher network e This represents the set of text feature vectors generated by the CLIP text encoder to represent the interaction category. This represents the i-th output feature vector; γ is a scaling parameter, m i It is a correction factor to eliminate invalid interaction categories, p i Represents the probability distribution of interaction categories; It is a combination strategy used to... Combine the non-rare interaction categories in P with the rare interaction categories in P; It is the final set of pseudo-labels generated by the Teacher network, used to supervise the output of unlabeled data through the Student network.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by the processor to implement the semi-supervised human-object interaction detection method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction and at least one program segment, which are loaded and executed by a processor to implement the semi-supervised human-object interaction detection method as described in any one of claims 1 to 6.

Images