A secondary path guided primary path learning interaction detection method based on encoder-decoder

Abstract

The present invention discloses a method for detecting interaction in which a secondary path guides a primary path learning process based on an encoder / decoder. The method comprises inputting a labeled interaction image into a feature extraction module to obtain image features output by the feature extraction module; inputting the image features into the primary and secondary paths of the dual-path learning module to obtain features for people, objects, and interactions output by the primary and secondary paths, respectively; inputting the features for people, objects, and interactions output by the primary and secondary paths into the interaction prediction module to obtain prediction results corresponding to the primary and secondary paths, respectively; performing consistency learning based on the prediction results corresponding to the primary and secondary paths, performing loss supervision on the primary and secondary paths, and updating the network parameters of a deep learning model; and taking the optimal deep learning model after training to output prediction results for the interaction image to be detected. The present invention introduces a multi-layer perceptron for global receptive field encoding to overcome the technical problems raised in the above-mentioned background technology.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an encoder-decoder-based secondary-path guided primary-path learning interaction detection method. Background Art

[0002] Detecting human-object interactions plays a crucial role in understanding high-level information. Therefore, improvements in interaction detection are crucial for human-robot interaction. In robotic manipulation, robots need to mimic human actions and complete them, or infer subsequent actions from human actions and complete them. In behavioral monitoring, robots need to be able to detect dangerous behaviors and directly warn or remedy them. Consequently, some erroneous cooperative behaviors can be avoided by monitoring and understanding human interactions.

[0003] At present, interaction detection is mainly divided into two stages and one stage. The second stage relies on an excellent target detector. After obtaining high-quality target categories and target bounding boxes, the targets are paired, and finally the paired results are interactively detected. This method generally uses graph convolution to fuse the features of the two paired targets to obtain an interactive feature, just like the information of the two vertices of the graph is fused to the corresponding edges. Therefore, this method has the following disadvantages: 1) The pairing process is actually a forced combination process. Many forced pairings are meaningless and prone to uneven positive and negative samples; 2) When there are many targets in the image, approximately N targets will be combined. 2 For interaction, this will consume a lot of computing resources.

[0004] Unlike the two-stage approach, the one-stage approach also relies on an object detector, but it performs pairing before obtaining a detection result. This means that this feature can learn potential, more meaningful interaction information, which is more conducive to interaction detection than the two-stage approach. Secondly, since the current one-stage approach is almost always based on an encoder-decoder, the number of pairings is a constant K, which to some extent alleviates the computational resource consumption problem of the two-stage approach. However, the current one-stage approach only uses a single detection path to complete detection, while there are actually many detection paths, which is an underutilization of the one-stage approach. At the same time, some detection paths are designed for the two-stage approach and cannot be directly migrated to the one-stage approach, resulting in the designed detection paths not being fully utilized. Summary of the Invention

[0005] The purpose of the present invention is to provide a secondary-path guided primary-path learning interaction detection method based on an encoder-decoder, and introduce a multi-layer perceptron for global receptive field encoding to overcome the technical problems raised in the above background technology.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for detecting interaction between a secondary path and a primary path in learning based on an encoder-decoder is provided. The method performs interaction detection by constructing a deep learning model. The deep learning model includes a feature extraction module, a dual-path learning module, and an interaction prediction module. The method includes:

[0008] Taking the labeled interaction picture and inputting it into the feature extraction module to obtain the picture features output by the feature extraction module, the feature extraction module includes a deep residual network and an encoder;

[0009] Inputting the image features into a primary path and a secondary path of the dual-path learning module, respectively, to obtain features for people, objects, and interactions output by the primary path and the secondary path, respectively, wherein the primary path includes a decoder unit and the secondary path includes a two-stage decoder unit;

[0010] Input the features for people, objects and interactions output by the primary and secondary paths into the interaction prediction module to obtain prediction results Pr1, Pr2∈{people, objects, interactions} corresponding to the primary and secondary paths respectively;

[0011] Perform consistency learning based on the prediction results corresponding to the primary path and the secondary path, perform loss supervision on the primary path and the secondary path, and update the network parameters of the deep learning model;

[0012] The best deep learning model after training is used to output the prediction results for the interaction image to be detected.

[0013] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.

[0014] Preferably, the deep residual network is a deep residual network ResNet-50.

[0015] Preferably, the processing of the decoder unit includes:

[0016] Take the input image features as the query library X and initialize a query Q hoi , using query Q hoi Get features for people, objects, and interactions in the query repository X

[0017] The processing of the two-stage decoder unit includes:

[0018] Take the input image features as the query library X and initialize the query Q for people and objects h ,Q o, query Q h ,Q o Merge into query Q ins , using query Q ins Get the features for people in the query library X and characteristics of objects Then the features and Weighted feature The features The query Q′ as the initial interaction of prior information inter , by querying Q′ inter Get features for interaction in query library X The features and features Perform feature fusion to obtain the final interactive features of the secondary path

[0019] As an advantage, the feature and features Perform feature fusion to obtain the final interactive features of the secondary path include:

[0020] The features and After splicing, fusion is performed and the fusion result is normalized to obtain the instance attention weight α, which is specifically expressed as follows:

[0021]

[0022] Among them, sigmoid(·) represents normalization processing, MLP1(·) represents feature fusion, and concat(·,·) represents feature concatenation;

[0023] Then the features And the instance attention weight α is multiplied to obtain the instance enhanced interaction feature, and finally the feature is connected by short circuit. Instance-enhanced interaction features and features mapped to the interaction domain Add together to get the final features for interaction The specific expressions are as follows:

[0024]

[0025] Here, ⊙ represents dot product and MLP2(·) represents spatial mapping.

[0026] Preferably, the prediction results are as follows:

[0027]

[0028]

[0029] Among them, Pr1 and Pr2 represent the main road Heci Road The prediction results, Represent the regression or classification functions for people on the main road and the secondary road respectively, Represent the regression or classification functions for objects on the main road and the secondary road respectively, Represent the regression or classification functions used for interaction on the main road and the secondary road respectively, They represent the prediction results output by the corresponding regression or classification function respectively.

[0030] Preferably, the performing consistency learning according to the prediction results corresponding to the primary path and the secondary path includes:

[0031] Get the prediction set h corresponds to people, o corresponds to objects, and inter corresponds to interactions. The binary matching relationship between the prediction result and the true value of each path is defined as Represents the true value, and the three-way prediction results output by the primary and secondary paths are paired according to the bipartite matching relationship to obtain the two-way pairing set δ;

[0032] After obtaining the correspondence between the two paths, the KLoss method is used to measure the similarity of the two path matrices based on the consistency in classification. The mathematical expression is as follows:

[0033]

[0034]

[0035] in, is the similarity of the prediction results of the two paths for the features used for objects, is the similarity of the prediction results of the two paths for the features used for interaction, KLoss(·||·) represents the divergence of two features or two probability distributions, z(f,m) represents the mth f, Represents the prediction result of feature q in path k, q∈(o,inter);

[0036] For the consistency of bounding box positions, the mean absolute error is used to measure the consistency between paired groups. The mathematical expression is as follows:

[0037]

[0038]

[0039] in, represents the mean absolute loss of the bounding box position prediction results for human features, represents the mean absolute loss of the bounding box position prediction results for object features, ||·||1 represents the L1 paradigm, represents the bounding box position prediction result of feature n in path k, n∈(h,o), The total number of sets representing true values;

[0040] So the consistency loss function of the two paths is It is expressed as follows:

[0041]

[0042] Among them, λ1, λ2 and λ3 are hyperparameters used to adjust the loss ratio.

[0043] Preferably, the loss monitoring of the primary and secondary roads includes:

[0044] Each path consists of three losses: the mean absolute error of the bounding box position, the generalized intersection-over-union ratio of the bounding box size, and the classification cross entropy. Therefore, the interaction triplet loss function of the two paths is The mathematical expression is as follows:

[0045]

[0046]

[0047]

[0048] Among them, θ k Represents the pairing set obtained by bipartite matching of the prediction result and the true value of path k, Represents the true value of the bounding box position of feature n, represents the true classification value of feature q, CE(·,·) represents the cross entropy of the two classification results, represents the mean absolute loss of the bounding box position prediction results of feature n in path k, represents the generalized intersection-over-union ratio of the bounding box size of feature n in path k, represents the classification cross entropy of the prediction result of feature q in path k, λ4, λ5 and They are all hyperparameters. In the end, the total loss function for:

[0049]

[0050] Among them, w(t) is a linear growth function with the number of training rounds t. As the number of training rounds t increases, the proportion of consistency loss gradually increases.

[0051] The encoder-decoder-based secondary-path guided main-path learning interaction detection method provided by the present invention introduces a dual-path module, so that the model can learn more generalization capabilities and resist noise in the image. The strength attention interaction module is used to enable the detection path transplanted from the second stage to overcome the large-scale classification errors and attention range errors caused by the intermediate unsupervised transplantation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flowchart of the encoder-decoder-based secondary path guiding primary path learning interaction detection method of the present invention;

[0053] Figure 2 Schematic diagram of the structure of the deep learning model of the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0056] In one embodiment, Figure 1 As shown in the figure, a method for detecting interaction between secondary-path guided primary-path learning based on an encoder-decoder is proposed. Interaction detection is performed by constructing a deep learning model. The deep learning model includes a feature extraction module, a dual-path learning module, and an interaction prediction module. The method for detecting interaction between secondary-path guided primary-path learning based on an encoder-decoder includes:

[0057] Step S1: Take the labeled interactive image and input it into the feature extraction module to obtain the image features output by the feature extraction module. The feature extraction module includes a deep residual network and an encoder.

[0058] For images that need interaction detection, this implementation first randomly enlarges and randomly crops the image and its corresponding annotation to obtain input I∈R 3×H×W , H is the length of the image, W is the width of the image, and then I is input into the network.

[0059] For example, after enlarging the image by 1 times, the size and coordinates of the bounding boxes of the corresponding people and objects must also increase by a multiple; if the image is randomly cropped, the size and coordinates of the bounding boxes must be based on the cropping area, and the intersection of the two must be taken to correct the size and coordinates of the bounding boxes.

[0060] The feature extraction module of this embodiment uses the pre-trained deep residual network ResNet-50 and the encoder of the encoder-decoder to extract the image features X∈R from the processed image I. D×H×W , D is the number of channels.

[0061] Step S2: Input the image features into the main path and the secondary path of the dual-path learning module respectively, and obtain the features for people, objects and interactions output by the main path and the secondary path respectively, where the main path includes a decoder unit and the secondary path includes a two-stage decoder unit.

[0062] like Figure 2 As shown, the decoder unit of this embodiment includes a decoder, and the processing process of the decoder unit includes: taking the input image features as the query library X and initializing a query Q hoi ∈R K×D , K is the number of queries, query Q hoi Get features for people, objects, and interactions in the query repository X

[0063] The two-stage decoder unit of this embodiment includes a decoder, a decoder, and an instance attention interaction unit connected in sequence. The processing process of the two-stage decoder unit includes:

[0064] Take the input image features as the query library X and initialize the query Q for people and objects h ,Q o ∈R K×D , query Q h ,Q o Merge into query Q ins , using query Q ins Get the features for people in the query library X and characteristics of objects Then the features and Weighted feature The features The query Q′ as the initial interaction of prior information inter ∈R K×D , by querying Q′ inter Get features for interaction in query library X

[0065] In order to make the query interaction completely based on the interaction instance (people and objects), after obtaining the interaction features Then, the features and features Input instance attention interaction unit to perform feature fusion and obtain the final feature for interaction in the sub-path

[0066] The specific processing process of the instance attention interaction unit is as follows:

[0067] The features and After splicing, fusion is performed and the fusion result is normalized to obtain the instance attention weight α∈R K×1 , specifically expressed as follows:

[0068]

[0069] Among them, sigmoid(·) represents normalization processing, MLP1(·) represents feature fusion, and concat(·,·) represents feature concatenation.

[0070] Then the features And the instance attention weight α is multiplied to obtain the instance enhanced interaction feature, and finally the feature is connected by short circuit. Instance-enhanced interaction features and instance features mapped to the interaction domain Add together to get the final features for interaction The specific expressions are as follows:

[0071]

[0072] Here, ⊙ represents the dot product, and MLP2(·) represents the spatial mapping. It should be noted that the prototypes of MLP1(·) and MLP2(·) are both MLP(·), and the subscript form is used for easy distinction.

[0073] Combining the overall extraction process of this step, we can obtain the specific feature extraction mathematical formulas for the two paths:

[0074]

[0075]

[0076] Among them, f 1,1 (·,·) means using the first decoder on the first path to decode the feature, f 2,1 (·,·) means using the first decoder on the second path to decode the feature, f 2,2 (·,·) indicates that the second decoder on the second path is used to decode the feature. In this embodiment, path 1 or the first path refers to the main path, path 2 or the second path refers to the secondary path, and V erepresents the visual features obtained by the backbone network and encoder, Represent the feature outputs of the main path and the secondary path respectively.

[0077] Step S3: Input the features for people, objects, and interactions output by the primary and secondary paths into the interaction prediction module to obtain prediction results Pr1, Pr2∈{people, objects, interactions} corresponding to the primary and secondary paths, respectively.

[0078] After obtaining the respective features from the decoders on the two paths, the features are sent to their respective interactive prediction modules to obtain their respective prediction results. The prediction process of the two paths is shown as follows:

[0079]

[0080]

[0081] Among them, Pr1 and Pr2 represent the prediction results of the main road and the secondary road respectively. Represent the regression or classification functions for people on the main road and the secondary road respectively, Represent the regression or classification functions for objects on the main road and the secondary road respectively, Represent the regression or classification functions used for interaction on the main road and the secondary road respectively. and The regression or classification functions used are the same, only the function parameters are different. and The regression or classification functions used are the same, only the function parameters are different. are the classification prediction results output by the corresponding functions respectively.

[0082] Furthermore, the regression or classification function in this embodiment adopts an existing function, such as logistic regression, quantile regression, least squares regression, etc.

[0083] Step S4: performing consistency learning based on the prediction results corresponding to the primary path and the secondary path to obtain the final prediction result, performing loss supervision on the primary path and the secondary path, and updating the network parameters of the deep learning model.

[0084] The consistency calculation of the two paths is divided into two parts: 1) First, the prediction results of the two paths are matched with the true values by binary matching. Then, the transitivity of the two binary matching results is used to make the two paths correspond to each other. 2) For the correspondence, the classification cross entropy, the mean absolute error of the bounding box position, and the generalized intersection-over-union ratio of the bounding box size are calculated respectively.

[0085] The specific steps are as follows: h corresponds to people, o corresponds to objects, and inter corresponds to interactions. The binary matching relationship between the prediction result and the true value of each path is defined as Represents the true value, observes the relationship between the pairing results, and finds that Associate the two paths. For example, The matching result is The matching result is Found that they all 2 pairs, so it can be considered The 1 in is 3 in the pair. As above, you can and Convert to Aggregate all the pairing results of the two paths to get δ, which is used to represent the pairing set of the two paths. After obtaining the correspondence between the two paths, the KLoss method is used to measure the similarity of the two path matrices based on the consistency of classification. The mathematical expression is as follows:

[0086]

[0087]

[0088] in, is the similarity of the prediction results of the two paths for the features used for objects, is the similarity of the prediction results of the two paths for the features used for interaction, KLoss(·||·) represents the divergence of two features or two probability distributions, z(f,m) represents the mth f, Represents the prediction result of feature q in path k, q∈(o,inter).

[0089] That is, in the formula Indicates the main path The classification prediction results of the feature o used for the object, since multiple features of the same type can be extracted from one image, the same type of feature has multiple classification prediction results. To obtain a list containing multiple prediction results The ath one in . The same applies to the others.

[0090] For the consistency of bounding box positions, a simple mean absolute error is used to measure the consistency between paired groups. The mathematical expression is as follows:

[0091]

[0092]

[0093] in, represents the mean absolute loss of the bounding box position prediction results for human features, represents the mean absolute loss of the bounding box position prediction results for object features, ||·||1 represents the L1 paradigm, represents the bounding box position prediction result of feature n in path k, n∈(h,o), The total number of sets representing true values.

[0094] That is, in the formula Indicates the main path The bounding box position prediction results of the feature h used for the person. Since multiple features of the same type can be extracted from an image, the same type of feature has multiple bounding box position prediction results. To obtain a prediction result containing multiple bounding boxes The ath one in . The same applies to the others.

[0095] Therefore, the total consistency loss function of the two paths is expressed as follows:

[0096]

[0097] Among them, λ1, λ2 and λ3 are hyperparameters used to adjust the loss ratio.

[0098] In the loss supervision of each path, this example uses bipartite matching to match the predicted result with the true value to obtain the paired result, and then calculates the loss. Each path consists of three losses: the mean absolute error of the bounding box position, the generalized intersection-over-union ratio of the bounding box size, and the classification cross entropy. Therefore, the triple loss function of the two paths is The mathematical expression is as follows:

[0099]

[0100]

[0101]

[0102] Among them, θ k Represents the pairing set obtained by bipartite matching of the prediction result and the true value of path k, Represents the true value of the bounding box position of feature n, represents the true classification value of feature q, CW(·,·) represents the cross entropy of the two classification results, represents the mean absolute loss of the bounding box position prediction results of feature n in path k, represents the generalized intersection-over-union ratio of the bounding box size of feature n in path k, represents the classification cross entropy of the prediction result of feature q in path k, λ4, λ5 and They are all hyperparameters. In the end, the total loss function for:

[0103]

[0104] Here, w(t) is a linear function that increases with the number of training rounds, for example, w(t) = t + 1. As the number of training rounds increases, the weight of the consistency loss gradually increases. Based on the final loss, the gradient is backpropagated to optimize the model and complete the training.

[0105] During the training process, a total of 90 rounds of training were conducted with a batch size of 12. The learning rates of ResNet and encoder-decoder were initialized to 0.00001 and 0.0001, respectively. The AdamW algorithm was used for 10 iterative trainings and the results of each iteration were saved. In the 60th round, the learning rates were reduced to one-tenth of the original values.

[0106] Step S5: Take the optimal deep learning model after training and output the prediction results for the interaction image to be detected. Consistency learning is only used during training. In actual application, only the prediction results output by the main path are used.

[0107] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for detecting interaction between secondary and primary paths based on encoder-decoder learning, which performs interaction detection by constructing a deep learning model, characterized in that: The deep learning model includes a feature extraction module, a dual-path learning module, and an interaction prediction module. The encoder-decoder-based secondary-path guided primary-path learning interaction detection method includes: Taking the labeled interaction picture and inputting it into the feature extraction module to obtain the picture features output by the feature extraction module, the feature extraction module includes a deep residual network and an encoder; Inputting the image features into a primary path and a secondary path of the dual-path learning module, respectively, to obtain features for people, objects, and interactions output by the primary path and the secondary path, respectively, wherein the primary path includes a decoder unit and the secondary path includes a two-stage decoder unit; Input the features for people, objects and interactions output by the primary and secondary paths into the interaction prediction module to obtain prediction results Pr1, Pr2∈{people, objects, interactions} corresponding to the primary and secondary paths respectively; Perform consistency learning based on the prediction results corresponding to the primary path and the secondary path, perform loss supervision on the primary path and the secondary path, and update the network parameters of the deep learning model; The best deep learning model after training is used to output the prediction results for the interaction image to be detected.

2. The encoder-decoder-based secondary path guided primary path learning interaction detection method according to claim 1, characterized in that: The deep residual network is the deep residual network ResNet-50.

3. The encoder-decoder-based secondary path guided primary path learning interaction detection method according to claim 1, characterized in that: The processing of the decoder unit includes: Take the input image features as the query library X and initialize a query Q hoi , using query Q hoi Get features for people, objects, and interactions in the query repository X The processing of the two-stage decoder unit includes: Take the input image features as the query library X and initialize the query Q for people and objects h ,Q o , query Q h ,Q o Merge into query Q ins , using query Q ins Get the features for people in the query library X and characteristics of objects Then the features and Weighted feature The features The query Q′ as the initial interaction of prior information inter , by querying Q′ inter Get features for interaction in query library X The features and features Perform feature fusion to obtain the final interactive features of the secondary path 4. The encoder-decoder-based secondary path guided primary path learning interaction detection method according to claim 3, characterized in that: The characteristics and features Perform feature fusion to obtain the final interactive features of the secondary path include: The features and After splicing, fusion is performed and the fusion result is normalized to obtain the instance attention weight α, which is specifically expressed as follows: Among them, sigmoid(·) represents normalization processing, MLP1(·) represents feature fusion, and concat(·,·) represents feature concatenation; Then the features And the instance attention weight α is multiplied to obtain the instance enhanced interaction feature, and finally the feature is connected by short circuit. Instance-enhanced interaction features and features mapped to the interaction domain Add together to get the final features for interaction The specific expressions are as follows: Here, ⊙ represents dot product and MLP2(·) represents spatial mapping.

5. The encoder-decoder-based secondary path guided primary path learning interaction detection method according to claim 3, characterized in that: The prediction results are as follows: Among them, Pr1 and Pr2 represent the main road Heci Road The prediction results, Represent the regression or classification functions for people on the main road and the secondary road respectively, Represent the regression or classification functions for objects on the main road and the secondary road respectively, Represent the regression or classification functions used for interaction on the main road and the secondary road respectively, They represent the prediction results output by the corresponding regression or classification function respectively.

6. The encoder-decoder-based secondary path guided primary path learning interaction detection method according to claim 5, characterized in that: The performing consistency learning according to the prediction results corresponding to the primary path and the secondary path includes: Get the prediction set h corresponds to people, o corresponds to objects, and inter corresponds to interactions. The binary matching relationship between the prediction result and the true value of each path is defined as Represents the true value, and the three-way prediction results output by the primary and secondary paths are paired according to the bipartite matching relationship to obtain the two-way pairing set δ; After obtaining the correspondence between the two paths, the KLoss method is used to measure the similarity of the two path matrices based on the consistency in classification. The mathematical expression is as follows: in, is the similarity of the prediction results of the two paths for the features used for objects, is the similarity of the prediction results of the two paths for the features used for interaction, KLoss(·||·) represents the divergence of two features or two probability distributions, z(f,m) represents the mth f, Represents the prediction result of feature q in path k, q∈(o,inter); For the consistency of bounding box positions, the mean absolute error is used to measure the consistency between paired groups. The mathematical expression is as follows: in, represents the mean absolute loss of the bounding box position prediction results for human features, represents the mean absolute loss of the bounding box position prediction results for object features, ||·||1 represents the L1 paradigm, represents the bounding box position prediction result of feature n in path k, n∈(h,o), The total number of sets representing true values; So the consistency loss function of the two paths is It is expressed as follows: Among them, λ1, λ2 and λ3 are hyperparameters used to adjust the loss ratio.

7. The encoder-decoder-based secondary path guided primary path learning interaction detection method according to claim 6, characterized in that: The loss monitoring of the primary and secondary roads includes: Each path consists of three losses: the mean absolute error of the bounding box position, the generalized intersection-over-union ratio of the bounding box size, and the classification cross entropy. Therefore, the interaction triplet loss function of the two paths is The mathematical expression is as follows: Among them, θ k Represents the pairing set obtained by bipartite matching of the prediction result and the true value of path k, Represents the true value of the bounding box position of feature n, represents the true classification value of feature q, CE(·,·) represents the cross entropy of the two classification results, represents the mean absolute loss of the bounding box position prediction results of feature n in path k, represents the generalized intersection-over-union ratio of the bounding box size of feature n in path k, represents the classification cross entropy of the prediction result of feature q in path k, λ4, λ5 and They are all hyperparameters. In the end, the total loss function for: Among them, w(t) is a linear growth function with the number of training rounds t. As the number of training rounds t increases, the proportion of consistency loss gradually increases.

Images