Entity interaction detection method, method and device for establishing entity interaction detection model

Abstract

The embodiment of the application discloses a kind of entity interaction detection method, the method and device for establishing entity interaction detection model, it is related to artificial intelligence technical field.The method comprises: embedding processing is carried out to the image to be detected, and the embedding feature of the image to be detected is obtained;The embedding feature of the image to be detected is input into entity interaction detection model;Encoding processing is carried out by shared coding network in entity interaction detection model, and the feature representation of the image to be detected is obtained;Entity decoding network predicts the position information of the first entity and the second entity contained in the image to be detected using the feature representation of the image to be detected;Interaction decoding network predicts the interaction type information contained in the image to be detected and the pointer representation of the type interaction relationship using the feature representation of the image to be detected;Correlation matching network associates the prediction results of entity decoding network and interaction decoding network, and obtains the interaction information between the first entity and the second entity.The application can improve the accuracy of entity interaction detection.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an entity interaction detection method, a method and apparatus for establishing an entity interaction detection model. Background Technology

[0002] Various service scenarios generate massive amounts of image data, which can effectively reflect service status. One image analysis requirement is to detect the interaction relationships between entities in images. For example, human-object interaction (HOI) detection detects the interaction relationships between people and objects, thus providing valuable information for subsequent data mining. Therefore, a method that can accurately detect entity interactions is needed. Summary of the Invention

[0003] In view of this, this application provides a highly accurate entity interaction detection method, a method and apparatus for establishing an entity interaction detection model.

[0004] This application provides the following solution:

[0005] Firstly, a method for entity interaction detection is provided, the method comprising:

[0006] The image to be detected is embedded to obtain its embedding features;

[0007] The embedded features of the image to be detected are input into the entity interaction detection model to obtain the interaction information between the first entity and the second entity in the image to be detected.

[0008] The entity interaction detection model includes a shared encoding network, an entity decoding network, an interaction decoding network, and an association matching network. The shared encoding network encodes the embedded features of the image to be detected to obtain a feature representation of the image to be detected. The entity decoding network uses the feature representation of the image to be detected to predict the location information of the first entity and the second entity contained in the image to be detected. The interaction decoding network uses the feature representation of the image to be detected to predict the interaction type information contained in the image to be detected and the pointer representation of the interaction relationship of that type. The association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity.

[0009] According to one achievable method in the embodiments of this application, the embedding process of the image to be detected to obtain the embedding features of the image to be detected includes: performing content-based embedding and position-based embedding on the image blocks in the image to be detected respectively to obtain the content embedding features and position embedding features of the image to be detected.

[0010] The shared coding network is implemented based on a multi-layer Transformer, which includes a self-attention layer and a feedforward network (FFN) layer.

[0011] The self-attention layer uses the concatenation result of the content embedding feature and the position embedding feature to obtain the query vector and the key vector, and uses the content embedding feature to obtain the value vector. After performing attention processing on the query, key and value, the processing result is output to the FFN layer.

[0012] According to one achievable method in an embodiment of this application, the interactive decoding network includes a multi-layer decoder and a mapping layer;

[0013] The multi-layer decoder uses the feature representation of the image to be detected and the first interactive query to perform decoding processing to obtain the second feature representation of the image to be detected. The first interactive query is learned by the entity interaction model during training.

[0014] The mapping layer uses the second feature representation of the image to be detected to perform mapping, thereby obtaining interaction type information and pointer representations of interaction relationships of that type.

[0015] According to one achievable method in an embodiment of this application, each decoder in the multi-layer decoder includes a self-attention layer, a cross-attention layer, and an FFN layer;

[0016] The cross-attention layer obtains the query by concatenating the output of the self-attention layer and the second interactive query, obtains the key and value by concatenating the output of the shared coding network and the location embedding feature, and after attention processing using the query, key and value, the processing result is output to the FFN layer.

[0017] The second interactive query is obtained from the interactive reference point and the output of the previous layer decoder. The interactive reference point is determined by the reference point of the first entity and the reference point of the second entity. The reference point of the first entity and the reference point of the second entity are predicted based on the first interactive query.

[0018] According to one achievable method in the embodiments of this application, the self-attention layer in the first layer decoder of the interactive decoding network obtains the query and key by concatenating the output of the shared encoder and the first interactive query, obtains the value by using the output of the shared encoder, and outputs the processing result to the cross-attention layer after performing attention processing on the query, key and value.

[0019] The self-attention layer in the other layers of the interactive decoding network uses the concatenation result of the output of the previous layer decoder and the first interactive query to obtain the query and key, uses the output of the previous layer decoder to obtain the value, and after performing attention processing on the query, key and value, outputs the processing result to the cross-attention layer.

[0020] According to one achievable method in an embodiment of this application, the association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity, including:

[0021] The association matching network matches the location information of the first entity and the second entity with pointer representations that have an interaction relationship. If a matching pointer representation exists, it is determined that the first entity and the second entity have an interaction, and the interaction type is the type corresponding to the matched pointer representation.

[0022] Secondly, a method for establishing an entity interaction detection model is provided, the method comprising:

[0023] Acquire training data, which includes image samples and labels that annotate the interaction information between a first entity and a second entity in the image samples;

[0024] Embedding processing is performed on the image samples in the training data to obtain the embedding features of the image samples;

[0025] The embedded features of the image samples are used as input to the entity interaction detection model to train the entity interaction detection model. The training objective includes minimizing the difference between the interaction information between the first entity and the second entity output by the entity interaction detection model and the label of the image sample in the training data.

[0026] The entity interaction detection model includes a shared encoding network, an entity decoding network, an interaction decoding network, and an association matching network. The shared encoding network encodes the embedded features of the image sample to obtain a feature representation of the image sample. The entity decoding network uses the feature representation of the image sample to predict the location information of the first entity and the second entity contained in the image sample. The interaction decoding network uses the feature representation of the image sample to predict the interaction type information contained in the image sample and the pointer representation of the interaction relationship of that type. The association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity.

[0027] According to one achievable method in the embodiments of this application, the image samples in the training data are embedded to obtain the embedding features of the image samples, including: performing content-based embedding and location-based embedding on the image blocks in the image samples respectively to obtain the content embedding features and location embedding features of the image samples.

[0028] The shared coding network is implemented based on a multi-layer Transformer, which includes a self-attention layer and an FFN layer.

[0029] The self-attention layer obtains the query and key by concatenating the content embedding feature and the position embedding feature, obtains the value by using the content embedding feature, performs attention processing on the query, key and value, and outputs the processing result to the FFN layer.

[0030] According to one achievable method in an embodiment of this application, the interactive decoding network includes a multi-layer decoder and a mapping layer;

[0031] The multi-layer decoder uses the feature representation of the image sample and the first interactive query to perform decoding processing to obtain the second feature representation of the image sample. The first interactive query is updated during the training process after initialization.

[0032] The mapping layer uses the second feature representation of the image sample to perform mapping to obtain interaction information, which includes interaction type information and pointer representation of the existence of interaction relationship of that type.

[0033] According to one achievable embodiment of this application, the decoder includes a self-attention layer, a cross-attention layer, and an FFN layer;

[0034] The cross-attention layer obtains the query by concatenating the output of the self-attention layer and the second interactive query, obtains the key and value by concatenating the output of the shared coding network and the location embedding feature, and after attention processing using the query, key and value, the processing result is output to the FFN layer.

[0035] The second interactive query is obtained from the interactive reference point and the output of the previous layer decoder. The interactive reference point is determined by the reference point of the first entity and the reference point of the second entity. The reference point of the first entity and the reference point of the second entity are predicted based on the first interactive query.

[0036] According to one achievable method in the embodiments of this application, the self-attention layer in the first layer decoder of the interactive decoding network obtains the query and key by concatenating the output of the shared encoder and the first interactive query, obtains the value by using the output of the shared encoder, and outputs the processing result to the cross-attention layer after performing attention processing on the query, key and value.

[0037] The self-attention layer in the other layers of the interactive decoding network uses the concatenation result of the output of the previous layer decoder and the first interactive query to obtain the query and key, uses the output of the previous layer decoder to obtain the value, and after performing attention processing on the query, key and value, outputs the processing result to the cross-attention layer.

[0038] According to one achievable method in an embodiment of this application, the association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity, including:

[0039] The association matching network matches the location information of the first entity and the second entity with pointer representations that have an interaction relationship. If a matching pointer representation exists, it is determined that the first entity and the second entity have an interaction, and the interaction type is the type corresponding to the matched pointer representation.

[0040] Thirdly, an entity interaction detection device is provided, the device comprising:

[0041] The image acquisition unit is configured to acquire the image to be detected;

[0042] An embedding processing unit is configured to perform embedding processing on the image to be detected to obtain the embedding features of the image to be detected;

[0043] An interaction detection unit is configured to input the embedded features of the image to be detected into an entity interaction detection model to obtain the interaction information between the first entity and the second entity in the image to be detected.

[0044] The entity interaction detection model includes a shared encoding network, an entity decoding network, an interaction decoding network, and an association matching network. The shared encoding network encodes the embedded features of the image to be detected to obtain a feature representation of the image to be detected. The entity decoding network uses the feature representation of the image to be detected to predict the location information of the first entity and the second entity contained in the image to be detected. The interaction decoding network uses the feature representation of the image to be detected to predict the interaction type information contained in the image to be detected and the pointer representation of the interaction relationship of that type. The association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity.

[0045] Fourthly, an apparatus for establishing an entity interaction detection model is provided, the apparatus comprising:

[0046] The sample acquisition unit is configured to acquire training data, which includes image samples and labels that annotate the interaction information between a first entity and a second entity in the image samples.

[0047] An embedding processing unit is configured to perform embedding processing on image samples in the training data to obtain embedding features of the image samples;

[0048] The model training unit is configured to take the embedded features of the image samples as input to the entity interaction detection model and train the entity interaction detection model. The training objective includes minimizing the difference between the interaction information between the first entity and the second entity output by the entity interaction detection model and the label of the image sample in the training data.

[0049] The entity interaction detection model includes a shared encoding network, an entity decoding network, an interaction decoding network, and an association matching network. The shared encoding network encodes the embedded features of the image sample to obtain a feature representation of the image sample. The entity decoding network uses the feature representation of the image sample to predict the location information of the first entity and the second entity contained in the image sample. The interaction decoding network uses the feature representation of the image sample to predict the interaction type information contained in the image sample and the pointer representation of the interaction relationship of that type. The association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity.

[0050] According to a fifth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0051] According to a sixth aspect, an electronic device is provided, characterized in that it comprises:

[0052] One or more processors; and

[0053] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first aspects above.

[0054] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0055] 1) This application employs an end-to-end model for entity interaction detection. After encoding the embedded features of the image to be detected using a shared coding network, two parallel entity decoding networks and interaction decoding networks predict entity prediction and interaction representations, respectively. Then, an association matching network correlates the two prediction results to obtain the interaction information between the first and second entities. This end-to-end prediction method avoids error accumulation, thereby improving the accuracy of entity interaction detection. Furthermore, the parallel prediction based on the embedded features of the image to be detected by the entity decoding network and interaction decoding network significantly improves detection efficiency.

[0056] 2) The shared coding network uses the content embedding features and position embedding features of the image to be detected for encoding to obtain the feature representation of the image to be detected. This enables the shared coding network to distinguish the positional relationship of image blocks during encoding, thereby enhancing the feature representation globally and further improving the accuracy of interactive detection.

[0057] 3) The cross-attention layer of the cross-decoding network uses the output of the self-attention layer and the second interaction query to obtain the query. The second interaction query is obtained by the interaction reference point and the output of the previous layer decoder, which is more conducive to locating the interaction-related region in the image to be detected, narrowing the query range of the cross-attention module and improving the accuracy of interaction detection.

[0058] 4) The association matching network matches the location information of the first and second entities with pointer representations that have an interaction relationship. If a matching pointer representation exists, it is determined that the first and second entities have an interaction and the interaction type is the type corresponding to the matched pointer representation. This approach comprehensively considers the results of entity prediction and interaction prediction, resulting in more accurate interaction detection.

[0059] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a system architecture diagram applicable to the embodiments of this application;

[0062] Figure 2 A flowchart of the entity interaction detection method provided in the embodiments of this application;

[0063] Figure 3 A schematic structural diagram of the entity interaction detection model provided in the embodiments of this application;

[0064] Figure 4 A schematic structural diagram of the interactive decoding network provided in the embodiments of this application;

[0065] Figure 5 A schematic structural diagram of the decoder in the interactive decoding network provided in the embodiments of this application;

[0066] Figure 6 A flowchart illustrating the method for establishing an entity interaction detection model provided in this application embodiment;

[0067] Figure 7 A schematic structural diagram of the entity interaction detection device provided in the embodiments of this application;

[0068] Figure 8 A schematic structural diagram illustrating the establishment of an entity interaction detection model provided in an embodiment of this application;

[0069] Figure 9 This is an architectural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0071] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0072] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0073] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0074] Current HOI detection methods primarily employ a two-stage approach, decoupling the HOI detection task into object detection and interaction classification tasks. First, an object detection network performs object detection. The detected objects (people and objects) are then paired and input into an interaction classification network for interaction category detection. However, this method has the following drawbacks:

[0075] 1) This two-stage detection method requires pairwise combinations of people and objects detected by the target, and then inputting them into the interaction classification network. The interaction classification network then performs interaction category detection for each combination, which is inefficient.

[0076] 2) In this two-stage detection method, the results of interactive category detection are easily affected by the accuracy of the first-stage target detection, resulting in error accumulation and thus low accuracy of the entire HOI detection.

[0077] In view of this, this application adopts a novel approach, using an end-to-end method to implement entity interaction detection. To facilitate understanding of this application, the system architecture to which this application is applied will be briefly described first. Figure 1 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 1 As shown, the system architecture includes a model building device that builds an entity interaction detection model offline, and an entity interaction detection device that performs entity interaction detection online.

[0078] After acquiring training data, the model building device can use the method provided in the embodiments of this application to train the model and obtain an entity interaction detection model.

[0079] The entity interaction detection device uses an established entity interaction detection model to perform entity interaction detection on the image to be detected, obtaining the interaction information between the first entity and the second entity contained in the image. It typically outputs triples, which include: the first entity, the second entity, and the interaction information between the first and second entities.

[0080] The model building device and the entity interaction detection device can be configured as independent servers, or they can be configured on the same server or server group, or they can be configured on a separate server or the same cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPs) services, such as high management difficulty and weak service scalability. The model training device and the prediction processing device can also be configured on a computer terminal with strong computing power.

[0081] It should be understood that Figure 1 The number of model building devices, entity interaction detection devices, and entity interaction detection models shown in the diagram is merely illustrative. Depending on implementation needs, any number of model building devices, entity interaction detection devices, and entity interaction detection models can be included.

[0082] Figure 2 This is a flowchart of an entity interaction detection method provided in an embodiment of this application. The method can be... Figure 1 The entity interaction detection device in the system shown is executed. For example... Figure 2 As shown, the method may include:

[0083] Step 202: Perform embedding processing on the image to be detected to obtain the embedding features of the image to be detected.

[0084] Step 204: Input the embedded features of the image to be detected into the entity interaction detection model to obtain the interaction information between the first entity and the second entity in the image to be detected; wherein the entity interaction detection model includes: a shared encoding network, an entity decoding network, an interaction decoding network, and an association matching network; the shared encoding network encodes the embedded features of the image to be detected to obtain the feature representation of the image to be detected; the entity decoding network uses the feature representation of the image to be detected to predict the position information of the first entity and the second entity contained in the image to be detected; the interaction decoding network uses the feature representation of the image to be detected to predict the interaction type information contained in the image to be detected and the pointer representation of the interaction relationship of that type; the association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity.

[0085] As can be seen from the above process, this embodiment of the application uses an end-to-end model for entity interaction detection. After encoding the embedded features of the image to be detected using a shared coding network, two parallel entity decoding networks and interaction decoding networks predict entity prediction and interaction representations, respectively. Then, an association matching network associates the two prediction results to obtain the interaction information between the first entity and the second entity. This end-to-end prediction method avoids error accumulation, thereby improving the accuracy of entity interaction detection. Furthermore, the parallel prediction based on the embedded features of the image to be detected by the entity decoding network and interaction decoding network significantly improves detection efficiency.

[0086] It should be noted that the terms "first," "second," etc., used in the embodiments of this application do not have limitations on size, order, or quantity; they are merely used to distinguish them by name. "First entity" and "second entity" are used only to distinguish two entities by name; these two entities can be of different types or the same type. For example, the first entity is a person, and the second entity is an object. Another example: the first entity is a person, and the second entity is another person. Yet another example: the first entity is an object, and the second entity is another object. And so on. "First feature representation" and "second feature representation" are used only to distinguish two feature representations by name. Other similar examples are not listed here.

[0087] The following sections describe each step in the above process in detail.

[0088] First, the above step 202, namely "performing embedding processing on the image to be detected to obtain the embedding features of the image to be detected", will be described in detail with reference to the embodiments.

[0089] Depending on the specific application scenario of entity interaction detection, the image to be detected can come from different sources. For example, in the field of autonomous driving, the image to be detected can be an environmental image captured by an autonomous vehicle. In the field of intelligent transportation, the image to be detected can be an image of the road captured by a traffic camera, and so on.

[0090] The embedding process performed on the image to be detected in this step can include content-based embedding and location-based embedding.

[0091] Content-based embedding can be performed on a sequence of image patches of the image to be detected. This involves encoding each image patch with graph vectors, and the resulting graph vector representations of these patches constitute the content embedding features of the image to be detected. Content-based embedding can be implemented using a backbone network (such as ResNet50).

[0092] Location-based embedding involves performing location-based embedding processing on a sequence of image patches in the image to be detected. This means encoding the position of each image patch within the image to be detected, thus obtaining a positional representation. When encoding the position of the image patch within the image to be detected, formats such as cosine coding or sine coding can be used.

[0093] In one preferred embodiment, before embedding the image to be detected, the image can be preprocessed to normalize it into image blocks of the same number and size. The embedding process is then performed on these image blocks, mapping each block to a fixed dimension to obtain a fixed-dimensional vector, which represents the content embedding feature and the location embedding feature mentioned above.

[0094] The following describes in detail step 204, namely, "inputting the embedded features of the image to be detected into the entity interaction detection model to obtain the interaction information between the first entity and the second entity in the image to be detected," with reference to the embodiments.

[0095] The entity interaction detection model provided in this application embodiment can be as follows: Figure 3 As shown, it includes a shared encoding network, an entity decoding network, an interactive decoding network, and an association matching network.

[0096] Shared coding networks are used to encode the embedded features of the image to be detected, thereby obtaining a feature representation of the image.

[0097] The shared coding network can have multiple encoders, for example, it can be implemented based on multiple Transformers. Each Transformer layer can include an attention layer and an FFN (feed-forward network) layer. The attention layer needs a query vector, a key vector, and a value vector when performing attention processing. As a preferred implementation, the query and key can be obtained by concatenating content embedding features and location embedding features, and the value can be obtained by using content embedding features. Then, the query, key, and value are used for attention processing, and the processing result is output to the FFN layer.

[0098] As can be seen, the content embedding features and the position encoding features are jointly input into the Transformer-based shared encoder. The shared encoder can distinguish the relative position information in the features and model the global relationship through the self-attention mechanism, thereby enhancing the feature representation.

[0099] One possible approach is to configure the shared coding network to have 6 encoder layers, and the dimension of the FFN can be set to 1024.

[0100] After obtaining the feature representation of the image to be detected, the shared coding network outputs it to two parallel decoding networks: an entity decoding network and an interaction decoding network. The entity decoding network is responsible for finding regions related to entities. The interaction decoding network finds regions related to interactions.

[0101] The entity decoding network is used for target detection. In this embodiment, it uses the feature representation of the image to be detected to predict the location information of the first and second entities contained in the image. The entity decoding network may include multiple layers of decoders and mapping layers.

[0102] The multi-layer decoder uses the feature representation of the image to be detected for decoding to obtain the first feature representation of each image block in the image to be detected. The multi-layer decoder can be implemented based on Transformer.

[0103] The mapping layer uses the first feature representation of each image patch for mapping to obtain the entity type information and location information of each image patch. For example, the mapping layer can use a classification network to classify whether each image patch belongs to a specific type of entity, and obtain the location information of each entity, such as the bounding box of each entity, based on the classification result. In this embodiment, for M image patches, M queries are generated and self-attention processing is performed in a multi-layer decoder to obtain the first feature representation of M image patches. The mapping layer maps the first feature representation to a preset entity type, thereby obtaining the result of whether each image patch belongs to that entity type. In this embodiment, what needs to be detected finally is the interaction information between two entities, namely a first entity and a second entity. Then, the entity type information obtained by the entity decoding network can include whether it is the first entity and whether it is the second entity. For example, the type of the first entity is a person, and the type of the second entity is a certain object, etc.

[0104] The interactive decoding network uses the feature representation of the image to predict the interaction type information contained in the image and the pointer representation of the interaction relationship of that type.

[0105] The structure of an interactive decoding network can be as follows: Figure 4 As shown, it includes a multi-layer decoder and a mapping layer. The multi-layer decoder performs decoding processing using the feature representation of the image to be detected, obtaining a second feature representation of the image to be detected. The structure of each decoder may include a self-attention layer, a cross-attention layer, and an FFN layer.

[0106] One possible approach is for the self-attention layer in the first decoder to obtain the query, key, and value from the feature representation of the image to be detected output by the shared coding network. After attention processing, the result is output to the cross-attention layer. Subsequent self-attention layers in the decoder use the output of the previous layer to obtain the query, key, and value, perform attention processing, and then output the result to the cross-attention layer. This self-attention mechanism enables the model to notice the correlations between different image patches in the entire input, achieving an effect similar to non-maximum suppression.

[0107] Assuming the input to the self-attention layer is X, the query, key, and value obtained using X can be represented as Q, K, and V, respectively, using the following formula:

[0108] Q = W Q X (1)

[0109] K = W k X (2)

[0110] V = W v X (3)

[0111] Among them, W Q W k and W v is the weight matrix, which contains the parameters that the model needs to learn.

[0112] Then, the following attention processing is performed:

[0113]

[0114] Where Attention() represents the attention processing procedure, Q, K, and V are matrices composed of the query vector, key vector, and value vector, respectively, and d k K represents the spatial dimension.

[0115] As another possible approach, such as Figure 5As shown, the self-attention layer in the first decoder layer uses the concatenation result of the shared encoder output and the first interaction query to obtain the query (represented as V) and key (represented as K) in the figure, and uses the output of the shared encoder to obtain the value (represented as V) in the figure. After performing attention processing on the query, key, and value, the processing result is output to the cross-attention layer. The self-attention layers in other decoder layers use the concatenation result of the previous decoder output and the first interaction query to obtain the query and key, use the output of the previous decoder to obtain the value, and use the query, key, and value to perform attention processing on the query, key, and value, and the processing result is output to the cross-attention layer.

[0116] Since the interactions represent the relationships between image patches, the number of first interaction queries is determined by the number of image patches. The number of first interaction queries, N, is M×M. The multi-layer decoder of the final interaction decoding network obtains N interaction representations, which constitute the second feature representation mentioned above. The first interaction queries are obtained during model training, which will be explained in detail in the subsequent model training process.

[0117] like Figure 5 As shown, the cross-attention layer obtains the query by concatenating the output of the self-attention layer (content query) and the second interaction query (conditional interaction query), and obtains the key and value by concatenating the output of the shared coding network (content key) and the spatial embedding feature (spatial key). Then, after attention processing using the query, key, and value, the processing result is output to the FFN layer.

[0118] The second conditional interaction query is a query specifically designed in this application embodiment. The resulting vector is unrelated to the image content and represents more spatial information, helping to locate interaction-related regions. Therefore, when designing the second conditional interaction query, two main factors are considered: the output f of the previous layer decoder and the interaction reference point. a For example, the following formula can be used:

[0119] p q =Tp s (5)

[0120] T = MLP(f) (6)

[0121] ps =sinusoidal(point a (7)

[0122] point a =α*o_reference_point+(1-α)*h_reference_point (8)

[0123] Where α is a hyperparameter, taking values ​​from the range [0,1], and can be empirical or experimental values. h_reference_point and o_reference_point are the reference points of the first and second entities, respectively, and are two-dimensional coordinates predicted by the first interaction query. These two-dimensional coordinates are used to represent the reference positions of the first and second entities. sinusoidal(point) a ) indicates that the point a Mapped to a sinusoidal encoding space to make it the same dimension as the spatial key. MLP(f) is processed by f through a feedforward layer (e.g., through two multilayer perceptrons).

[0124] After obtaining the query, key, and value, attention processing is performed as shown in formula (4), and the processing result is output to FFN.

[0125] The mapping layer (also known as the detector head) in the interactive decoding network predicts pointer representations of image patches with interactions and interaction type information. The interaction type information can be implemented using a classification network or a regression network, which will not be detailed here.

[0126] For pointer representations of interacting image blocks, taking the first entity and the second entity as an example, if the first entity and the second entity interact, then the pointer representation H_Pointer_reprs of the image block containing the first entity and the pointer representation O_Pointer_reprs of the image block containing the second entity can be represented by the following formulas:

[0127] H_Pointer_reprs=normalize(FFN h1 (f)+FFN h2 (h_reference_point)) (9)

[0128] O_Pointer_reprs=normalize(FFN o1 (f)+FFN o2 (o_reference_point)) (10)

[0129] In equations (9) and (10), f represents the output of the last decoder layer in the interactive decoding network. The mapping layer can consist of multiple FFNs, for example, FFN... h1 FFN h2 FFN o1 and FFN o2 FFN in the above formula h1 (), FFN h2 (), FFN o1 () and FFN o2 () are FFN h1 FFN h2 FFN o1 and FFN o2 The corresponding processing function. First, from FFN h1 and FFN o1 The output f of the multi-layer decoder is processed to obtain preliminary pointer representations of the first and second entities, which are then processed by FFN. h2 and FFN o2 Map the reference points of the first and second entities to FFN. h1 (f) FFN o1 (f) The same dimensions are used, and the results of the two are added together. Normalize() refers to performing the L2 norm.

[0130] See also Figure 3 The association matching network is used to associate the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity.

[0131] As one possible approach, the association matching network can match the location information of the first entity and the second entity with pointer representations that have an interaction relationship. If a matching pointer representation exists, it is determined that the first entity and the second entity have an interaction, and the interaction type is the type corresponding to the matched pointer representation.

[0132] Since the entity decoding network outputs the prediction result of the entity type, such as whether each image patch in the image belongs to the first entity or the second entity, the positional information of the first and second entities is obtained. Positional information can be represented as bounding box information or as information about the image patches included by the entity. The interaction decoding network outputs interaction type information and pointer representations indicating the existence of that interaction type. For example, it can output multiple interaction information, each of which can include: interaction type 1, pointer representation 1, and pointer representation 2. Specifically, the image patch pointed to by pointer representation 1 and the image patch pointed to by pointer representation 2 have an interaction as shown in interaction type 1. The positional information of the first and second entities output by the entity decoding network is matched with the interaction information output by the interaction decoding network. For example, if the positional information of the first entity matches pointer representation 1 and the positional information of the second entity matches pointer representation 2, then there is an interaction as shown in interaction type 1 between the first and second entities. The matching of the positional information with the pointer representations can include: closest distance, distance less than a preset threshold, etc.

[0133] The final association matching network can output triple information, which includes information about a first entity, information about a second entity, and interaction information between the first and second entities. The first and second entity information may include entity type and location information, and the interaction information between the first and second entities may include interaction type information.

[0134] The method for establishing the above entity interaction detection model is described below. Figure 6 This is a flowchart illustrating a method for establishing an entity interaction detection model provided in an embodiment of this application. This method can be... Figure 1 The model building device in the system shown is executed. For example... Figure 6 As shown, the method may include the following steps:

[0135] Step 602: Obtain training data, which includes image samples and labels that annotate the interaction information between the first entity and the second entity in the image samples.

[0136] In this embodiment, some images can be pre-acquired as image samples, and the interaction information between the first entity and the second entity in the image samples can be labeled. Furthermore, the location information and type of the first and second entities can be further labeled. Depending on the specific application scenario of entity interaction detection, the images to be detected can have different sources. For example, in the field of autonomous driving, the image samples can be environmental images collected by autonomous vehicles. As another example, in the field of intelligent transportation, the image samples can be images of the road collected by traffic cameras, and so on.

[0137] Depending on the specific entity detection requirements, image samples containing specific types of entities and interactions between entities can be obtained. For example, images containing interactions between people and objects can be acquired as image samples. The location information of people, objects, and the type of interaction between people and objects in the image samples can be labeled to obtain location labels for people, location labels for objects, and interaction type labels.

[0138] The labeled image samples are used to construct training data for training the subsequent entity interaction detection model.

[0139] Step 604: Embed the image samples in the training data to obtain the embedding features of the image samples.

[0140] Embedding processing of image samples can include content-based embedding and location-based embedding. That is, content-based embedding and location-based embedding are performed on image patches in the image sample respectively to obtain the content embedding features and location embedding features of the image sample.

[0141] Content-based embedding can be performed on a sequence of image patches of an image sample. This involves encoding each image patch using image vectors, resulting in graph vector representations of each patch that constitute the content embedding features of the image sample. Content-based embedding can be implemented using a backbone network (e.g., ResNet50).

[0142] Position-based embedding can be performed on a sequence of image patches in an image sample, that is, encoding the position of each image patch in the image sample to obtain a positional representation. When encoding the position of the image patch in the image sample, formats such as cosine coding and sine coding can be used.

[0143] As a preferred implementation, before embedding the image samples, they can be preprocessed to normalize them into image patches of the same number and size. The embedding process is then based on these image patches, mapping each patch to a fixed dimension to obtain a fixed-dimensional vector, which represents the content embedding feature and the location embedding feature mentioned above.

[0144] Step 606: Use the embedded features of the image samples as input to the entity interaction detection model to train the entity interaction detection model. The training objective includes minimizing the difference between the interaction information between the first entity and the second entity output by the entity interaction detection model and the label of the image sample in the training data.

[0145] The structure of the entity interaction detection model is as follows: Figure 3As shown, it includes: a shared encoding network, an entity decoding network, an interactive decoding network, and an association matching network.

[0146] During training, the shared coding network encodes the embedded features of image samples to obtain feature representations of the image samples.

[0147] Shared coding networks can have multiple encoders, for example, they can be implemented based on multiple Transformers, which include self-attention layers and FFN layers.

[0148] The self-attention layer can obtain the query and key by concatenating the content embedding features and the position embedding features, obtain the value by using the content embedding features, and then perform attention processing on the query, key and value before outputting the processing result to the FFN layer.

[0149] Content embedding features and position encoding features are jointly input into a Transformer-based shared encoder. The shared encoder can distinguish the relative position information in the features and model the global relationship through a self-attention mechanism, thereby enhancing the feature representation.

[0150] After obtaining the feature representations of the image samples, the shared encoding network outputs them to two parallel decoding networks: an entity decoding network and an interaction decoding network. The entity decoding network essentially finds the regions related to entities. The interaction decoding network finds the regions related to interactions.

[0151] The entity decoding network uses the feature representation of image samples to predict the location information of the first and second entities contained in the image samples.

[0152] Entity decoding networks can include multiple layers of decoders and mapping layers.

[0153] A multi-layer decoder uses the feature representations of image samples for decoding to obtain the first feature representation of each image patch in the image sample. The multi-layer decoder can be implemented based on Transformer.

[0154] The mapping layer uses the first feature representation of each image patch to perform mapping, obtaining entity type information and location information for each image patch. For example, the mapping layer can use a classification network to classify whether each image patch belongs to a specific type of entity, and obtain the location information of each entity, such as the bounding box of each entity, based on the classification result. In this embodiment, the entity type information may include whether it is a first entity and whether it is a second entity.

[0155] The interaction decoding network uses the feature representation of image samples to predict the interaction type information contained in the image samples and the pointer representation of the interaction relationship of that type; the association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information of the first entity and the second entity.

[0156] The structure of an interactive decoding network can be as follows: Figure 4 As shown, it includes a multi-layer decoder and a mapping layer.

[0157] The multi-layer decoder uses the feature representations of image samples for decoding, obtaining a second feature representation of the image samples. This second feature representation differs from the first feature representation obtained in the entity decoder; it learns the interaction feature representations in the image. The structure of each decoder in the multi-layer decoder is described below. Each decoder's structure may include a self-attention layer, a cross-attention layer, and an FFN layer.

[0158] One possible approach is to use the self-attention layer in the first decoder layer to obtain the query, key, and value from the feature representation of the image samples output by the shared coding network. After attention processing, the result is output to the cross-attention layer. Subsequent self-attention layers in each decoder layer use the output of the previous layer to obtain the query, key, and value, perform attention processing, and then output the result to the cross-attention layer. This self-attention mechanism enables the model to notice the correlations between different image patches in the entire input, achieving an effect similar to non-maximum suppression.

[0159] As another possible approach, the self-attention layer in the first-layer decoder uses the concatenation result of the shared encoder output and the first interaction query to obtain the query and key, uses the output of the shared encoder to obtain the value, and after performing attention processing on the query, key and value, outputs the processing result to the cross-attention layer.

[0160] Since the interaction reflects the relationship between image patches, the number of first interaction queries is determined by the number of image patches M. The number N of first interaction queries is M×M. The multi-layer decoder of the interaction decoding network ultimately obtains N interaction representations, which constitute the second feature representation mentioned above. The first interaction queries are initialized before model training; they can be initialized randomly or according to preset content. The first interaction queries are gradually learned and updated during model training. Specifically, the M×M first interaction queries can be treated as a matrix, randomly initialized before model training, and gradually updated during training.

[0161] The self-attention layers in the other layers of the interactive decoding network use the concatenation result of the output of the previous layer decoder and the first interactive query to obtain the query and key, use the output of the previous layer decoder to obtain the value, and after performing attention processing on the query, key and value, output the processing result to the cross-attention layer.

[0162] The structure of the cross-attention layer can be as follows: Figure 5 As shown, the query is obtained by concatenating the output of the self-attention layer and the second conditional interaction query. The key and value are obtained by concatenating the output of the shared coding network (content key) and the spatial embedding feature. After attention processing is performed on the query, key, and value, the processing result is output to the FFN layer.

[0163] The second conditional interaction query is obtained from the interaction reference point and the output of the previous layer decoder. The interaction reference point is determined by the reference point of the first entity and the reference point of the second entity. The reference point of the first entity and the reference point of the second entity are predicted based on the first interaction query.

[0164] The specific methods for determining the query, key, and value in the cross-attention layer can be found in the relevant descriptions in formulas (5) to (8) in the embodiment of the entity interaction detection method.

[0165] The mapping layer uses the second feature representation of the image samples to perform mapping, thereby obtaining interaction type information and pointer representations of interactions of that type.

[0166] When the association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity, it can match the position information of the first entity and the second entity with the pointer representations that have an interaction relationship. If a matching pointer representation exists, it is determined that the first entity and the second entity have an interaction, and the interaction type is the type corresponding to the matched pointer representation.

[0167] As one possible approach, the training objective is to minimize the difference between the interaction information between the first and second entities in the model output and the corresponding labels of the image samples in the training data.

[0168] Furthermore, the type and location information of the first and second entities can be further labeled in the training data. During the training process, the training objective can further include minimizing the difference between the type and location information of the first and second entities output by the model and the corresponding labels of the image samples in the training data.

[0169] In the embodiments of this specification, a loss function can be constructed based on the above-mentioned training objective. In each iteration, the model parameters are updated using the value of the loss function and methods such as gradient descent, until a preset training termination condition is met. The training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0170] The method provided in this application embodiment can be applied to various application scenarios, one of which is listed here:

[0171] In the field of autonomous driving, the image data collected by autonomous vehicles is used for entity interaction detection using the method provided in this application embodiment. After content embedding and location embedding processing of the images, content embedding features and location embedding features are obtained, respectively. Then, the shared encoding network in the entity interaction detection model uses the content embedding features and location embedding features to extract the feature representation of the image. This feature representation is input into two parallel decoding networks, namely the entity decoding network and the interaction decoding network. The entity decoding network uses the feature representation of the image to predict the location information of people and objects in the image, and can further identify the specific object type. The interaction decoding network uses the feature representation of the image to predict the interaction type information and pointer representations of interaction relationships in the image. If there is an interaction relationship between a person and an object, there is at least one interaction information including the interaction type between the person and the object, the pointer representation of the person, and the pointer representation of the object. The association matching network matches the location information of the person and the object output by the entity decoding network with the interaction information output by the interaction decoding network, for example, by performing distance calculation. If the location information of the person and the object matches the pointer representation in one of the interaction information, it is determined that there is a relationship of the interaction type in the interaction information between the person and the object. The final output triple can include the information of the person, the information of the object, and the interaction type between the person and the object. Among them, the information of the person can include location information. The information about the object includes location information, and may further include type information.

[0172] The detection results can be provided to the decision-making devices of autonomous vehicles to assist in decision-making.

[0173] For example, if a person is detected opening a car door from image data collected by an autonomous vehicle, the detection result can be used by the decision-making device to predict that the motion state of the vehicle in front is likely to be static, and strategies such as avoidance can be generated.

[0174] For example, if a person is detected holding an umbrella from image data collected by an autonomous vehicle, the detection result can be used by the decision-making device to predict that the pedestrian's view may be obstructed, and strategies such as driving at low speed and maintaining a certain distance to avoid the obstacle can be generated.

[0175] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0176] According to another embodiment, an entity interaction detection device is provided. Figure 7 A schematic block diagram of an entity interaction detection device according to one embodiment is shown. Figure 7 As shown, the device 700 includes: an image acquisition unit 701, an embedding processing unit 702, and a cross-detection unit 703. The main functions of each component are as follows:

[0177] Image acquisition unit 701 is configured to acquire an image to be detected.

[0178] The embedding processing unit 702 is configured to perform embedding processing on the image to be detected to obtain the embedding features of the image to be detected.

[0179] The interaction detection unit 703 is configured to input the embedded features of the image to be detected into the entity interaction detection model to obtain the interaction information between the first entity and the second entity in the image to be detected.

[0180] The entity interaction detection model includes a shared encoding network, an entity decoding network, an interaction decoding network, and an association matching network. The shared encoding network encodes the embedded features of the image to be detected to obtain a feature representation of the image. The entity decoding network uses the feature representation of the image to predict the position information of the first and second entities contained in the image. The interaction decoding network uses the feature representation of the image to predict the interaction type information contained in the image and the pointer representation of the interaction relationship. The association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first and second entities.

[0181] As one possible implementation, the embedding processing unit 702 can be specifically configured to: perform content-based embedding and location-based embedding on image blocks in the image to be detected, respectively, to obtain the content embedding features and location embedding features of the image to be detected.

[0182] Accordingly, the shared coding network can be implemented based on a multi-layer Transformer, which includes a self-attention layer and a feedforward network (FFN) layer.

[0183] The self-attention layer uses the concatenation result of the content embedding feature and the position embedding feature to obtain the query vector and the key vector, uses the content embedding feature to obtain the value vector, and performs attention processing on the query, key and value, and outputs the processing result to the FFN layer.

[0184] As one possible implementation, the interactive decoding network includes a multi-layer decoder and a mapping layer. The multi-layer decoder performs decoding processing using the feature representation of the image to be detected, obtaining a second feature representation of the image. The mapping layer uses the second feature representation of the image to be detected for mapping, obtaining interaction information, which includes interaction type information and pointers indicating the existence of that type of interaction relationship.

[0185] As one possible implementation, each decoder in the multi-layer decoder includes a self-attention layer, a cross-attention layer, and an FFN layer.

[0186] The cross-attention layer obtains the query by concatenating the output of the self-attention layer and the second interactive query, obtains the key and value by concatenating the output of the shared coding network and the location embedding feature, and then performs attention processing on the query, key and value before outputting the processing result to the FFN layer.

[0187] The second interactive query is obtained from the interactive reference point and the output of the previous layer decoder. The interactive reference point is determined by the reference point of the first entity and the reference point of the second entity. The reference point of the first entity and the reference point of the second entity are predicted based on the first interactive query. The first interactive query is learned by the entity interaction model during the training process.

[0188] As one possible implementation, the self-attention layer in the first layer decoder of the interactive decoding network obtains the query and key by concatenating the output of the shared encoder and the first interactive query, obtains the value by using the output of the shared encoder, performs attention processing on the query, key and value, and outputs the processing result to the cross-attention layer.

[0189] As another possible approach, the self-attention layer in the other layers of the interactive decoding network obtains the query and key by concatenating the output of the previous layer decoder and the first interactive query, obtains the value by using the output of the previous layer decoder, performs attention processing on the query, key and value, and outputs the processing result to the cross-attention layer.

[0190] As one possible approach, the association matching network matches the location information of the first entity and the second entity with pointer representations that have an interaction relationship. If a matching pointer representation exists, it is determined that the first entity and the second entity have an interaction, and the interaction type is the type corresponding to the matched pointer representation.

[0191] The interactive detection model, interactive decoding network, and the structure of the decoder in the interactive decoding network can be described as follows: Figures 3-5 As shown, for details please refer to Figure 2 The relevant details in the corresponding method embodiments will not be repeated here.

[0192] Figure 8 A schematic block diagram of an apparatus for establishing an entity interaction detection model according to one embodiment is shown. Figure 7 As shown, the device 700 includes: a sample acquisition unit 801, an embedding processing unit 802, and a model training unit 803. The main functions of each component are as follows:

[0193] The sample acquisition unit 801 is configured to acquire training data, which includes image samples and labels that annotate the interaction information between a first entity and a second entity in the image samples.

[0194] The embedding processing unit 802 is configured to perform embedding processing on image samples in the training data to obtain the embedding features of the image samples.

[0195] The model training unit 803 is configured to use the embedded features of the image samples as input to the entity interaction detection model, and to train the entity interaction detection model. The training objective includes minimizing the difference between the interaction information between the first entity and the second entity output by the entity interaction detection model and the label of the image sample in the training data.

[0196] The entity interaction detection model includes a shared encoding network, an entity decoding network, an interaction decoding network, and an association matching network. The shared encoding network encodes the embedded features of the image sample to obtain the feature representation of the image sample. The entity decoding network uses the feature representation of the image sample to predict the location information of the first entity and the second entity contained in the image sample. The interaction decoding network uses the feature representation of the image sample to predict the interaction type information contained in the image sample and the pointer representation of the interaction relationship of that type. The association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity.

[0197] As one possible implementation, the embedding processing unit 802 can be specifically configured to: perform content-based embedding and location-based embedding on image blocks in the image sample respectively, to obtain the content embedding features and location embedding features of the image sample.

[0198] Accordingly, the shared coding network is implemented based on a multi-layer Transformer, which includes a self-attention layer and an FFN layer;

[0199] The self-attention layer obtains the query and key by concatenating the content embedding feature and the position embedding feature, obtains the value by using the content embedding feature, performs attention processing on the query, key and value, and outputs the processing result to the FFN layer.

[0200] As one possible implementation, the interactive decoding network includes a multi-layer decoder and a mapping layer. The multi-layer decoder performs decoding processing using the feature representation of the image sample to obtain a second feature representation of the image sample; the mapping layer uses the second feature representation of the image sample to perform mapping to obtain interaction information, which includes interaction type information and pointer representations indicating the existence of interaction relationships of that type.

[0201] One possible implementation is a decoder that includes a self-attention layer, a cross-attention layer, and an FFN layer.

[0202] The cross-attention layer obtains the query by concatenating the output of the self-attention layer and the second interactive query, and obtains the key and value by concatenating the output of the shared coding network and the location embedding feature. After attention processing using the query, key, and value, the processing result is output to the FFN layer.

[0203] The second interactive query is obtained from the interactive reference point and the output of the previous layer decoder. The interactive reference point is determined by the reference point of the first entity and the reference point of the second entity. The reference point of the first entity and the reference point of the second entity are predicted based on the first interactive query. The first interactive query is updated during the training process after initialization.

[0204] As one possible implementation, the self-attention layer in the first layer decoder of the interactive decoding network obtains the query and key by concatenating the output of the shared encoder and the first interactive query, obtains the value by using the output of the shared encoder, performs attention processing on the query, key and value, and outputs the processing result to the cross-attention layer.

[0205] As another possible approach, the self-attention layer in the other layers of the interactive decoding network obtains the query and key by concatenating the output of the previous layer decoder and the first interactive query, obtains the value by using the output of the previous layer decoder, performs attention processing on the query, key and value, and outputs the processing result to the cross-attention layer.

[0206] As one possible implementation, the association matching network matches the location information of the first entity and the second entity with pointer representations that have an interaction relationship. If a matching pointer representation exists, it is determined that the first entity and the second entity have an interaction, and the interaction type is the type corresponding to the matched pointer representation.

[0207] The interactive detection model, interactive decoding network, and the structure of the decoder in the interactive decoding network can be described as follows: Figures 3-5 As shown, for details please refer to Figure 6 The relevant details in the corresponding method embodiments will not be repeated here.

[0208] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0209] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0210] And an electronic device, comprising:

[0211] One or more processors; and

[0212] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0213] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0214] in, Figure 9 An exemplary architecture of an electronic device is shown, which may include a processor 910, a video display adapter 911, a disk drive 912, an input / output interface 913, a network interface 914, and a memory 920. The processor 910, video display adapter 911, disk drive 912, input / output interface 913, network interface 914, and memory 920 can communicate with each other via a communication bus 930.

[0215] The processor 910 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0216] The memory 920 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 920 can store the operating system 921 for controlling the operation of the electronic device 900, and the basic input / output system (BIOS) 922 for controlling the low-level operations of the electronic device 900. Additionally, it can store a web browser 923, a data storage management system 924, and a physical interaction detection device / device for establishing a physical interaction detection model 925, etc. The aforementioned physical interaction detection device / device for establishing a physical interaction detection model 925 can be the application program that specifically implements the aforementioned steps in the embodiments of this application. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910.

[0217] Input / output interface 913 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0218] Network interface 914 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0219] Bus 930 includes a pathway for transmitting information between various components of the device, such as processor 910, video display adapter 911, disk drive 912, input / output interface 913, network interface 914, and memory 920.

[0220] It should be noted that although the above-described device only shows the processor 910, video display adapter 911, disk drive 912, input / output interface 913, network interface 914, memory 920, bus 930, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0221] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0222] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0223] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting entity interactions, characterized in that, The method includes: The image to be detected is embedded to obtain its embedding features; The embedded features of the image to be detected are input into the entity interaction detection model to obtain the interaction information between the first entity and the second entity in the image to be detected. The entity interaction detection model includes a shared encoding network, an entity decoding network, an interaction decoding network, and an association matching network. The shared encoding network encodes the embedded features of the image to be detected to obtain a feature representation of the image. The entity decoding network uses the feature representation of the image to predict the location information of a first entity and a second entity contained in the image. The interaction decoding network uses the feature representation of the image to predict the interaction type information contained in the image and pointer representations indicating the existence of such interaction relationships. The association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity. The interactive decoding network includes a multi-layer decoder and a mapping layer; The multi-layer decoder uses the feature representation of the image to be detected and the first interactive query to perform decoding processing to obtain the second feature representation of the image to be detected. The first interactive query is learned by the entity interaction model during training. The mapping layer uses the second feature representation of the image to be detected to perform mapping, thereby obtaining interaction type information and pointer representations of interaction relationships of that type.

2. The method according to claim 1, characterized in that, The embedding process for the image to be detected to obtain the embedding features of the image to be detected includes: performing content-based embedding and position-based embedding on image blocks in the image to be detected to obtain the content embedding features and position embedding features of the image to be detected. The shared coding network is implemented based on a multi-layer Transformer, which includes a self-attention layer and a feedforward network (FFN) layer. The self-attention layer uses the concatenation result of the content embedding feature and the position embedding feature to obtain the query vector and the key vector, and uses the content embedding feature to obtain the value vector. After performing attention processing on the query, key and value, the processing result is output to the FFN layer.

3. The method according to claim 1, characterized in that, Each decoder in the multi-layer decoder includes a self-attention layer, a cross-attention layer, and an FFN layer; The cross-attention layer obtains the query by concatenating the output of the self-attention layer and the second interactive query, obtains the key and value by concatenating the output of the shared coding network and the location embedding feature, and after attention processing using the query, key and value, the processing result is output to the FFN layer. The second interactive query is obtained from the interactive reference point and the output of the previous layer decoder. The interactive reference point is determined by the reference point of the first entity and the reference point of the second entity. The reference point of the first entity and the reference point of the second entity are predicted based on the first interactive query.

4. The method according to any one of claims 1 to 3, characterized in that, The association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity, including: The association matching network matches the location information of the first entity and the second entity with pointer representations that have an interaction relationship. If a matching pointer representation exists, it is determined that the first entity and the second entity have an interaction, and the interaction type is the type corresponding to the matched pointer representation.

5. A method for establishing an entity interaction detection model, characterized in that, The method includes: Acquire training data, which includes image samples and labels that annotate the interaction information between a first entity and a second entity in the image samples; Embedding processing is performed on the image samples in the training data to obtain the embedding features of the image samples; The embedded features of the image samples are used as input to the entity interaction detection model to train the entity interaction detection model. The training objective includes minimizing the difference between the interaction information between the first entity and the second entity output by the entity interaction detection model and the label of the image sample in the training data. The entity interaction detection model includes a shared encoding network, an entity decoding network, an interaction decoding network, and an association matching network. The shared encoding network encodes the embedded features of the image samples to obtain a feature representation of the image samples. The entity decoding network uses the feature representation of the image samples to predict the location information of the first and second entities contained in the image samples. The interaction decoding network uses the feature representation of the image samples to predict the interaction type information contained in the image samples and pointer representations indicating the existence of such interaction relationships. The association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first and second entities. The interactive decoding network includes a multi-layer decoder and a mapping layer; The multi-layer decoder uses the feature representation of the image sample and the first interactive query to perform decoding processing to obtain the second feature representation of the image sample. The first interactive query is updated during the training process after initialization. The mapping layer uses the second feature representation of the image sample to perform mapping to obtain interaction information, which includes interaction type information and pointer representation of the existence of interaction relationship of that type.

6. The method according to claim 5, characterized in that, Embedding processing is performed on the image samples in the training data to obtain the embedding features of the image samples, including: performing content-based embedding and location-based embedding on the image blocks in the image samples respectively to obtain the content embedding features and location embedding features of the image samples. The shared coding network is implemented based on a multi-layer Transformer, which includes a self-attention layer and an FFN layer. The self-attention layer obtains the query and key by concatenating the content embedding feature and the position embedding feature, obtains the value by using the content embedding feature, performs attention processing on the query, key and value, and outputs the processing result to the FFN layer.

7. The method according to claim 5, characterized in that, The decoder includes a self-attention layer, a cross-attention layer, and an FFN layer; The cross-attention layer obtains the query by concatenating the output of the self-attention layer and the second interactive query, obtains the key and value by concatenating the output of the shared coding network and the location embedding feature, and after attention processing using the query, key and value, the processing result is output to the FFN layer. The second interactive query is obtained from the interactive reference point and the output of the previous layer decoder. The interactive reference point is determined by the reference point of the first entity and the reference point of the second entity. The reference point of the first entity and the reference point of the second entity are predicted based on the first interactive query.

8. The method according to any one of claims 5 to 7, characterized in that, The association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity, including: The association matching network matches the location information of the first entity and the second entity with pointer representations that have an interaction relationship. If a matching pointer representation exists, it is determined that the first entity and the second entity have an interaction, and the interaction type is the type corresponding to the matched pointer representation.

9. A physical interaction detection device, characterized in that, The device includes: The image acquisition unit is configured to acquire the image to be detected; An embedding processing unit is configured to perform embedding processing on the image to be detected to obtain the embedding features of the image to be detected; An interaction detection unit is configured to input the embedded features of the image to be detected into an entity interaction detection model to obtain the interaction information between the first entity and the second entity in the image to be detected. The entity interaction detection model includes a shared encoding network, an entity decoding network, an interaction decoding network, and an association matching network. The shared encoding network encodes the embedded features of the image to be detected to obtain a feature representation of the image. The entity decoding network uses the feature representation of the image to predict the location information of a first entity and a second entity contained in the image. The interaction decoding network uses the feature representation of the image to predict the interaction type information contained in the image and pointer representations indicating the existence of such interaction relationships. The association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first entity and the second entity. The interactive decoding network includes a multi-layer decoder and a mapping layer; The multi-layer decoder uses the feature representation of the image to be detected and the first interactive query to perform decoding processing to obtain the second feature representation of the image to be detected. The first interactive query is learned by the entity interaction model during training. The mapping layer uses the second feature representation of the image to be detected to perform mapping, thereby obtaining interaction type information and pointer representations of interaction relationships of that type.

10. An apparatus for establishing an entity interaction detection model, characterized in that, The device includes: The sample acquisition unit is configured to acquire training data, which includes image samples and labels that annotate the interaction information between a first entity and a second entity in the image samples. An embedding processing unit is configured to perform embedding processing on image samples in the training data to obtain embedding features of the image samples; The model training unit is configured to take the embedded features of the image samples as input to the entity interaction detection model and train the entity interaction detection model. The training objective includes minimizing the difference between the interaction information between the first entity and the second entity output by the entity interaction detection model and the label of the image sample in the training data. The entity interaction detection model includes a shared encoding network, an entity decoding network, an interaction decoding network, and an association matching network. The shared encoding network encodes the embedded features of the image samples to obtain a feature representation of the image samples. The entity decoding network uses the feature representation of the image samples to predict the location information of the first and second entities contained in the image samples. The interaction decoding network uses the feature representation of the image samples to predict the interaction type information contained in the image samples and pointer representations indicating the existence of such interaction relationships. The association matching network associates the prediction results of the entity decoding network and the interaction decoding network to obtain the interaction information between the first and second entities. The interactive decoding network includes a multi-layer decoder and a mapping layer; The multi-layer decoder uses the feature representation of the image sample and the first interactive query to perform decoding processing to obtain the second feature representation of the image sample. The first interactive query is updated during the training process after initialization. The mapping layer uses the second feature representation of the image sample to perform mapping to obtain interaction information, which includes interaction type information and pointer representation of the existence of interaction relationship of that type.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 8.

12. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 8.

Images