Interaction relationship detection method, model training method, device and storage medium

Abstract

The embodiment of the present application provides an interactive relationship detection method, a model training method, a device and a storage medium. The method includes the following steps: obtaining an image to be detected; performing interactive relationship detection on the image to be detected using a trained interactive relationship detection model; the interactive relationship detection model is used to: determine the visual features of multiple targets in the image to be detected based on the image to be detected; perform feature fusion on the visual features of the multiple targets to obtain the fused visual features of each of the multiple targets; perform interactive relationship detection on the first target and the second target according to the fused visual features of each of the multiple targets to obtain a first interactive relationship detection result. The interactive relationship detection solution provided by the embodiment of the present application has a high detection accuracy.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an interaction relationship detection method, a model training method, a device, and a storage medium. Background Art

[0002] In recent years, significant progress has been made in object detection and classification technologies. However, the scene information contained in an image cannot be determined using basic object detection and classification methods. This requires deeper semantic recognition, such as identifying illegal dog walking and street vendors. Understanding the scene information in an image requires understanding how the various objects in the image interact. Interaction relationship detection is a highly semantic, fine-grained understanding task of understanding the interactions between multiple objects in an image. Its goal is to identify the triples of <object 1, relationship, object 2> that represent interactive relationships in the image. Objects can be objects (such as items, animals, plants, etc.) or people. Human-object interaction (HOI) detection is an important branch of interaction relationship detection, where the objects are people and objects.

[0003] Currently, existing interaction relationship detection methods have the problem of low detection accuracy. Summary of the Invention

[0004] In view of the above problems, the present application is proposed to provide an interaction relationship detection method, model training method, device and storage medium that solve the above problems or at least partially solve the above problems.

[0005] Therefore, in one embodiment of the present application, a method for detecting an interactive relationship is provided. The method includes:

[0006] Obtain the image to be detected;

[0007] Performing interaction relationship detection on the image to be detected using a trained interaction relationship detection model;

[0008] Among them, the interaction relationship detection model is used to: determine the visual features of multiple targets in the image to be detected based on the image to be detected; perform feature fusion on the visual features of the multiple targets to obtain the fused visual features of each of the multiple targets; perform interaction relationship detection on the first target and the second target among the multiple targets based on the fused visual features of each of the first target and the second target to obtain a first interaction relationship detection result.

[0009] In another embodiment of the present application, a model training method is provided, which includes:

[0010] Get sample images with interaction relationship labels;

[0011] Performing interaction relationship detection on the sample image using an interaction relationship detection model to obtain a target sample interaction relationship detection result;

[0012] Optimizing parameters of the interaction relationship detection model according to the target sample interaction relationship detection result and the interaction relationship label;

[0013] Among them, the interaction relationship detection model is used to: determine the sample visual features of multiple sample targets in the sample image based on the sample image; perform feature fusion on the sample visual features of the multiple sample targets to obtain the fused sample visual features of each of the multiple sample targets; perform interaction relationship detection on the first sample target and the second sample target in the multiple sample targets based on their respective fused sample visual features to obtain a first sample interaction relationship detection result; the target sample interaction relationship detection result is determined based on the first sample interaction relationship detection result.

[0014] In another embodiment of the present application, a method for detecting an interaction relationship is provided, which includes:

[0015] Acquire a traffic image captured by a traffic camera device; the traffic image includes a plurality of targets; the plurality of targets include a first person and a first vehicle;

[0016] Performing interaction relationship detection on the traffic image using a trained interaction relationship detection model;

[0017] Among them, the interaction relationship detection model is used to: determine the visual features of multiple targets in the traffic image based on the traffic image; perform feature fusion on the visual features of the multiple targets to obtain the fused visual features of each of the multiple targets; perform interaction relationship detection on the first person and the first vehicle based on the fused visual features of each of the first person and the first vehicle to obtain a first interaction relationship detection result.

[0018] In another embodiment of the present application, an electronic device is provided, comprising: a memory and a processor, wherein:

[0019] The memory is used to store programs;

[0020] The processor is coupled to the memory and is used to execute the program stored in the memory to implement any of the above-mentioned interaction relationship detection methods or the above-mentioned model training methods.

[0021] In another embodiment of the present application, a computer-readable storage medium storing a computer program is provided, wherein the computer program, when executed by a computer, can implement any of the above-mentioned interaction relationship detection methods or the above-mentioned model training methods.

[0022] In the technical solution provided by the embodiments of this application, the interaction relationship detection model performs feature fusion on the visual features of multiple targets to obtain the fused visual features of each target. In this way, the fused visual features of each target not only contain its own visual features, but also incorporate the visual features of other targets in the image, making the fused visual features of each target more expressive of the target, thereby realizing the recognition of the interaction relationship between two targets from a global perspective and improving the accuracy of interaction relationship detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1a A flowchart of an interactive relationship detection method provided in one embodiment of the present application;

[0025] Figure 1b A flowchart of an interaction relationship detection method provided in yet another embodiment of the present application;

[0026] Figure 2a A flowchart of a model training method provided in one embodiment of the present application;

[0027] Figure 2b A flowchart of a model training method provided in another embodiment of the present application;

[0028] Figure 3 An example diagram of a method provided in one embodiment of the present application;

[0029] Figure 4 A flowchart of an interactive relationship detection method provided in another embodiment of the present application;

[0030] Figure 5 This is a structural block diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] Currently, the interactive relationship detection solution is to perform target detection on the image, pair the targets according to the target detection results, and then perform interactive relationship detection on the two paired targets.

[0032] Through research, the inventors found that the existing interactive relationship detection scheme does not identify the relationship between two targets from a global perspective, that is, when identifying the interactive relationship between two targets in an image, it does not refer to the features of other targets in the image. In order to improve the accuracy of the interactive relationship detection results, the embodiment of the present application proposes to identify the relationship between two targets from a global perspective. Specifically, the interactive relationship detection model will perform feature fusion on the visual features of multiple targets to obtain the fused visual features of the multiple targets. In this way, the fused visual features of each target not only contain its own visual features, but also integrate the visual features of other targets in the image, so that the fused visual features of each target have better expression capabilities for the target, thereby realizing the identification of the interactive relationship between the two targets from a global perspective, which can improve the accuracy of interactive relationship detection.

[0033] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below based on the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0034] In addition, some of the processes described in the specification, claims and the above-mentioned figures of this application include multiple operations that appear in a specific order. These operations may not be executed in the order in which they appear in this document or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit "first" and "second" to be different types.

[0035] Figure 1a FIG. 1 shows a flow chart of a data processing method provided by an embodiment of the present application. Figure 1a As shown, the method includes:

[0036] 101. Obtain an image to be detected.

[0037] 102. Perform interaction relationship detection on the image to be detected using the trained interaction relationship detection model.

[0038] In the above 101, the image to be detected may be an image in any application field, for example, an image taken by a road camera or an image taken by a public security camera.

[0039] In the above 102, the above interaction relationship detection model can be specifically an interaction relationship detection model based on a neural network. The interaction relationship detection model can be trained by sample images with interaction relationship labels. The specific training process will be described in detail in the following embodiments.

[0040] like Figure 1b As shown, the interaction relationship detection model is used to perform the following steps:

[0041] S11. Determine visual features of multiple targets in the image to be detected based on the image to be detected.

[0042] S12. Perform feature fusion on the visual features of the multiple targets to obtain fused visual features of the multiple targets.

[0043] S13. Perform interaction relationship detection on a first target and a second target among the multiple targets based on their respective fused visual features to obtain a first interaction relationship detection result.

[0044] The plurality of targets include a first target and a second target. In one example, the plurality of targets may further include other targets in addition to the first target and the second target.

[0045] In the above S11, target detection can be performed on the image to be detected first to obtain a target detection result. In one example, the target detection result may include location information and target categories of multiple targets. The location information may specifically be bounding box information. Based on the image to be detected and the target detection result, the visual features of the multiple targets in the image to be detected are determined. The above target detection steps can be performed by the interactive relationship detection model, for example: the interactive relationship detection model includes a target detection network for performing the above target detection steps; or, the above target detection steps can be performed by the target detection model, and after the target detection model performs the above target detection steps, the target detection results are input into the above interactive relationship detection model.

[0046] In one feasible solution, the step of “determining visual features of multiple targets in the image to be detected based on the image to be detected” in S11 may specifically include the following steps:

[0047] S111a: Feature extraction can be performed on the image to be detected to obtain a visual feature map.

[0048] S112a. Determine, in the visual feature map, the visual features of each of the multiple targets based on their respective position information in the image to be detected.

[0049] In S111a above, the interaction relationship detection model may include a visual feature extraction network that extracts features from the image to be detected to obtain a visual feature map. Specifically, the visual feature extraction network may be a convolutional neural network. The image to be detected is input to the visual feature extraction network, and the visual feature extraction network outputs a visual feature map. Typically, the size of the visual feature map is smaller than the size of the image to be detected.

[0050] In the above S112a, the multiple targets include a third target, and the third target refers to any one of the multiple targets, for example, it can be the above first target or the above second target.

[0051] In a specific example, according to the position information of each of the multiple targets in the image to be detected, a region of interest (ROI) pooling operation is used to determine the visual features of each of the multiple targets in the visual feature map.

[0052] In another specific example, the above-mentioned position information includes bounding box information. Based on the bounding box information of the third target in the image to be detected and the size ratio of the image to be detected and the visual feature map, a related area corresponding to the bounding box area of ​​the third target is determined in the visual feature map; and the features in the related area in the visual feature map are used as the visual features of the third target. The bounding box area of ​​the third target refers to the area surrounded by the bounding box of the third target in the image to be detected. The size ratio of the bounding box area and the related area is determined by the size ratio of the image to be detected and the visual feature map, and the two are positively correlated. In one example, the two are equal.

[0053] In another feasible solution, the above-mentioned position information includes bounding box information. The above-mentioned S11 "determining visual features of multiple targets in the image to be detected based on the image to be detected" may specifically include the following steps:

[0054] S111b: Cut out image regions of the multiple targets from the image to be detected according to the bounding box information of the multiple targets.

[0055] The image area of ​​each target is the area surrounded by the bounding box of the target in the image to be detected.

[0056] S112b, performing feature extraction on the image regions of the multiple targets respectively to obtain visual features of the multiple targets.

[0057] The above-mentioned interaction relationship detection model may include a visual feature extraction network, the input of which is the image region of each of the multiple cut-out targets, and the output is the visual feature of each target.

[0058] The above two methods for determining visual features can be selected according to actual needs, and the embodiments of the present application do not specifically limit this.

[0059] In the above S12, the fused visual features of each target include not only its own visual features, but also the visual features of other targets in the multiple targets. In the embodiment of the present application, the fused visual features of different targets are different.

[0060] For a third target, the visual features of multiple targets are fused to obtain fused visual features of the third target. This may specifically include: correcting the visual features of the third target based on the visual features of targets other than the third target in the multiple targets to obtain fused visual features of the third target. In other words, the fusion process is also a correction process. Specifically, based on the visual features of targets other than the third target in the multiple targets, a region of interest in the visual features of the third target is determined; and the region of interest in the visual features of the third target is enhanced to obtain fused visual features of the third target. There may be multiple regions of interest.

[0061] In a feasible solution, the interactive relationship detection model includes a first self-attention network. The first self-attention network is used to perform feature fusion on the visual features of the multiple targets to obtain the fused visual features of each of the multiple targets. The visual features of the multiple targets can be input into the first self-attention network, and the first self-attention network uses the self-attention mechanism to perform feature fusion on the input visual features of the multiple targets, and outputs the fused visual features of each of the multiple targets. The first self-attention network has a good feature fusion capability, and the fused visual features of each target it outputs have a better expression capability for the target. The first self-attention network is used to perform feature fusion on the visual features of the multiple targets to obtain the fused visual features of each of the multiple targets, which can also be understood as: based on the visual features of other targets among the multiple targets except the third target, the visual features of the third target are corrected to obtain the fused visual features of the third target.

[0062] In the above S13, in one example, the first target and the second target can be any two targets from the plurality of targets. Considering that practical applications primarily involve detecting interactions between people, in another example, one of the first target and the second target can be a person from the plurality of targets, and the other can be an object from the plurality of targets.

[0063] In one feasible solution, the fused visual features of the first target and the fused visual features of the second target can be spliced ​​together to obtain a spliced ​​visual feature. Based on the spliced ​​visual feature, an interaction relationship detection is performed on the first target and the second target to obtain a first interaction relationship detection result. The first interaction relationship detection result includes a first probability that the first target and the second target belong to each of multiple alternative interaction relationship categories. The spliced ​​visual feature can be input into a first classifier in an interaction relationship detection model, so that the first classifier performs interaction relationship detection to obtain a first interaction relationship detection result.

[0064] In the technical solution provided by the embodiments of this application, the interaction relationship detection model performs feature fusion on the visual features of multiple targets to obtain the fused visual features of each target. In this way, the fused visual features of each target not only contain its own visual features, but also incorporate the visual features of other targets in the image, making the fused visual features of each target more expressive of the target, thereby realizing the recognition of the interaction relationship between two targets from a global perspective and improving the accuracy of interaction relationship detection.

[0065] Optionally, the interaction relationship detection model is further used to:

[0066] S14. Determine spatial features of the first target and the second target based on position information of the first target and the second target in the image to be detected.

[0067] S15. Determine a combined visual feature of the first target and the second target based on respective visual features of the first target and the second target.

[0068] S16. Perform interaction relationship detection on the first target and the second target based on the combined visual feature and the spatial feature to obtain a second interaction relationship detection result.

[0069] S17: Determine a target interaction relationship detection result based on the first interaction relationship detection result and the second interaction relationship detection result.

[0070] In the above S14, the position information may include: bounding box information.

[0071] In one feasible solution, the step of “determining the spatial features of the first target and the second target based on the position information of the first target and the second target in the image to be detected” in S14 can be implemented by the following steps:

[0072] S141 . Generate a binary image corresponding to the first target according to the bounding box information of the first target in the image to be detected.

[0073] The size of the binary image corresponding to the first target is consistent with the size of the image to be detected, and the pixel value within the boundary box of the first target in the binary image corresponding to the first target is 1, and the rest is 0.

[0074] S142: Generate a binary image corresponding to the second target according to the bounding box information of the second target in the image to be detected.

[0075] The size of the binary image corresponding to the second target is consistent with the size of the image to be detected, and the pixel value within the boundary box of the second target in the binary image corresponding to the second target is 1, and the rest is 0.

[0076] S143, obtaining a two-channel binary image according to the binary images corresponding to the first target and the second target;

[0077] S144. Utilize the spatial feature extraction network in the interaction relationship detection model to perform feature extraction on the two-channel binary image to obtain spatial features.

[0078] The spatial feature extraction network may be a convolutional neural network. The spatial features may be in vector form.

[0079] In S15 above, in one example, the visual features may be in vector form. The visual features of the first and second objects may be concatenated to obtain a combined visual feature of the first and second objects. For example, if the visual features are 256-dimensional vectors, then the combined visual features are 512 (256*2)-dimensional vectors.

[0080] In order to further improve the accuracy of interaction relationship detection, in another example, the interaction relationship detection model is further used to:

[0081] S18: Determine, in the image to be detected, a bounding box surrounding the first object and a bounding box surrounding the second object.

[0082] S19: Determine visual features of the circumscribed frame area in the visual feature map according to the position information of the circumscribed frame in the image to be detected.

[0083] Accordingly, the step of “determining the combined visual features of the first target and the second target based on the respective visual features of the first target and the second target” in S15 specifically includes:

[0084] S151: Combine the visual features of the first target and the second target and the visual features of the circumscribed frame area to obtain a combined visual feature of the first target and the second target.

[0085] In S18 above, the bounding box includes not only the first and second objects, but also background information near the first and second objects. This background information is very important auxiliary information for identifying the interactive relationship between the first and second objects. In this embodiment, the bounding box is used to limit the background information near the first and second objects, which helps to eliminate background information that may cause interference and retain useful background information.

[0086] In the above S19, the visual features of the outer frame area can be determined in the visual feature map based on the position information of the outer frame in the image to be detected and the size ratio between the image to be detected and the visual feature map. The visual features of the outer frame area refer to the features in the relevant area corresponding to the outer frame area in the visual feature map. The size ratio of the outer frame area to the relevant area is determined by the size ratio between the image to be detected and the visual feature map, and the two are positively correlated. In one embodiment, the two are equal.

[0087] In S151, the visual features of the first and second objects are combined with the visual features of the bounding box area to obtain the combined visual features of the first and second objects. This combined visual feature not only includes the visual features of the first and second objects, but also includes the background features near the first and second objects, which helps improve the accuracy of interaction relationship detection.

[0088] In one feasible solution, in S16 above, the combined visual features and the spatial features may be concatenated, and based on the concatenated features, interaction relationship detection may be performed on the first object and the second object to obtain a second interaction relationship detection result. The concatenated features may be input into a second classifier in the interaction relationship detection model, so that the second classifier performs interaction relationship detection on the first object and the second object and outputs a second interaction relationship detection result.

[0089] To improve the accuracy of interaction relationship recognition, in another feasible solution, the above-mentioned step S16 of "performing interaction relationship detection on the first object and the second object based on the combined visual feature and the spatial feature to obtain a second interaction relationship detection result" can be specifically implemented by the following steps:

[0090] S161. Perform feature fusion on the combined visual features and the spatial features to obtain fused combined visual features and fused spatial features.

[0091] S162. Perform interaction relationship detection on the first target and the second target based on the fused combined visual features and the fused spatial features to obtain a second interaction relationship detection result.

[0092] In the above S161, the spatial features are corrected according to the combined visual features to obtain the fused spatial features. According to the spatial features, the combined visual features are corrected to obtain the fused combined visual features. The specific process can be described as follows: according to the combined visual features, the regions of interest in the spatial features are determined, the regions of interest in the spatial features are enhanced, and the fused spatial features are obtained; according to the spatial features, the regions of interest in the combined visual features are determined, the regions of interest in the combined visual features are enhanced, and the fused combined visual features are obtained. Among them, the regions of interest refer to the regions that play an important role in the identification of interactive relationships.

[0093] In a specific example, the second self-attention network in the interaction relationship detection model is used to perform feature fusion on the combined visual features and the spatial features to obtain fused combined visual features and fused spatial features. The second self-attention network can perform feature fusion on the combined visual features and the spatial features to obtain fused combined visual features and fused spatial features. The input of the second self-attention network is the combined visual features and the spatial features; the second self-attention network can use the self-attention mechanism to perform feature fusion on the combined visual features and the spatial features, and its output is fused combined visual features and fused spatial features. The working principle of the second self-attention network can also be understood as: according to the combined visual features, the spatial features are modified to obtain fused spatial features; according to the spatial features, the combined visual features are modified to obtain fused combined visual features.

[0094] Note: Both the first self-attention network and the second self-attention network are Transformer structures.

[0095] In the above S162, in one example, the fused combined visual features and the fused spatial features may be spliced ​​together to obtain spliced ​​features, and based on the spliced ​​features, the interaction relationship between the first target and the second target may be detected to obtain a second interaction relationship detection result.

[0096] The spliced ​​features can be input into a second classifier in the interaction relationship detection model, so that the second classifier performs interaction relationship detection on the first target and the second target, and outputs a second interaction relationship detection result.

[0097] In the above S17, the first interaction relationship detection result includes the first probability that the first target and the second target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship categories. The second interaction relationship detection result includes the second probability that the first target and the second target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship types. Among them, the target interaction relationship detection result includes the target probability that the first target and the second target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship categories. Specifically, the product of the first probability and the second probability that the first target and the second target belong to the first alternative interaction relationship category can be used as the target probability that the first target and the second target belong to the first alternative interaction relationship category. The first alternative interaction relationship category is any one of a plurality of alternative interaction relationship types.

[0098] In this embodiment, the final detection result is determined by combining the detection results of two branches: one branch detects the interaction relationship from a global perspective, and the other branch detects the interaction relationship from a perspective combining spatial features and visual features.

[0099] In practical applications, a branch can be added: interaction relationship detection is performed based on spatial features only. Specifically, the interaction relationship detection model is also used to:

[0100] S20: Perform interaction relationship detection on the first target and the second target based on the spatial feature to obtain a third interaction relationship detection result.

[0101] The spatial features may be input into a third classifier in the interaction relationship detection model, so that the third classifier performs interaction relationship detection to obtain a third interaction relationship detection result.

[0102] Correspondingly, the above S17 "determine the target interaction relationship detection result based on the first interaction relationship detection result and the second interaction relationship detection result" may specifically include: determining the target interaction relationship detection result based on the first interaction relationship detection result, the second interaction relationship detection result and the third interaction relationship detection result.

[0103] The first interaction relationship detection result includes a first probability that the first target and the second target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship categories. The second interaction relationship detection result includes a second probability that the first target and the second target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship types. The third interaction relationship detection result includes a third probability that the first target and the second target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship types. Among them, the target interaction relationship detection result includes the target probability that the first target and the second target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship categories. Specifically, the product of the first probability, the second probability and the third probability that the first target and the second target belong to the first alternative interaction relationship category can be used as the target probability that the first target and the second target belong to the first alternative interaction relationship category. The first alternative interaction relationship category is any one of the plurality of alternative interaction relationship types.

[0104] In the embodiment of the present application, integrating the interaction relationship detection results of multiple branches helps to improve the accuracy of interaction relationship detection.

[0105] The following describes a training method for the aforementioned interaction relationship detection model. As shown in Figure 2, the method includes:

[0106] 201. Obtain sample images with interaction relationship labels.

[0107] 202. Perform interaction relationship detection on the sample image using an interaction relationship detection model to obtain a target sample interaction relationship detection result.

[0108] 203. Optimize parameters of the interaction relationship detection model according to the target sample interaction relationship detection result and the interaction relationship label.

[0109] In the above 201, in practical applications, the V-COCO dataset can be used to train the interaction relationship detection model. The V-COCO dataset contains 10,346 sample images. 2,533 of these images can be used for model training, 2,867 for model validation, and 4,946 for model testing. In the sample images, the two objects that have an interaction relationship are annotated with an interaction relationship label.

[0110] In the above 202, the sample image is input into the interaction relationship detection model, so that the interaction relationship detection model outputs the target sample interaction relationship detection result.

[0111] In the above 203, parameters of the interaction relationship detection model are optimized according to the target sample interaction relationship detection result and the interaction relationship label.

[0112] In one feasible solution, a loss function can be used to calculate a loss value based on the target sample interaction detection results and interaction relationship labels. This loss value is then used to optimize the parameters of the interaction detection demon system. The specific optimization process can be referenced in existing technologies and will not be detailed here.

[0113] The interaction relationship detection model is used to:

[0114] S31 . Determine sample visual features of a plurality of sample objects in the sample image according to the sample image.

[0115] S32: Perform feature fusion on the sample visual features of the multiple sample targets to obtain fused sample visual features of the multiple sample targets.

[0116] S33. Perform interaction relationship detection on a first sample target and a second sample target among the multiple sample targets based on their respective fused sample visual features to obtain a first sample interaction relationship detection result.

[0117] The target sample interaction relationship detection result is determined based on the first sample interaction relationship detection result.

[0118] The plurality of sample targets include a first sample target and a second sample target. In one example, the plurality of sample targets may further include other sample targets in addition to the first sample target and the second sample target.

[0119] In S31 above, in one example, the interaction relationship detection model may first perform object detection on the sample image to obtain a sample object detection result. In one example, the sample object detection result may include location information and object categories of multiple sample objects. The location information may specifically be bounding box information. Based on the sample image and the sample object detection result, sample visual features of the multiple sample objects in the sample image are determined.

[0120] In another example, the sample image is annotated with the location label and target category label of each target. The location label can specifically be a bounding box. In this way, the interaction relationship detection model does not need to perform the target detection step. The location information and target category of multiple target samples can be determined based on the location labels and target category labels of the target samples in the sample image.

[0121] In one feasible solution, the step of “determining, according to the sample image, sample visual features of a plurality of sample objects in the sample image” in S31 may specifically include the following steps:

[0122] S311a: Feature extraction can be performed on the sample image to obtain a sample visual feature map.

[0123] S312a: Determine, in the sample visual feature map, sample visual features of each of the multiple sample targets based on their respective position information in the sample image.

[0124] In S311a above, the interaction relationship detection model may include a visual feature extraction network, which extracts features from the sample image to obtain a sample visual feature map. Specifically, the visual feature extraction network may be a convolutional neural network. The sample image is input to the visual feature extraction network, and the sample visual feature map is output from the visual feature extraction network. Typically, the size of the sample visual feature map is smaller than the size of the sample image.

[0125] In the above S312a, the multiple sample targets include a third sample target. The third sample target refers to any sample target among the multiple sample targets, for example, it can be the above first sample target or the above second sample target.

[0126] In a specific example, according to the position information of each of the multiple targets in the sample image, a region of interest (ROI) pooling operation is used to determine the sample visual features of each of the multiple sample targets in the sample visual feature map.

[0127] In another specific example, the above-mentioned position information includes bounding box information. Based on the bounding box information of the third sample target in the sample image and the size ratio of the sample image and the sample visual feature map, a related area corresponding to the bounding box area of ​​the third sample target is determined in the sample visual feature map; the features in the related area in the sample visual feature map are used as the visual features of the third sample target. The bounding box area of ​​the third sample target refers to the area surrounded by the bounding box of the third sample target in the sample image. The size ratio of the bounding box area and the related area is determined by the size ratio of the sample image and the sample visual feature map, and the two are positively correlated. In one example, the two are equal.

[0128] In another feasible solution, the above-mentioned position information includes bounding box information. The above-mentioned S31 "determining sample visual features of multiple sample objects in the sample image according to the sample image" may specifically include the following steps:

[0129] S311b: Cut out image regions of the multiple sample objects from the sample image according to the respective bounding box information of the multiple sample objects.

[0130] The image area of ​​each sample object is the area surrounded by the bounding box of the object in the sample image.

[0131] S312b: Perform feature extraction on the image regions of the multiple sample targets respectively to obtain sample visual features of the multiple sample targets.

[0132] The above-mentioned interaction relationship detection model may include a visual feature extraction network, the input of which is the image region of each of the multiple sample targets cut out, and the output is the sample visual feature of each sample target.

[0133] The above two methods for determining the visual features of the samples can be selected according to actual needs, and the embodiments of the present application do not impose specific limitations on this.

[0134] In the above S32, the fused sample visual features of each sample target not only include its own visual features, but also incorporate the visual features of other sample targets in the multiple targets. Obviously, in the embodiment of the present application, the fused sample visual features of different sample targets are different.

[0135] For the third sample target, the sample visual features of multiple sample targets are subjected to feature fusion to obtain the fused sample visual features of the third sample target. Specifically, the process may include: correcting the sample visual features of the third sample target based on the visual features of other sample targets among the multiple sample targets except the third sample target, to obtain the fused sample visual features of the third sample target. In other words, the fusion process is also a correction process. Based on the sample visual features of other sample targets among the multiple sample targets except the third sample target, a region of interest in the sample visual features of the third sample target is determined; and the region of interest in the sample visual features of the third sample target is enhanced to obtain the fused sample visual features of the third sample target. There may be multiple regions of interest.

[0136] In one feasible solution, the interaction relationship detection model includes a first self-attention network. The first self-attention network is used to perform feature fusion on the sample visual features of the multiple sample targets using the self-attention mechanism to obtain the fused sample visual features of each of the multiple sample targets. The sample visual features of the multiple sample targets can be input into the first self-attention network, and the first self-attention network performs feature fusion on the input visual features of the multiple sample targets internally, and outputs the fused sample visual features of each of the multiple sample targets. The first self-attention network has excellent feature fusion capabilities, and the fused sample visual features of each sample target outputted by it have better expression capabilities for the sample target. The first self-attention network is used to perform feature fusion on the sample visual features of the multiple sample targets using the self-attention mechanism to obtain the fused sample visual features of each of the multiple sample targets. This can also be understood as: based on the visual features of other sample targets among the multiple sample targets except the third sample target, the visual features of the third sample target are corrected to obtain the fused sample visual features of the third sample target.

[0137] In the aforementioned S33, in one example, the first sample target and the second sample target can be any two sample targets from the plurality of sample targets. Considering that practical applications primarily involve detecting interactions between people, in another example, one of the first sample target and the second sample target can be a human target from the plurality of sample targets, and the other can be an object target from the plurality of sample targets.

[0138] In one feasible solution, the fused sample visual features of the first sample target and the fused sample visual features of the second sample target can be spliced ​​together to obtain a spliced ​​sample visual feature. Based on the spliced ​​sample visual feature, an interaction relationship detection is performed on the first sample target and the second sample target to obtain a first interaction relationship detection result. The first interaction relationship detection result includes a first probability that the first sample target and the second sample target belong to each of a plurality of candidate interaction relationship categories.

[0139] In the technical solution provided by the embodiments of this application, the interaction relationship detection model performs feature fusion on the visual features of multiple targets to obtain the fused visual features of each target. In this way, the fused visual features of each target not only contain its own visual features, but also incorporate the visual features of other targets in the image, making the fused visual features of each target more expressive of the target, thereby realizing the recognition of the interaction relationship between two targets from a global perspective and improving the accuracy of interaction relationship detection.

[0140] Optionally, the interaction relationship detection model is further used to:

[0141] S34: Determine sample space features of the first sample object and the second sample object according to position information of the first sample object and the second sample object in the sample image.

[0142] S35: Determine a combined sample visual feature of the first sample target and the second sample target based on the respective sample visual features of the first sample target and the second sample target.

[0143] S36. Perform interaction relationship detection on the first sample target and the second sample target based on the combined sample visual feature and the sample spatial feature to obtain a second sample interaction relationship detection result.

[0144] S37: Determine a target interaction relationship detection result based on the first interaction relationship detection result and the second interaction relationship detection result.

[0145] In the above S34, the position information may include: bounding box information.

[0146] In one feasible solution, the step of “determining the sample spatial features of the first sample target and the second sample target according to the position information of the first sample target and the second sample target in the sample image” in S34 can be implemented by the following steps:

[0147] S341 : Generate a binary image corresponding to the first sample object according to the bounding box information of the first sample object in the sample image.

[0148] The size of the binary image corresponding to the first sample target is consistent with the size of the sample image, and the pixel value within the boundary box of the first sample target in the binary image corresponding to the first sample target is 1, and the rest is 0.

[0149] S342: Generate a binary image corresponding to the second sample target according to the bounding box information of the second sample target in the sample.

[0150] The size of the binary image corresponding to the second sample target is consistent with the size of the sample image, and the pixel value within the boundary box of the second sample target in the binary image corresponding to the second sample target is 1, and the rest is 0.

[0151] S343, obtaining a two-channel binary image according to the binary images corresponding to the first sample target and the second sample target;

[0152] S344. Utilize the spatial feature extraction network in the interaction relationship detection model to perform feature extraction on the two-channel binary image to obtain sample spatial features.

[0153] The spatial feature extraction network may be a convolutional neural network. The sample spatial features may be in vector form.

[0154] In S35 above, in one example, the sample visual features can be in vector form. The sample visual features of the first sample object and the second sample object can be concatenated to obtain a combined sample visual feature of the first sample object and the second sample object. For example, if the sample visual features are 256-dimensional vectors, then the combined sample visual features are 512 (256*2)-dimensional vectors.

[0155] In order to further improve the accuracy of interaction relationship detection, in another example, the interaction relationship detection model is further used to:

[0156] S38: Determine, in the sample image, a bounding box surrounding the first sample object and a bounding box surrounding the second sample object.

[0157] S39. Determine, in the sample visual feature map, sample visual features of the circumscribed frame area according to the position information of the circumscribed frame in the sample image.

[0158] Accordingly, the step of “determining the combined sample visual features of the first sample target and the second sample target based on the respective sample visual features of the first sample target and the second sample target” in S35 specifically includes:

[0159] S351 , combining the sample visual features of the first sample target and the second sample target and the sample visual features of the circumscribed frame area to obtain a combined sample visual feature of the first sample target and the second sample target.

[0160] In the above S38 , the external frame includes not only the first sample target and the second sample target, but also background information around the first sample target and the second sample target.

[0161] In the above S39, the sample visual features of the outer frame area can be determined in the sample visual feature map based on the position information of the outer frame in the sample image and the size ratio between the sample image and the sample visual feature map. The sample visual features of the outer frame area refer to the features in the relevant area corresponding to the outer frame area in the sample visual feature map. The size ratio of the outer frame area to the relevant area is determined by the size ratio between the sample image and the sample visual feature map, and the two are positively correlated. In one embodiment, the two are equal.

[0162] In S351 above, the sample visual features of the first and second sample targets are combined with the sample visual features of the bounding box area to obtain the combined sample visual features of the first and second sample targets. This combined sample visual features not only include the sample visual features of the first and second sample targets, but also the background features near the first and second sample targets, helping to improve the accuracy of interaction relationship detection.

[0163] In a feasible solution, in the above S36, the combined sample visual features and the sample spatial features may be spliced ​​together, and based on the spliced ​​sample features, the interaction relationship detection may be performed on the first sample target and the second sample target to obtain a second sample interaction relationship detection result.

[0164] To improve the accuracy of interaction relationship recognition, in another feasible solution, the above step S3 of "performing interaction relationship detection on the first sample target and the second sample target based on the combined sample visual features and the sample spatial features to obtain a second interaction relationship detection result" can be implemented by the following steps:

[0165] S361 , performing feature fusion on the combined sample visual features and the sample spatial features to obtain fused combined sample visual features and fused sample spatial features.

[0166] S362: Perform interaction relationship detection on the first sample target and the second sample target based on the fused combined sample visual features and the fused sample spatial features to obtain a second sample interaction relationship detection result.

[0167] In the above S361, the sample space features are corrected according to the combined sample visual features to obtain the fused sample space features. According to the sample space features, the combined sample visual features are corrected to obtain the fused combined sample visual features. The specific process can be described as: according to the combined sample visual features, the region of interest in the sample space features is determined, the region of interest in the sample space features is enhanced, and the fused sample space features are obtained; according to the sample space features, the region of interest in the combined sample visual features is determined, the region of interest in the combined sample visual features is enhanced, and the fused combined sample visual features are obtained. Among them, the region of interest refers to the area that plays an important role in the identification of interactive relationships.

[0168] In a specific example, the second self-attention network in the interaction relationship detection model is used to perform feature fusion on the combined sample visual features and the sample space features to obtain fused combined sample visual features and fused sample space features. The second self-attention network can perform feature fusion on the combined sample visual features and the sample space features to obtain fused combined sample visual features and fused sample space features. The input of the second self-attention network is the combined sample visual features and the sample space features; the second self-attention network uses the self-attention mechanism to perform feature fusion on the combined sample visual features and the sample space features, and its output is fused combined sample visual features and fused sample space features. The working principle of the second self-attention network can also be understood as: according to the combined sample visual features, the sample space features are modified to obtain fused sample space features; according to the sample space features, the combined sample visual features are modified to obtain fused combined sample visual features.

[0169] In the above S362, in one instance, the fused combined sample visual features and the fused sample spatial features can be spliced ​​together to obtain spliced ​​sample features. Based on the spliced ​​sample features, the first sample target and the second sample target are subjected to interaction relationship detection to obtain a second interaction relationship detection result.

[0170] The spliced ​​sample features can be input into a second classifier in the interaction relationship detection model, so that the second classifier performs interaction relationship detection on the first sample target and the second sample target, and outputs a second interaction relationship detection result.

[0171] In the above S37, the first sample interaction relationship detection result includes the first probability that the first sample target and the second sample target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship categories. The second sample interaction relationship detection result includes the second probability that the first sample target and the second sample target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship types. Among them, the target interaction relationship detection result includes the target probability that the first sample target and the second sample target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship categories. Specifically, the product of the first probability and the second probability that the first sample target and the second sample target belong to the first alternative interaction relationship category can be used as the target probability that the first sample target and the second sample target belong to the first alternative interaction relationship category. The first alternative interaction relationship category is any one of a plurality of alternative interaction relationship types.

[0172] In this embodiment, the detection results of the two branches are combined to determine the final detection result. Figure 3As shown, the two branches are the first branch and the second branch. The first branch detects interaction relationships from a global perspective, and the second branch detects interaction relationships from the perspective of combining spatial features with visual features.

[0173] In practical applications, such as Figure 3 As shown, a third branch can be added: interaction relationship detection is performed based on spatial features only. Specifically, the interaction relationship detection model is also used to:

[0174] S30: Perform interaction relationship detection on the first sample target and the second sample target according to the sample space feature to obtain a third sample interaction relationship detection result.

[0175] Correspondingly, the above-mentioned S37 "determining the target sample interaction relationship detection result based on the first sample interaction relationship detection result and the second sample interaction relationship detection result" may specifically include: determining the target sample interaction relationship detection result based on the first sample interaction relationship detection result, the second sample interaction relationship detection result and the third sample interaction relationship detection result.

[0176] The first sample interaction relationship detection result includes the first probability that the first sample target and the second sample target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship categories. The second sample interaction relationship detection result includes the second probability that the first sample target and the second sample target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship types. The third sample interaction relationship detection result includes the third probability that the first sample target and the second sample target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship types. Among them, the target sample interaction relationship detection result includes the target probability that the first sample target and the second sample target belong to each alternative interaction relationship category in a plurality of alternative interaction relationship categories. Specifically, the product of the first probability, the second probability and the third probability of the first sample target and the second sample target belonging to the first alternative interaction relationship category can be used as the target probability that the first sample target and the second sample target belong to the first alternative interaction relationship category. The first alternative interaction relationship category is any one of the plurality of alternative interaction relationship types.

[0177] In the embodiment of the present application, integrating the interaction relationship detection results of multiple branches helps to improve the accuracy of interaction relationship detection.

[0178] The following will take the human-vehicle interaction relationship detection as an example to introduce the interaction relationship detection method provided in the above embodiments. That is, in the above embodiments, the image to be detected is specifically a traffic image taken by a traffic camera device, and the multiple targets may include: at least one person and at least one vehicle. Of course, the multiple targets may also include other types of targets besides people and vehicles, such as trees on both sides of the road, bus signs, etc. The first target is the first person, and the second target is the first vehicle. Figure 4 As shown, the method includes:

[0179] 401. Obtain a traffic image captured by a traffic camera.

[0180] The traffic image includes multiple targets; the multiple targets include a first person and a first vehicle.

[0181] 402. Perform interaction relationship detection on the traffic image using the trained interaction relationship detection model.

[0182] Among them, the interaction relationship detection model is used to: determine the visual features of multiple targets in the traffic image based on the traffic image; perform feature fusion on the visual features of the multiple targets to obtain the fused visual features of each of the multiple targets; perform interaction relationship detection on the first person and the first vehicle based on the fused visual features of each of the first person and the first vehicle to obtain a first interaction relationship detection result.

[0183] The specific implementation process of the above steps and the interactive relationship detection model can be found in the corresponding content of the above embodiments and will not be repeated here.

[0184] It should be noted that for any steps not fully described in detail in the methods provided in the embodiments of the present application, reference may be made to the corresponding contents in the aforementioned embodiments, and no further elaboration is required here. Furthermore, in addition to the aforementioned steps, the methods provided in the embodiments of the present application may also include some or all of the other steps in the aforementioned embodiments, for which reference may be made to the corresponding contents in the aforementioned embodiments, and no further elaboration is required here.

[0185] Figure 5 FIG. 1 shows a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 5As shown, the electronic device includes a memory 1101 and a processor 1102. The memory 1101 can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device. The memory 1101 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0186] The memory 1101 is used to store programs;

[0187] The processor 1102 is coupled to the memory 1101 and is used to execute the program stored in the memory 1101 to implement the interaction relationship detection method and model training method provided by the above-mentioned method embodiments.

[0188] Further, if Figure 5 As shown, the electronic device also includes: a communication component 1103, a display 1104, a power component 1105, an audio component 1106 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 Components shown.

[0189] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the interaction relationship detection method and model training method provided in the above-mentioned method embodiments.

[0190] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0191] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting an interactive relationship, wherein: include: Obtain the image to be detected; Performing interaction relationship detection on the image to be detected using a trained interaction relationship detection model; Among them, the interaction relationship detection model is used to: determine the visual features of multiple targets in the image to be detected based on the image to be detected; use the first self-attention network to perform feature fusion on the visual features of the multiple targets to obtain the fused visual features of each of the multiple targets, the fused visual features of each target are fused with its own visual features and the visual features of other targets in the multiple targets, and the fused visual features of different targets are different; based on the fused visual features of the first target and the second target in the multiple targets, perform interaction relationship detection on the first target and the second target to obtain a first interaction relationship detection result.

2. The method according to claim 1, wherein The interaction relationship detection model includes a first self-attention network; The first self-attention network is used to perform feature fusion on the visual features of the multiple targets to obtain fused visual features of each of the multiple targets.

3. The method according to claim 1 or 2, wherein: The interaction relationship detection model is also used to: determining spatial features of the first target and the second target based on position information of the first target and the second target in the image to be detected; determining a combined visual feature of the first object and the second object based on respective visual features of the first object and the second object; performing interaction relationship detection on the first object and the second object based on the combined visual feature and the spatial feature to obtain a second interaction relationship detection result; A target interaction relationship detection result is determined based on the first interaction relationship detection result and the second interaction relationship detection result.

4. The method according to claim 3, wherein: Performing interaction relationship detection on the first object and the second object based on the combined visual feature and the spatial feature to obtain a second interaction relationship detection result includes: Using the second self-attention network in the interaction relationship detection model to perform feature fusion on the combined visual features and the spatial features to obtain fused combined visual features and fused spatial features; An interaction relationship detection is performed on the first target and the second target based on the fused combined visual features and the fused spatial features to obtain a second interaction relationship detection result.

5. The method according to claim 3, wherein Determining visual features of multiple targets in the image to be detected according to the image to be detected includes: Performing feature extraction on the image to be detected to obtain a visual feature map; According to the position information of each of the multiple targets in the image to be detected, the visual features of each of the multiple targets are determined in the visual feature map.

6. The method according to claim 5, wherein: The interaction relationship detection model is also used to: Determining, in the image to be detected, a bounding box surrounding the first object and a bounding box surrounding the second object; Determining visual features of the circumscribed frame area in the visual feature map according to position information of the circumscribed frame in the image to be detected; Determining a combined visual feature of the first object and the second object based on respective visual features of the first object and the second object includes: The visual features of each of the first target and the second target and the visual features of the circumscribed frame area are combined to obtain the combined visual features of the first target and the second target.

7. The method according to claim 3, wherein: The interaction relationship detection model is also used to: performing interaction relationship detection on the first object and the second object according to the spatial feature to obtain a third interaction relationship detection result; Determining a target interaction relationship detection result according to the first interaction relationship detection result and the second interaction relationship detection result includes: A target interaction relationship detection result is determined according to the first interaction relationship detection result, the second interaction relationship detection result, and the third interaction relationship detection result.

8. The method according to claim 1 or 2, wherein: Also includes: Get sample images with interaction relationship labels; Performing interaction relationship detection on the sample image using the interaction relationship detection model to obtain a target sample interaction relationship detection result; Optimizing parameters of the interaction relationship detection model according to the target sample interaction relationship detection result and the interaction relationship label; Among them, the interaction relationship detection model is used to: determine the sample visual features of multiple sample targets in the sample image based on the sample image; perform feature fusion on the sample visual features of the multiple sample targets to obtain the fused sample visual features of each of the multiple sample targets; perform interaction relationship detection on the first sample target and the second sample target in the multiple sample targets based on their respective fused sample visual features to obtain a first sample interaction relationship detection result; the target sample interaction relationship detection result is determined based on the first sample interaction relationship detection result.

9. A model training method, wherein: include: Get sample images with interaction relationship labels; Performing interaction relationship detection on the sample image using an interaction relationship detection model to obtain a target sample interaction relationship detection result; Optimizing parameters of the interaction relationship detection model according to the target sample interaction relationship detection result and the interaction relationship label; Among them, the interaction relationship detection model is used to: determine the sample visual features of multiple sample targets in the sample image based on the sample image; use the first self-attention network to perform feature fusion on the sample visual features of the multiple sample targets to obtain the fused sample visual features of each of the multiple sample targets, the fused visual features of each sample target are fused with its own visual features and the visual features of other sample targets in the multiple sample targets, and the fused sample visual features of different sample targets are different; based on the fused sample visual features of the first sample target and the second sample target in the multiple sample targets, perform interaction relationship detection on the first sample target and the second sample target to obtain a first sample interaction relationship detection result; the target sample interaction relationship detection result is determined based on the first sample interaction relationship detection result.

10. The method according to claim 9, wherein: The interaction relationship detection model is also used to: determining sample space features of the first sample object and the second sample object according to position information of the first sample object and the second sample object in the sample image; determining a combined sample visual feature of the first sample object and the second sample object based on respective sample visual features of the first sample object and the second sample object; Performing feature fusion on the combined sample visual features and the sample spatial features to obtain fused combined sample visual features and fused sample spatial features; performing interaction relationship detection on the first sample target and the second sample target based on the fused combined sample visual features and the fused sample spatial features to obtain a second sample interaction relationship detection result; The target interaction relationship detection result is determined according to the first sample interaction relationship detection result and the second sample interaction relationship detection result.

11. A method for detecting an interactive relationship, wherein: include: Acquiring traffic images captured by traffic camera equipment; The traffic image includes a plurality of objects; the plurality of objects include a first person and a first vehicle; Performing interaction relationship detection on the traffic image using a trained interaction relationship detection model; The interactive relationship detection model is used to: determine visual features of multiple targets in the traffic image based on the traffic image; A first self-attention network is used to perform feature fusion on the visual features of the multiple targets to obtain fused visual features of each of the multiple targets. The fused visual features of each target include its own visual features and the visual features of other targets in the multiple targets, and the fused visual features of different targets are different. Based on the fused visual features of the first person and the first vehicle, an interaction relationship detection is performed on the first person and the first vehicle to obtain a first interaction relationship detection result.

12. An electronic device, wherein: include: memory and a processor, wherein The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the interactive relationship detection method described in any one of claims 1 to 8 and 11 or the model training method described in any one of claims 9 and 10.

13. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a computer, it can implement the interactive relationship detection method described in any one of claims 1 to 8 and 11 or the model training method described in any one of claims 9 and 10.

Images