A method, device, computer equipment and storage medium for detecting relationship

Abstract

The present disclosure provides a relationship detection method, apparatus, computer equipment and storage medium, comprising: acquiring an image to be detected; performing feature extraction on the image to be detected to determine a first feature sequence corresponding to the image to be detected; performing object prediction based on the first feature sequence to determine object information in the image to be detected; and performing relationship prediction based on the first feature sequence to determine relationship information of the image to be detected; matching the object information with the relationship information to determine the relationship between target objects in the image to be detected.

Description

Technical Field

[0001] The present disclosure relates to the field of image detection technology, and in particular to a relationship detection method, device, computer equipment and storage medium. Background Art

[0002] Currently, in the scenario of dangerous action recognition, it can be determined whether there is a dangerous action in the image by identifying objects in the image and the relationship between the objects.

[0003] In the related art, when performing action recognition between objects in an image, the objects in the image are generally recognized first, and then the relationship information between the objects is recognized based on the features corresponding to the area where the objects are located. However, in this method, on the one hand, when recognizing the relationship information, the recognition accuracy will be affected by the recognition accuracy of the objects; on the other hand, since the relationship information is recognized based on the features corresponding to the area where the objects are located, and the relationship information may also be affected by other features, the relationship information recognized based solely on the features corresponding to the area where the objects are located is inaccurate. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a relationship detection method, apparatus, computer equipment, and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a relationship detection method, including:

[0006] Acquire the image to be detected;

[0007] Extracting features from the image to be detected to determine a first feature sequence corresponding to the image to be detected;

[0008] Based on the first feature sequence, performing object prediction to determine object information in the image to be detected; and based on the first feature sequence, performing relationship prediction to determine relationship information of the image to be detected;

[0009] The object information is matched with the relationship information to determine the relationship between the target objects in the image to be detected.

[0010] The relationship detection method provided by the embodiment of the present disclosure can simultaneously perform object prediction and relationship prediction based on the first feature sequence after extracting the first feature sequence corresponding to the image to be detected, detect the object information and relationship information in the image to be detected, and then match the object information and the relationship information to determine the relationship between the target objects in the image to be detected. In this way, since both the object information and the relationship information are detected based on the first feature sequence, the detection results of the object information and the relationship information do not affect each other; the relationship information is detected based on the first feature sequence, so when predicting the relationship information, the global features of the image to be detected can be combined, and the detected relationship information is more accurate, and thus the relationship between the target objects in the image to be detected is also detected with higher accuracy.

[0011] In a possible implementation manner, extracting features from the image to be detected and determining a first feature sequence corresponding to the image to be detected includes:

[0012] Extracting features from the image to be detected to obtain an intermediate feature map corresponding to the image to be detected;

[0013] For each feature point in the intermediate feature map, the feature value of the feature point is adjusted based on the feature values ​​of other feature points in the intermediate feature map except the feature point, so as to obtain the first feature sequence.

[0014] When extracting features from the image to be detected, the only features that can be combined are the features of the pixels around the pixel. In the resulting intermediate feature map, the receptive field of the feature value of each feature point is small. Therefore, by adjusting the feature value of each feature point in the intermediate feature map, each feature point can be integrated with the global features, thereby achieving higher accuracy when predicting objects and relationships based on the first feature sequence.

[0015] In a possible implementation, performing object prediction based on the first feature sequence to determine object information in the image to be detected includes:

[0016] The following steps are executed repeatedly until the preset number of cycles is reached:

[0017] Based on a pre-trained first search feature sequence for searching for object features in the image to be detected, searching from the first feature sequence for a first matching feature sequence that matches the first search feature sequence;

[0018] Merging the first matching feature sequence with the first search feature sequence as a new first search feature sequence, and returning to the step of searching for the first matching feature sequence;

[0019] Based on the first matching feature sequence found last time, object information in the image to be detected is determined.

[0020] When the object information in the image to be detected is determined based on finding the first matching feature sequence multiple times, the determined object information has a higher accuracy.

[0021] In a possible implementation manner, the step of searching for a first matching feature sequence matching the first search feature sequence from the first feature sequence based on a pre-trained first search feature sequence for object features in the image to be detected includes:

[0022] Performing a first fully connected mapping and a second fully connected mapping on the first feature sequence respectively to obtain a first mapping sequence and a second mapping sequence; the first fully connected mapping and the second fully connected mapping correspond to different mapping parameter information, and the mapping parameter information is pre-trained;

[0023] Determine a similarity matrix between the first mapping sequence and the first search feature sequence based on the first mapping sequence and the first search feature sequence;

[0024] Based on the similarity matrix and the second mapping sequence, the first matching feature sequence is determined.

[0025] In a possible implementation, the object information includes: category information of the object in the image to be detected, object semantic information corresponding to the object, and position information of a detection frame corresponding to the object.

[0026] In a possible implementation manner, performing relationship prediction based on the first feature sequence to determine the relationship information of the image to be detected includes:

[0027] The following steps are executed repeatedly until the preset number of cycles is reached:

[0028] Based on a pre-trained second search feature sequence for searching for relational features in the image to be detected, searching from the first feature sequence for a second matching feature sequence that matches the second search feature sequence;

[0029] Based on the second matching feature sequence, searching for a third matching feature sequence matching the second matching feature sequence from the first matching feature sequence matching the current number of cycles;

[0030] Merging the third matching feature sequence, the second matching feature sequence, and the second search feature sequence as a new second search feature sequence, and returning to the step of searching for the second matching feature sequence;

[0031] Based on the second matching feature sequence and the third matching feature sequence found last time, the relationship information of the image to be detected is determined.

[0032] When determining the relationship between objects, in addition to considering the characteristics of the relationship in the image, it is also necessary to consider the characteristics of the objects in the image. Therefore, by searching for a third matching feature sequence that matches the second matching feature sequence from the first matching feature sequence, and then combining the third matching feature sequence to determine the relationship information of the object to be detected, the relationship information determined in this way can be more accurate.

[0033] In a possible implementation manner, the relationship information includes:

[0034] A relationship vector, relationship category information corresponding to the relationship vector, and first object semantic information and second object semantic information of a relationship corresponding to the relationship vector;

[0035] The relationship vector is a vector pointing from the location of the first object where the relationship corresponding to the relationship category information occurs to the location of the second object where the relationship corresponding to the relationship category information occurs.

[0036] In a possible implementation manner, matching the object information with the relationship information to determine the relationship between the target objects in the image to be detected includes:

[0037] For any relationship vector in the relationship information, based on the position information of the detection frames corresponding to each object in the object information and the relationship vector, determine the image distance between any two detection frames and the relationship vector; and,

[0038] Determine a semantic distance between the arbitrary two detection frames and the relationship vector based on the object semantic information corresponding to the arbitrary two detection frames and the first object semantic information and the second object semantic information corresponding to the relationship vector;

[0039] Based on the semantic distance and the image distance corresponding to each relationship vector under the any two detection frames, the relationship between the target objects in the to-be-detected image is determined.

[0040] When determining the relationship between target objects in an image to be detected, the image distance and the semantic distance are considered simultaneously, so that the detection accuracy of the relationship between target objects can be prevented from being affected by the single distance detection accuracy.

[0041] In a possible implementation manner, determining the relationship between the target objects in the to-be-detected image based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames includes:

[0042] The relationship between the target objects in the image to be detected is determined based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames and the confidence of the category information of the different objects.

[0043] When determining the relationship between target objects in an image to be detected, combining the confidence of the category information can improve the accuracy of detecting the relationship between the target objects.

[0044] In a possible implementation, determining the relationship between the target objects in the image to be detected based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames and the confidence of the category information of the different objects includes:

[0045] Determine the matching cost of each relationship vector under the arbitrary two detection frames based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames and the confidence of the category information of the different objects;

[0046] For any relationship vector, determine the two target detection boxes with the minimum matching cost corresponding to the relationship vector;

[0047] Based on the confidence of the object in the target detection box corresponding to each relationship vector and the confidence of the relationship category information corresponding to each relationship vector, the relationship between the target objects in the image to be detected is determined.

[0048] In a possible implementation, after acquiring the image to be detected, the step of determining the relationship between the target objects in the image to be detected is performed by a neural network;

[0049] The neural network is trained by calculating the following loss values ​​during training:

[0050] A first loss value for representing an object prediction loss, a second loss value for representing a relationship prediction loss, and a third loss value for representing a semantic embedding information prediction loss;

[0051] The first loss value includes the loss between the position information of the detection box corresponding to the predicted object and the position information of the marked detection box, and the loss between the category information of the predicted object and the category information of the marked object;

[0052] The second loss value includes the loss between the predicted relationship vector and the labeled relationship vector, and the confidence of the predicted relationship category information;

[0053] The third loss value includes a difference value between semantic embedding information of each object in the object information, and a similarity value between the relationship information and semantic embedding information of the same object in the object information.

[0054] During the training process of the neural network, by combining multiple loss values ​​and adjusting the different processing modules of the neural network in a targeted manner, the training speed and accuracy of the neural network are improved.

[0055] In a second aspect, the embodiment of the present disclosure further provides a relationship detection method, which uses a neural network, including:

[0056] Feature extraction sub-network, object prediction sub-network, relation prediction sub-network, and matching network;

[0057] The feature extraction subnetwork is used to extract features from the image to be detected, determine a first feature sequence corresponding to the image to be detected, and transmit the first feature sequence to the object prediction subnetwork and the relationship prediction subnetwork respectively;

[0058] The object prediction subnetwork is used to perform object prediction based on the first feature sequence, determine the object information in the image to be detected, and transmit the object information to the matching network; and perform relationship prediction on the first feature sequence based on the relationship prediction subnetwork, determine the relationship information of the image to be detected, and transmit the relationship information to the matching network;

[0059] The matching network is used to match the object information with the relationship information to determine the relationship between the target objects in the image to be detected.

[0060] In a possible implementation, the relationship prediction subnetwork obtains feature information from the object prediction subnetwork through an information fusion module, and performs relationship prediction based on the obtained feature information and the first feature sequence to determine the relationship information of the image to be detected.

[0061] In a third aspect, the present disclosure also provides a relationship detection device, including:

[0062] An acquisition module, used for acquiring an image to be detected;

[0063] A first determining module, used to extract features from the image to be detected and determine a first feature sequence corresponding to the image to be detected;

[0064] A second determination module is used to perform object prediction based on the first feature sequence to determine the object information in the image to be detected; and perform relationship prediction based on the first feature sequence to determine the relationship information of the image to be detected;

[0065] The third determination module is used to match the object information with the relationship information to determine the relationship between the target objects in the image to be detected.

[0066] In a possible implementation manner, the first determination module, when extracting features from the image to be detected and determining a first feature sequence corresponding to the image to be detected, is used to:

[0067] Extracting features from the image to be detected to obtain an intermediate feature map corresponding to the image to be detected;

[0068] For each feature point in the intermediate feature map, the feature value of the feature point is adjusted based on the feature values ​​of other feature points in the intermediate feature map except the feature point, so as to obtain the first feature sequence.

[0069] In a possible implementation manner, when the second determination module performs object prediction based on the first feature sequence to determine the object information in the image to be detected, is used to:

[0070] The following steps are executed repeatedly until the preset number of cycles is reached:

[0071] Based on a pre-trained first search feature sequence for searching for object features in the image to be detected, searching from the first feature sequence for a first matching feature sequence that matches the first search feature sequence;

[0072] Merging the first matching feature sequence with the first search feature sequence as a new first search feature sequence, and returning to the step of searching for the first matching feature sequence;

[0073] Based on the first matching feature sequence found last time, object information in the image to be detected is determined.

[0074] In a possible implementation manner, the second determination module, when searching for a first matching feature sequence matching the first search feature sequence from the first feature sequence based on a pre-trained first search feature sequence for searching for object features in the image to be detected, is configured to:

[0075] Performing a first fully connected mapping and a second fully connected mapping on the first feature sequence respectively to obtain a first mapping sequence and a second mapping sequence; the first fully connected mapping and the second fully connected mapping correspond to different mapping parameter information, and the mapping parameter information is pre-trained;

[0076] Determine a similarity matrix between the first mapping sequence and the first search feature sequence based on the first mapping sequence and the first search feature sequence;

[0077] Based on the similarity matrix and the second mapping sequence, the first matching feature sequence is determined.

[0078] In a possible implementation, the object information includes: category information of the object in the image to be detected, object semantic information corresponding to the object, and position information of a detection frame corresponding to the object.

[0079] In a possible implementation manner, the second determination module, when performing relationship prediction based on the first feature sequence to determine the relationship information of the to-be-detected image, is used to:

[0080] The following steps are executed repeatedly until the preset number of cycles is reached:

[0081] Based on a pre-trained second search feature sequence for searching for relational features in the image to be detected, searching from the first feature sequence for a second matching feature sequence that matches the second search feature sequence;

[0082] Based on the second matching feature sequence, searching for a third matching feature sequence matching the second matching feature sequence from the first matching feature sequence matching the current number of cycles;

[0083] Merging the third matching feature sequence, the second matching feature sequence, and the second search feature sequence as a new second search feature sequence, and returning to the step of searching for the second matching feature sequence;

[0084] Based on the second matching feature sequence and the third matching feature sequence found last time, the relationship information of the image to be detected is determined.

[0085] In a possible implementation manner, the relationship information includes:

[0086] A relationship vector, relationship category information corresponding to the relationship vector, and first object semantic information and second object semantic information of the relationship corresponding to the relationship vector;

[0087] The relationship vector is a vector pointing from the location of the first object where the relationship corresponding to the relationship category information occurs to the location of the second object where the relationship corresponding to the relationship category information occurs.

[0088] In a possible implementation manner, the third determination module, when matching the object information with the relationship information to determine the relationship between the target objects in the image to be detected, is used to:

[0089] For any relationship vector in the relationship information, based on the position information of the detection frames corresponding to each object in the object information and the relationship vector, determine the image distance between any two detection frames and the relationship vector; and,

[0090] Determine a semantic distance between the arbitrary two detection frames and the relationship vector based on the object semantic information corresponding to the arbitrary two detection frames and the first object semantic information and the second object semantic information corresponding to the relationship vector;

[0091] Based on the semantic distance and the image distance corresponding to each relationship vector under the any two detection frames, the relationship between the target objects in the to-be-detected image is determined.

[0092] In a possible implementation manner, the third determination module, when determining the relationship between the target objects in the to-be-detected image based on the semantic distance and the image distance corresponding to each relationship vector under the any two detection frames, is used to:

[0093] The relationship between the target objects in the image to be detected is determined based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames and the confidence of the category information of the different objects.

[0094] In a possible implementation manner, the third determination module, when determining the relationship between the target objects in the image to be detected based on the semantic distance and the image distance corresponding to each relationship vector under the any two detection frames and the confidence of the category information of the different objects, is used to:

[0095] Determine the matching cost of each relationship vector under the arbitrary two detection frames based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames and the confidence of the category information of the different objects;

[0096] For any relationship vector, determine the two target detection boxes with the minimum matching cost corresponding to the relationship vector;

[0097] Based on the confidence of the object in the target detection box corresponding to each relationship vector and the confidence of the relationship category information corresponding to each relationship vector, the relationship between the target objects in the image to be detected is determined.

[0098] In a possible implementation manner, after acquiring the image to be detected, the step of determining the relationship between the target objects in the image to be detected may be performed by a neural network;

[0099] The neural network is trained by calculating the following loss values ​​during training:

[0100] A first loss value for representing an object prediction loss, a second loss value for representing a relationship prediction loss, and a third loss value for representing a semantic embedding information prediction loss;

[0101] The first loss value includes the loss between the position information of the detection box corresponding to the predicted object and the position information of the marked detection box, and the loss between the category information of the predicted object and the category information of the marked object;

[0102] The second loss value includes the loss between the predicted relationship vector and the labeled relationship vector, and the confidence of the predicted relationship category information;

[0103] The third loss value includes a difference value between semantic embedding information of each object in the object information, and a similarity value between the relationship information and semantic embedding information of the same object in the object information.

[0104] In a fourth aspect, an embodiment of the present disclosure further provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect, are performed.

[0105] In a fifth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0106] For a description of the effects of the above-mentioned relationship detection apparatus, computer device, and computer-readable storage medium, please refer to the description of the above-mentioned relationship detection method, which will not be repeated here.

[0107] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0109] Figure 1 A flow chart of a relationship detection method provided by an embodiment of the present disclosure is shown;

[0110] Figure 2 A flowchart showing a specific method for determining a first feature sequence in the relationship detection method provided in an embodiment of the present disclosure is shown;

[0111] Figure 3 A flowchart showing a specific method for determining object information in the relationship detection method provided by an embodiment of the present disclosure;

[0112] Figure 4 A flowchart showing a specific method for determining relationship information in the relationship detection method provided by an embodiment of the present disclosure;

[0113] Figure 5 A flowchart showing a specific method for matching object information and relationship information in the relationship detection method provided by an embodiment of the present disclosure;

[0114] Figure 6 A flowchart showing a specific method for detecting the relationship between target objects in the relationship detection method provided by an embodiment of the present disclosure;

[0115] Figure 7a A schematic diagram showing a relationship detection method provided by an embodiment of the present disclosure is shown;

[0116] Figure 7b A schematic diagram of a network architecture of a neural network for performing relationship detection provided by an embodiment of the present disclosure is shown;

[0117] Figure 8 A schematic diagram of the architecture of a relationship detection device provided by an embodiment of the present disclosure is shown;

[0118] Fig. 9 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0119] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0120] In the related art, when performing action recognition between objects in an image, the objects in the image are generally recognized first, and then the relationship information between the objects is recognized based on the features corresponding to the area where the objects are located. However, in this method, on the one hand, when recognizing the relationship information, the recognition accuracy will be affected by the recognition accuracy of the objects; on the other hand, since the relationship information is recognized based on the features corresponding to the area where the objects are located, and the relationship information may also be affected by other features, the relationship information recognized based solely on the features corresponding to the area where the objects are located is inaccurate.

[0121] In addition, the related art also proposes a method for parallel recognition of the relationship between objects in an image. However, in this method, when determining the relationship between objects, multiple anchor frames or intermediate points of the relationship position are generally set in the image, and then the relationship prediction is performed based on the anchor frames or the intermediate points of the relationship position. Here, the anchor frame can be understood as multiple customized detection frames set in the image, and the intermediate point of the position can be understood as the midpoint of the line connecting the center points of two objects with a relationship in the image. This method is equivalent to giving the "relationship" between objects an exact position, and identifying the "relationship" between objects based on the characteristics of the position. However, when identifying the "relationship" between objects, in addition to the position factor, other contextual features need to be paid attention to. For example, even if two relationships occur in the same position, the image features that need to be paid attention to may not be the same. Therefore, this detection method is not accurate in predicting the relationship between objects.

[0122] Based on the above research, the present disclosure provides a relationship detection method. After extracting the first feature sequence corresponding to the image to be detected, object prediction and relationship prediction can be performed based on the first feature sequence at the same time, and the object information and relationship information in the image to be detected can be detected. Then, the object information and the relationship information are matched to determine the relationship between the target objects in the image to be detected. In this way, since both the object information and the relationship information are detected based on the first feature sequence, the detection results of the object information and the relationship information do not affect each other; the relationship information is detected based on the first feature sequence, so when predicting the relationship information, the global features of the image to be detected can be combined, and the detected relationship information is more accurate, and thus the relationship between the target objects in the image to be detected is also detected with higher accuracy.

[0123] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0124] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0125] To facilitate understanding of the present embodiment, a relationship detection method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the relationship detection method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the relationship detection method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0126] See also Figure 1 FIG. 1 is a flowchart of a relationship detection method provided by an embodiment of the present disclosure, wherein the method includes steps 101 to 104, wherein:

[0127] Step 101: Acquire an image to be detected.

[0128] Step 102: extract features from the image to be detected and determine a first feature sequence corresponding to the image to be detected.

[0129] Step 103: perform object prediction based on the first feature sequence to determine object information in the image to be detected; and perform relationship prediction based on the first feature sequence to determine relationship information of the image to be detected.

[0130] Step 104: Match the object information with the relationship information to determine the relationship between the target objects in the image to be detected.

[0131] The following is a detailed description of the above steps.

[0132] For step 101 and step 102,

[0133] In a possible implementation, the image to be detected may be an RGB color image, the image to be detected may include multiple objects to be detected, and the multiple objects to be detected may have a relationship. Exemplarily, the objects to be detected may include a subject object and an object object, the subject object may be the active party initiating the relationship, and the object object may be the recipient corresponding to the relationship. Exemplarily, the subject object may be a person, and the object object may be an object.

[0134] In a possible implementation manner, when extracting features from an image to be detected and determining a first feature sequence corresponding to the image to be detected, the following may be referred to: Figure 2 The method shown includes the following steps:

[0135] Step 201: extract features from the image to be detected to obtain an intermediate feature map corresponding to the image to be detected.

[0136] When extracting features from an image to be detected, the image to be detected can be exemplarily input into a convolutional neural network (CNN), and the convolutional neural network extracts features from the image to be detected to obtain the intermediate feature map.

[0137] Step 202: For each feature point in the intermediate feature map, based on the feature values ​​of other feature points in the intermediate feature map except the feature point, adjust the feature value of the feature point to obtain the first feature sequence.

[0138] Specifically, when extracting features of the image to be detected based on the convolutional neural network, since the convolutional neural network can only combine features of the pixels around the pixel when performing the convolution operation on the pixel in the image to be detected, the receptive field of the feature value of each feature point in the intermediate feature map obtained is small. Therefore, by adjusting the feature value of each feature point in the intermediate feature map through step 202, each feature point can be integrated with the global feature, thereby achieving higher accuracy when performing object prediction and relationship prediction based on the first feature sequence.

[0139] After feature extraction of the image to be detected, an intermediate feature map with a size of W*H can be obtained. Each feature point in the intermediate feature map corresponds to a d-dimensional feature vector, that is, the intermediate feature map is a d-dimensional feature map. After adjusting the feature values ​​of the feature points in the intermediate feature map, the size of the obtained first feature sequence is still W*H, and its corresponding dimension is still d-dimensional.

[0140] In practical applications, after executing step 101 to obtain the intermediate feature map, the intermediate feature map can be input into the encoder of the transformer model, and the encoder of the transformer model adjusts the feature values ​​of the feature points of the intermediate feature map.

[0141] Specifically, since the operation performed in the encoder of the transformer model is sequence-to-sequence, the intermediate feature map can be stretched into a one-dimensional sequence, and the values ​​in the sequence remain unchanged from the values ​​of the feature points of the intermediate feature map; for each feature point in the one-dimensional sequence corresponding to the intermediate feature map, the encoder of the transformer model can perform a self-attention operation based on the feature point and other feature points except the feature point, and adjust the feature value of the feature point. Specifically, the internal execution logic of the encoder of the transformer model will not be described in detail in this disclosure.

[0142] For step 103,

[0143] For object prediction:

[0144] The object information includes at least one of the following: category information of the object in the image to be detected, object semantic information corresponding to the object, and position information of a detection frame corresponding to the object.

[0145] The category information of the object may exemplarily include instance objects such as people, dogs, cats, tables, chairs, vehicles, etc.; the object semantic information corresponding to the object can be used to distinguish the object from other objects in the image to be detected, and can be understood as the unique identifier of the object in the image to be detected. In practical applications, the object semantic information corresponding to the object can be represented by semantic embedding embedding, which is a vector expression of a fixed length, and each position is represented by a floating point number. The semantic embeddings corresponding to different objects in the same image are different; the position information of the detection box corresponding to the object may refer to the position information of the smallest rectangular box containing the object in the image to be detected, and the position information may include, for example, the position coordinates of the four vertices of the rectangle.

[0146] In a possible implementation manner, when performing object prediction based on the first feature sequence to determine the object information in the image to be detected, reference may be made to Figure 3 The method shown includes the following steps:

[0147] Step 301: Based on a pre-trained first search feature sequence for searching for object features in a first feature sequence, search the first feature sequence for a first matching feature sequence that matches the first search feature sequence.

[0148] Here, the size of the first search feature sequence may be fixed during training, and an exemplary size may be N d *d-dimensional, here, the N d It may refer to the maximum number of objects contained in the image. The first search feature sequence may include N d The search feature vectors corresponding to the objects are all d-dimensional, that is, the first search feature sequence can be understood as N d Here, the dimension of the search feature vector corresponding to each object is the same as the dimension of the first feature sequence, which is d dimensions.

[0149] The first matching feature sequence found from the first feature sequence and matching the first search feature sequence may have the same size as the first search feature sequence, both of which are N d *d-dimensional.

[0150] Step 302: Determine whether the current number of cycles reaches the preset number of cycles.

[0151] If not, then execute step 303 in sequence; if yes, then execute step 304;

[0152] Step 303 : merge the first matching feature sequence and the first search feature sequence to form a new first search feature sequence, and return to step 301 .

[0153] Here, the fusion of the first matching feature sequence and the first search feature sequence may refer to performing a preset operation, such as summing, on the values ​​at corresponding positions of the first matching feature sequence and the second search feature sequence. Exemplarily, if the first matching feature sequence is [a1, a2, a3, a4] and the first search feature sequence is [b1, b2, b3, b4], then after the first matching feature sequence and the first search feature sequence are fused, the new first search feature sequence obtained is [a1+b1, a2+b2, a3+b3, a4+b4].

[0154] Step 304: Determine object information in the image to be detected based on the first matching feature sequence.

[0155] Here, the first matching feature sequence in step 304 may refer to the first matching feature sequence found last time from the first feature sequence.

[0156] In a possible implementation, when searching for a first matching feature sequence matching the first search feature sequence from the first feature sequence based on a pre-trained first search feature sequence, the following steps may be included:

[0157] Step a, performing a first fully connected mapping and a second fully connected mapping on the first feature sequence respectively, to obtain a first mapping sequence and a second mapping sequence; the mapping parameter information corresponding to the first fully connected mapping and the second fully connected mapping is different, and the mapping parameter information is pre-trained;

[0158] Step b: determining a similarity matrix between the first mapping sequence and the first search feature sequence based on the first mapping sequence and the first search feature sequence;

[0159] Step c: determining the first matching feature sequence based on the similarity matrix and the second mapping sequence.

[0160] Specifically, the fully connected mapping can be a linear mapping. For example, for the value of each feature point in the first feature sequence, the value can be linearly mapped to ax+b to obtain the value corresponding to the feature point in the mapping sequence. The mapping parameters a and b are different between the first fully connected mapping and the second fully connected mapping. For example, the mapping process of the first fully connected mapping is a1x+b1, and the mapping process of the second fully connected mapping is a2x+b2.

[0161] For example, it can be calculated by the following formula:

[0162]

[0163] Here, A1 represents the similarity matrix, and Q d Represents the first search feature sequence, I s represents the first characteristic sequence, a2I s +b2 represents the first mapping sequence, d represents the dimension, a1, b1, a2, b2 represent the mapping parameters, and the dimension of the first search feature sequence does not change after linear mapping, and remains N. d *d-dimensional sequence, the first mapping sequence has the same size as the first feature sequence, which is a W*H*d-dimensional sequence. Therefore, the similarity matrix A1 is N d A matrix of *W*H dimensions.

[0164] The similarity matrix includes W*H*N d A value is taken, and the value represents the similarity between each element in the first search feature sequence and each element in the first mapping sequence. When determining the first matching feature sequence, for each element in the first search feature sequence, the similarity of the element in the similarity matrix is ​​normalized by softmax to obtain a weight sequence corresponding to each element in the first matching feature sequence, and then the first matching feature sequence is determined based on the weight sequence corresponding to each element in the first matching feature sequence and the second mapping sequence.

[0165] Exemplarily, the first matching feature sequence may be determined by the following formula:

[0166] F d =M(a3I s +b3) (2)

[0167] Among them, F d represents the first matching feature sequence, M represents the matrix after normalizing the values ​​in the similarity matrix A1, a3 and b3 represent mapping parameters, and a3I s +b3 represents the second mapping sequence. The dimension of the matrix M is the same as the similarity matrix, which is N d *W*H dimensional matrix, the second mapping sequence has the same size as the first feature sequence, which is a W*H*d dimensional sequence, so the first matching feature sequence has the same dimension as the first search feature sequence, which is N d *d dimension.

[0168] In a possible implementation, when determining the object information in the image to be detected based on the first matching feature sequence, the first matching feature sequence can be input into multiple multi-layer perceptrons (Multi-Layer Perception, MLP) respectively to obtain the object information of the image to be detected, wherein the multiple multi-layer perceptrons are used to predict different object information respectively.

[0169] For relationship prediction:

[0170] The relationship information includes at least one of the following:

[0171] A relationship vector, relationship category information corresponding to the relationship vector, and first object semantic information and second object semantic information of the relationship corresponding to the relationship vector;

[0172] Among them, the relationship vector is a vector pointing from the position of the first object where the relationship corresponding to the relationship category information occurs (generally refers to the center of the detection box including the first object) to the position of the second object where the relationship corresponding to the relationship category information occurs (generally refers to the center of the detection box including the second object).

[0173] Considering that there may be more than one relationship between the first object and the second object, here, the relationship vector may correspond to multiple similar relationship category information, for example, it may include riding, grabbing, pulling, etc. Generally, the first object is the subject object, and the second subject is the object object. The subject object is the active party of the relationship, and the object object is the recipient of the relationship. The relationship generally refers to an interactive relationship.

[0174] In a possible implementation manner, when performing relationship prediction based on the first feature sequence to determine the relationship information of the image to be detected, it can be referred to Figure 4 The method shown includes the following steps:

[0175] Step 401: based on a pre-trained second search feature sequence for searching for relational features in the image to be detected, searching from the first feature sequence for a second matching feature sequence that matches the second search feature sequence;

[0176] Here, the size of the second search feature sequence during training may also be fixed, and exemplarily may be Nr*d dimensional, where Nr may refer to the number of relations contained in the image. The second search feature sequence may include search feature vectors corresponding to Nr relations, and the search feature vector corresponding to each relation is d dimensional.

[0177] The second matching feature sequence that matches the second search feature sequence found from the first feature sequence may have the same size as the second search feature sequence, both of which are Nr*d dimensions. The specific process of searching for the second matching feature sequence is similar to the process of searching for the first matching feature sequence, and may refer to the above formula, which will not be repeated here.

[0178] Step 402: Determine whether the current number of cycles reaches the preset number of cycles.

[0179] If not, then execute step 403 and step 404 in sequence; if so, then execute step 405.

[0180] Here, the preset number of cycles in step 402 may be the same as the preset number of cycles in step 302 , or the preset number of cycles in step 402 may be less than the preset number of cycles in step 302 .

[0181] Step 403: Based on the second matching feature sequence, search for a third matching feature sequence matching the second matching feature sequence from the first matching feature sequence matching the current cycle number.

[0182] In practical applications, when determining the relationship between objects, in addition to considering the characteristics of the relationship in the image, it is also necessary to consider the characteristics of the objects in the image. Therefore, by searching for a third matching feature sequence that matches the second matching feature sequence from the first matching feature sequence, and then combining the third matching feature sequence to determine the relationship information of the object to be detected, the relationship information determined in this way can be more accurate.

[0183] Since object prediction is also being looped during the relationship prediction loop (see Figure 3 Here, the first matching feature sequence that matches the current sub-cycle number may be the same first matching feature sequence as the current sub-cycle number.

[0184] Exemplarily, if the current cycle of relationship prediction is the Nth cycle, then when searching for the third matching feature sequence in the current cycle, it can be based on the second matching feature sequence in the Nth cycle of relationship prediction, and searching for the third matching feature sequence that matches the second matching feature sequence from the first matching feature sequence in the Nth cycle of object prediction.

[0185] In a possible implementation, when determining the third matching feature sequence, the first matching feature sequence can be first subjected to a third fully-connected mapping and a fourth fully-connected mapping, respectively, to obtain a third mapping sequence and a fourth mapping sequence, and the second matching feature sequence can be subjected to a fifth fully-connected mapping to obtain a fifth mapping sequence, and then a similarity matrix between the third mapping sequence and the fifth mapping sequence is determined, and then based on the similarity matrix between the third mapping sequence and the fifth mapping sequence, and the fourth mapping sequence, the third matching feature sequence is determined, wherein the mapping parameters between the third fully-connected mapping, the fourth fully-connected mapping, and the fifth fully-connected mapping are different.

[0186] For example, it can be calculated by the following formula:

[0187]

[0188] Among them, A2 represents the similarity matrix, F r represents the second matching feature sequence, F d represents the first matching feature sequence, W r , b r , W d , b d Represents a mapping parameter.

[0189] Furthermore, each value in the similarity matrix A2 may be normalized, for example, using softmax for normalization, for example, it may be calculated using the following formula:

[0190]

[0191] Among them, A ij Represents the value of the i-th row and j-th column in the similarity matrix A2.

[0192] F′ r =M(W′ d F d +b' d )+F r (5)

[0193] Among them, F′ r represents the third matching feature sequence, M represents the matrix after normalizing each value in the similarity matrix A2, and W′ d F d +b' d Represents the fourth mapping sequence.

[0194] Step 404: merge the third matching feature sequence, the second matching feature sequence, and the second search feature sequence to form a new second search feature sequence, and return to execute step 401.

[0195] Here, the fusion of the third matching feature sequence, the second matching feature sequence, and the second search feature sequence may be performing a preset operation on the values ​​at corresponding positions of the third matching feature sequence, the second matching feature sequence, and the second search feature sequence, for example, adding them to obtain a new second search feature sequence.

[0196] Step 405: Determine the relationship information of the image to be detected based on the second matching feature sequence and the third matching feature sequence.

[0197] Here, the second matching feature sequence and the third matching feature sequence in step 405 may refer to the second matching feature sequence and the third matching feature sequence found in the last cycle. Determining the relationship information of the image to be detected based on the second matching feature sequence and the third matching feature sequence may be to perform a preset operation on the values ​​at corresponding positions of the second matching feature sequence and the third matching feature sequence, for example, adding them to obtain a fourth matching feature sequence, and then determining the relationship information of the image to be detected based on the fourth matching feature sequence.

[0198] Specifically, when determining the relationship information of the image to be detected based on the fourth matching feature sequence, the fourth matching feature sequence can be input into multiple multilayer perceptrons respectively to obtain the relationship information of the image to be detected, wherein the multiple multilayer perceptrons can be used to predict different relationship information respectively.

[0199] Exemplarily, the fourth matching feature sequence can be input into three multilayer perceptrons, one multilayer perceptron is used to predict the relationship vector, one multilayer perceptron is used to predict the relationship category information corresponding to the relationship vector, and the other multilayer perceptron is used to predict the first object semantic information and the second object semantic information of the relationship between the relationship vector objects.

[0200] For step 104,

[0201] The relationship information and object information in the image to be detected can be determined through steps 101 to 103. However, for any relationship vector in the relationship information, although the first object semantic information and the second object semantic information of the relationship vector are known, and the semantic information of each object is also known in the object information, since the detection of relationship information and the detection of object information are performed through different branches, it may happen that the same object in the image to be detected is detected by two scores, resulting in two semantic information that are not completely the same.

[0202] After determining the relationship information in the image to be detected, although the relationship vector is determined, the detection frame of the first object and the second object of the relationship vector, as well as the category information of the first object and the second object are not known. Therefore, it is necessary to match the object information and the relationship information to determine the detection frame of the corresponding object for each relationship vector, as well as the category information of the object.

[0203] In a possible implementation manner, when matching the object information and the relationship information to determine the relationship between the target objects in the image to be detected, the following can be referred to: Figure 5 The method shown includes the following steps:

[0204] Step 501: for any relationship vector in the relationship information, determine the image distance between each detection frame and the relationship vector based on the position information of the detection frame corresponding to each object in the object information and the relationship vector; and determine the semantic distance based on the object semantic information corresponding to each object in the object information and the first object semantic information and the second object semantic information corresponding to the relationship vector.

[0205] Step 502: Determine the relationship between the target objects in the image to be detected based on the semantic distance and the image distance corresponding to each relationship vector under the category information of different objects.

[0206] In a possible implementation, when determining the relationship between target objects in the image to be detected based on the semantic distance and image distance corresponding to each relationship vector under the category information of different objects, the confidence of the object category can also be combined, that is, based on the semantic distance and image distance corresponding to each relationship vector under the category information of different objects, and the confidence of the category information of the different objects, the relationship between the target objects in the image to be detected is determined.

[0207] In a possible implementation manner, when determining the relationship between target objects in the image to be detected, the following may be referred to: Figure 6 The method shown includes the following steps:

[0208] Step 601: determine the matching cost of each relationship vector under the arbitrary two detection frames based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames and the confidence of the category information of the different objects.

[0209] For any relationship vector in the relationship information, it is necessary to determine the position information of the two detection boxes and the category information of the two objects for the relationship vector. Since the relationship vector is a vector pointing from the position of the first object to the position of the second object, the relationship vector has two specific coordinates in the image to be detected, one is the vector starting point position coordinate, and the other is the vector end point position coordinate.

[0210] Exemplarily, the image distance between any two detection boxes and the relationship vector can be calculated by the following formula:

[0211]

[0212] Where D represents the image distance, Represents the starting point position coordinates of the relationship vector, Represents the end point position coordinates of the relationship vector, Represents the coordinates of the center points of the two detection boxes predicted in the relationship information.

[0213] The semantic distance can be calculated by the following formula:

[0214]

[0215] Among them, R represents the semantic distance, ε h represents the semantic information of the first object, ε o Represents the semantic information of the second object, Represents the object semantic information corresponding to the predicted detection box in the relationship information.

[0216] The matching cost between any two detection boxes and relationship vectors can be calculated by the following formula:

[0217]

[0218] Among them, c represents the matching cost, s h and o Indicates the confidence of the object corresponding to the detection box, and o Can correspond to an object, and h Can correspond to an object.

[0219] Step 602: For any relationship vector, determine two target detection boxes with the minimum matching cost corresponding to the relationship vector.

[0220] Step 603: Determine the relationship between the target objects in the to-be-detected image based on the confidence of the object in the target detection box corresponding to each relationship vector and the confidence of the relationship category information corresponding to each relationship vector.

[0221] Specifically, since each relationship vector may correspond to multiple relationship category information, that is, one relationship vector represents multiple relationships, after determining the target detection box corresponding to each relationship vector, the triples corresponding to each relationship vector can be determined first, where the triples include <first object, second object, relationship>, and then the target triples are determined based on the product of the confidences corresponding to each triple.

[0222] Exemplarily, if the relationship category information corresponding to relationship vector 1 is a, b, c, and the objects of the two target detection boxes corresponding to relationship vector 1 are object 1 and object 2, then the triples corresponding to relationship vector 1 include three, namely <object 1, object 2, relationship a>, <object 1, object 2, relationship b>, and <object 1, object 2, relationship c>.

[0223] After determining the triplet corresponding to each relationship vector, the confidence of the first object in each triplet * the confidence of the second object * the confidence of the relationship can be used as the confidence of the triplet, and then the confidences of all triplets are sorted from large to small, and the first N triplets are selected as target triplets, and then based on the target triplets, the relationship between the target objects in the image to be detected is determined.

[0224] For example, during the prediction process, the relationship category information corresponding to the relationship vector 1 is a, b, c, but the triplet containing the relationship c is not the target triplet. Then, the relationship category information corresponding to the relationship vector finally determined is only a and b, that is, the relationship between the target objects corresponding to the relationship vector 1 is only a and b.

[0225] As another example, after screening, the target triplet is <object 1, object 2, relationship a>, <object 1, object 2, relationship b>, <object 3, object 4, relationship b>, <object 3, object 4, relationship c>, <object 3, object 4, relationship d>, then the target objects in the image to be detected are object 1, object 2, object 3 and object 4, the relationship between object 1 and object 2 is relationship a and relationship b, and the relationship between object 3 and object 4 is relationship c and relationship d.

[0226] In a possible implementation, after acquiring the image to be detected, the step of determining the relationship between the target objects in the image to be detected can be performed by a neural network, that is, after acquiring the image to be detected, the image to be detected can be input into the neural network, and the neural network executes steps 102 to 104.

[0227] The neural network is trained by calculating the following loss values ​​during training:

[0228] A first loss value for representing an object prediction loss, a second loss value for representing a relationship prediction loss, and a third loss value for representing a semantic embedding information prediction loss;

[0229] The first loss value includes the loss between the position information of the detection box corresponding to the predicted object and the position information of the marked detection box, and the loss between the category information of the predicted object and the category information of the marked object;

[0230] The second loss value includes the loss between the predicted relationship vector and the labeled relationship vector, and the confidence of the predicted relationship category information;

[0231] The third loss value includes a difference value between semantic embedding information of each object in the object information, and a similarity value between the relationship information and semantic embedding information of the same object in the object information.

[0232] Specifically, when the neural network is being trained, the supervision data of the sample image may include the position information of the detection frame of the object in the sample image, the category information of the object, and the relationship vector determined based on the position information of the detection frame of the object.

[0233] In one possible implementation, when the neural network predicts the detection boxes, the number of predicted detection boxes may exceed the number of annotated detection boxes in the sample image. For example, the number of predicted detection boxes may be fixed, for example, 100, while there are actually only 10 objects in the sample image, that is, there are 10 annotated boxes. Therefore, before calculating the loss value of the neural network, it is necessary to match the corresponding predicted detection box for each annotated detection box. For the predicted detection box that is not matched, the supervision data of the corresponding detection box may be empty, and the supervision data of the object category in the corresponding detection box is a preset negative sample category.

[0234] Among them, when matching each labeled detection box with the corresponding predicted detection box, it can be calculated by the following formula, for example:

[0235]

[0236] Among them, yi represents the supervised detection box corresponding to the i-th predicted detection box, It is represented as the i-th predicted detection box, and S represents the set of predicted detection boxes; Indicates that when the i-th labeled detection box is not empty, the sum of the matching costs between the i-th labeled detection box and the predicted detection box that matches the detection box is calculated. When the i-th labeled detection box is empty, L match The value of is set to zero; Represents the predicted detection box that matches each non-empty detection box when the matching cost is minimized.

[0237] Here, the predicted detection box matched with the labeled detection box can be understood as which labeled detection box the predicted detection box should use as supervision data.

[0238] When calculating the matching cost between the i-th annotated detection box and the predicted detection box that matches the detection box, the image distance (for example, by Formula 6) and the weighted sum of the semantic distance between the i-th annotated detection box and the predicted detection box can be calculated, and the semantic distance can be understood as the negative of the confidence of the predicted detection box.

[0239] The first loss can be calculated by the following formula:

[0240]

[0241] in, represents the confidence of the predicted category information corresponding to the i-th detection box, bi represents the position information of the i-th detection box, Represents the position information of the predicted detection box corresponding to the i-th detection box, It means that when the i-th detection box is not empty, the weighted sum of the l1 loss and the Generalized Intersection over Union (GIOU) loss between the i-th detection box and the predicted detection box corresponding to the i-th detection box is calculated.

[0242] When calculating the second loss, similar to the above method, it is also necessary to first match each labeled relationship vector with the corresponding predicted relationship vector. The calculation method is similar to formula (9), but for L match The calculation of gives the following definition:

[0243]

[0244] Here, the labeled relationship can be represented by t=(v, z), where v represents the relationship vector corresponding to the t-th relationship, z represents the set of relationship categories corresponding to the relationship vector, and L represents the number of relationship categories. represents the predicted relationship vector corresponding to the i-th relationship vector, It represents the score of the lth relationship category when determining whether it is the relationship category corresponding to the i-th relationship vector during classification; It means calculating the l1 distance between the i-th relationship vector and the predicted relationship vector corresponding to the i-th relationship vector.

[0245] When calculating the second loss value, you can refer to the following formula:

[0246]

[0247] Among them, L cls represents the focal loss, It means calculating the Focal loss between the set of labeled relationship categories corresponding to the i-th relationship vector and the set of predicted relationship categories corresponding to the i-th relationship vector. Indicates that when the labeled relationship vector corresponding to the i-th relationship vector is not empty, the l1 loss between the labeled relationship vector and the predicted relationship vector is calculated, λ cls , reg is the weight parameter.

[0248] When calculating the third loss value, the difference value can be calculated by referring to the following formula:

[0249]

[0250] t is the preset parameter value, Indicates the total number of annotated objects. Represents the semantic information of the predicted object that matches the i-th object.

[0251] The similarity value is calculated according to the following formula:

[0252]

[0253] in, represents the predicted first object semantic information, represents the predicted semantic information of the second object, Indicates the semantic information of the first object predicted during relationship information prediction. Indicates the semantic information of the second object predicted when performing relationship prediction.

[0254] In summary, when calculating the total loss in this training process based on the first loss value, the second loss value, and the third loss value, you can refer to the following formula:

[0255] L=L ins +L int +λ emb (L pull +L push ) (15)

[0256] Among them, λ emb is the weight parameter.

[0257] The following will describe the above relationship detection method in combination with specific examples. Figure 7aFIG. 1 is a schematic diagram of a relationship detection method provided by an embodiment of the present disclosure, including a feature extraction module, an object detection module, a relationship detection module, and a matching module, and specifically includes the following steps:

[0258] Feature extraction module:

[0259] Step 1: Input the acquired color image into the CNN network for feature extraction, and input the extracted intermediate feature map into the encoder of the transformer to obtain the first feature sequence Is' with a dimension of W*H*d.

[0260] Object Detection Module:

[0261] Step 2: The object detection module is based on the first search feature sequence Q d , find the first matching feature sequence F from the first feature sequence d .

[0262] Step 3: The first matching feature sequence F d The information is input into three MLPs to respectively predict the category information of the object in the image to be detected, the object semantic information corresponding to the object, and the position information of the detection box corresponding to the object.

[0263] Relationship detection module:

[0264] Step 4: The relationship detection module is based on the second search feature sequence Q r , find the second matching feature sequence F from the first feature sequence r .

[0265] Step 5: Based on the second matching feature sequence F r From the first matching feature sequence F d Find the matching feature sequence in and compare it with the second matching feature sequence F r Sum and get the second matching feature sequence F r '.

[0266] Step 6: Second matching feature sequence F r 'Input them into three MLPs respectively, and predict the relationship vector, the relationship category information corresponding to the relationship vector, and the first object semantic information and the second object semantic information of the relationship corresponding to the relationship vector respectively.

[0267] Matching module:

[0268] Step 7: Match the information obtained in step 3 and step 6 to determine the relationship between the target objects in the image to be detected.

[0269] It should be noted that there are loops in the execution process of the relationship detection module and the object detection module. The above description is only a brief description. The specific loop process refers to the detailed description in the above embodiment, which will not be repeated here.

[0270] The relationship detection method provided by the disclosed embodiment can simultaneously perform object prediction and relationship prediction based on the first feature sequence after extracting the first feature sequence corresponding to the image to be detected, detect the object information and relationship information in the image to be detected, and then match the object information and relationship information to determine the relationship between the target objects in the image to be detected. In this way, since both the object information and the relationship information are detected based on the first feature sequence, the detection results of the object information and the relationship information do not affect each other; the relationship information is detected based on the first feature sequence, so when predicting the relationship information, the global features of the image to be detected can be combined, and the detected relationship information is more accurate, and thus the relationship between the target objects in the image to be detected is also detected with higher accuracy.

[0271] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0272] Based on the same concept, the embodiment of the present disclosure also provides a method for relationship detection, which adopts a neural network architecture such as Figure 7b As shown, it includes a feature extraction subnetwork, an object prediction subnetwork, a relationship prediction subnetwork, and a matching network;

[0273] The feature extraction subnetwork is used to extract features from the image to be detected, determine a first feature sequence corresponding to the image to be detected, and transmit the first feature sequence to the object prediction subnetwork and the relationship prediction subnetwork respectively;

[0274] The object prediction subnetwork is used to perform object prediction based on the first feature sequence, determine the object information in the image to be detected, and transmit the object information to the matching network; and perform relationship prediction on the first feature sequence based on the relationship prediction subnetwork, determine the relationship information of the image to be detected, and transmit the relationship information to the matching network;

[0275] The matching network is used to match the object information with the relationship information to determine the relationship between the target objects in the image to be detected.

[0276] The relationship prediction subnetwork obtains feature information from the object prediction subnetwork through an information fusion module, and performs relationship prediction in combination with the obtained feature information and the first feature sequence to determine the relationship information of the image to be detected.

[0277] In fact, the feature extraction subnetwork is equivalent to Figure 7a The feature extraction module described in the invention is equivalent to the object detection module, the relationship prediction subnetwork is equivalent to the relationship detection module, and the matching network is equivalent to the matching network. The specific execution process of the feature extraction subnetwork, the object prediction subnetwork, the relationship prediction subnetwork, and the matching network can refer to the description of the above method and will not be repeated here.

[0278] It should be noted that the object prediction subnetwork and the relationship prediction subnetwork can be predicted simultaneously. The object prediction subnetwork and the relationship prediction subnetwork exchange information through the information fusion module. The specific information exchange process refers to the description of the embodiment of the method and will not be repeated here.

[0279] Combine the following Figure 7a An embodiment of the disclosed relationship detection method is briefly introduced in terms of the dimension of a neural network component module. In this embodiment, the relationship between a person and an object can be detected:

[0280] The feature extraction sub-network:

[0281] Input: The entire image.

[0282] Output: Image feature sequence Is'.

[0283] Function description: Use a convolutional network to extract visual features from the image, then use the self-attention module to use global features to fine-tune the features, merge the spatial dimensions of the output two-dimensional features I, and obtain a feature-fine-tuned feature sequence Is' (i.e. the first feature sequence mentioned above) of length WH.

[0284] The object prediction subnetwork:

[0285] Input: Image feature sequence Is'

[0286] Output: category information of the object in the image to be detected, object semantic information corresponding to the object, and location information of the detection box corresponding to the object.

[0287] Functional description: The object prediction subnetwork branch is used to locate and classify human / object instances. This branch consists of a multi-layer attention module-based decoder and some prediction heads based on feedforward networks. The input of the first layer of decoder is a learnable query embedding sequence set (i.e., the first search feature sequence mentioned above), and the input of each subsequent layer of decoder is the sum of the output of the previous layer and the query embedding sequence set. Each layer of this branch first adjusts the input features through the self-attention module, and then uses it as a new query input to the mutual attention module to aggregate the key features F related to the query content from the image feature sequence Is' obtained from the feature extraction subnetwork. d The prediction head is generally composed of a multi-layer perceptron or a single-layer linear mapping structure. Each layer of the decoder in this branch is connected to three prediction heads at the top, and F is obtained based on the key features. d Prediction results. One prediction head is composed of three layers of perceptrons, which is used to predict the relative position (cx, cy, w, h) of the instance (i.e., the position information of the detection box mentioned above); one prediction head is composed of a single-layer linear mapping structure, which is used to predict the category information of the instance, and each instance can only have one category; the remaining one is a semantic embedding prediction head composed of three layers of perceptrons, which predicts an 8-dimensional semantic embedding representation for each instance that is distinguished from other instances (i.e., the object semantic information mentioned above).

[0288] During the training process, the position distance and target category score between each target instance and the prediction result are comprehensively considered, the matching cost is calculated, and bipartite matching is used to match the most appropriate prediction in the prediction set for each target for supervision. The position distance is calculated using the L1 distance and the weighted sum of the GIoU of the prediction box and the target box. This method uses the learning of the target supervision detection branch by regressing the detection box, supervising the category prediction, and pulling the semantic embedding distance between instances.

[0289] The relationship prediction subnetwork:

[0290] Input: Image feature sequence Is'

[0291] Output: a relationship vector, relationship category information corresponding to the relationship vector, and first object semantic information and second object semantic information of the relationship corresponding to the relationship vector.

[0292] Functional description: The object prediction subnetwork branch is designed to locate relations and identify corresponding action categories. Similar to the object prediction subnetwork branch, this branch is also composed of a multi-layer decoder based on the attention module and some prediction heads based on the feedforward network. The difference is that the query of this branch is another learnable relation query embedding training set (i.e., the second search feature sequence mentioned above). Accordingly, the query set is aggregated from the image feature sequence Is' obtained from the feature extraction subnetwork through the mutual attention module to obtain the key feature sequence F r, key feature sequence F r It is a feature suitable for relationship prediction. This branch has four prediction heads. A three-layer perceptron prediction head is used to predict the relative position of the starting and ending points of the relationship vector (i.e. the above-mentioned relationship vector); a single-layer linear mapping prediction head is used to predict the action category for each relationship vector (i.e. the relationship category information corresponding to the above-mentioned relationship vector), and a relationship vector can correspond to multiple categories; and two three-layer perceptron prediction heads predict a semantic embedding vector for the person and object instance corresponding to the relationship vector respectively (i.e. the above-mentioned first object semantic information and second object semantic information). The semantic embedding vector predicted by this branch is as close as possible to the semantic embedding vector predicted by the detection branch for the same instance.

[0293] During the training process, similar to the object prediction subnetwork branch, this method uses bipartite matching to match the most appropriate prediction in the prediction set for each target for supervision. The L1 distance is used to calculate the distance between the predicted relationship vector position and the target relationship vector position, and the category distance corresponding to each relationship vector is the negative value of the sum of the target action category scores. This method supervises the relationship prediction branch by regressing the relationship vector position, supervising the action category prediction, and shortening the semantic embedding distance between the two branches pointing to the same instance.

[0294] The information integration module:

[0295] Input: The detection branch queries the key feature sequence F aggregated from the feature sequence obtained from the feature extraction subnetwork d , and the key feature sequence F obtained by query aggregation of the relationship prediction branch r .

[0296] Output: The key feature F of relationship prediction that combines the features of the detection branch r '

[0297] This module first calculates F d With F r The similarity matrix A is obtained by using the above formula (3).

[0298] Then, the detection feature weight corresponding to each relationship prediction is normalized using Softmax to obtain the detection feature weight M relative to each relationship. The detection feature is weighted using this weight, and the detected relationship-related feature is transformed. This feature is added to the original relationship key feature to obtain the refined relationship feature Fr' with detection feature guidance information. The transformation process is as shown in the above formula (5):

[0299] The visualization results show that this module mainly focuses on the posture features of the human body corresponding to the relationship vector (such as the head and arms of the person holding the umbrella), while the relationship prediction sub-network branch focuses on more boundary areas and object features between people and objects (such as the upper body of the person holding the umbrella and the umbrella). This module is mainly used to refine the dependent features of relationship prediction. If this feature is not used, the features aggregated by the relationship prediction sub-network branch need to pay attention to the features of people, objects, and related environments at the same time, which is relatively rough. Using this module can improve the effect of this method, but you can also choose not to use it.

[0300] The matching network:

[0301] After obtaining the prediction results of the two branches of the object prediction sub-network and the relationship prediction sub-network, the predicted relationship vector is matched with the detected instance according to the relationship vector and semantic embedding.

[0302] For each predicted relationship vector The above formula (6) can be used to calculate the matching distance D between the detected person and the object, where Represent the center points of detected people and objects respectively.

[0303] Furthermore, semantic embeddings can also be used to assist in matching. It is worth noting that when semantic embeddings are introduced during training, the semantic embeddings pointing to the same instance in the two branches are consistent. Therefore, the relationship prediction branch can be implicitly aggregated to features related to detection, which helps to improve the accuracy of relationship prediction. Compared with not using semantic embeddings, the method used can focus on more instance-related areas (such as the arm holding the umbrella, and the edge of the umbrella). However, you can also choose not to predict semantic embeddings and only use relationship vectors to match relationship triplets in post-processing. If semantic embeddings are used in post-processing, the accuracy of person-object relationship prediction can be stably and slightly improved. For the person embedding vector predicted by the relationship prediction branch and the embedding vector of the object The embedding vector ε of the branch prediction can be calculated and tested h and the object embedding vector ε o Calculating semantic embedding distance The formula is as shown in formula (7).

[0304] The final matching cost can be expressed as formula (8), where s h and o denote the confidence of the matched detected person and object, respectively. If semantic embedding is not used, the numerator is only D.

[0305] Furthermore, for each predicted relationship vector, the detected person-object instance pair with the lowest matching cost is selected, and finally several (e.g., 100) person-object relationship triplets with the highest product of person, object confidence and corresponding action category confidence are selected as the final prediction results.

[0306] Based on the same inventive concept, a relationship detection device corresponding to the relationship detection method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned relationship detection method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0307] Reference Figure 8 , which is a schematic diagram of the architecture of a relationship detection device provided by an embodiment of the present disclosure, the device includes: an acquisition module 801, a first determination module 802, a second determination module 803, and a third determination module 804; wherein,

[0308] The acquisition module 801 is used to acquire the image to be detected;

[0309] A first determining module 802 is used to extract features from the image to be detected and determine a first feature sequence corresponding to the image to be detected;

[0310] A second determination module 803 is configured to perform object prediction based on the first feature sequence to determine object information in the image to be detected; and perform relationship prediction based on the first feature sequence to determine relationship information of the image to be detected;

[0311] The third determination module 804 is used to match the object information with the relationship information to determine the relationship between the target objects in the image to be detected.

[0312] In a possible implementation manner, the first determining module 802, when extracting features from the image to be detected and determining a first feature sequence corresponding to the image to be detected, is configured to:

[0313] Extracting features from the image to be detected to obtain an intermediate feature map corresponding to the image to be detected;

[0314] For each feature point in the intermediate feature map, the feature value of the feature point is adjusted based on the feature values ​​of other feature points in the intermediate feature map except the feature point, so as to obtain the first feature sequence.

[0315] In a possible implementation manner, the second determination module 803, when performing object prediction based on the first feature sequence to determine the object information in the image to be detected, is configured to:

[0316] The following steps are executed repeatedly until the preset number of cycles is reached:

[0317] Based on a pre-trained first search feature sequence for searching for object features in the image to be detected, searching from the first feature sequence for a first matching feature sequence that matches the first search feature sequence;

[0318] Merging the first matching feature sequence with the first search feature sequence as a new first search feature sequence, and returning to the step of searching for the first matching feature sequence;

[0319] Based on the first matching feature sequence found last time, object information in the image to be detected is determined.

[0320] In a possible implementation manner, the second determination module 803, when searching for a first matching feature sequence matching the first search feature sequence from the first feature sequence based on a pre-trained first search feature sequence for searching for object features in the image to be detected, is configured to:

[0321] Performing a first fully connected mapping and a second fully connected mapping on the first feature sequence respectively to obtain a first mapping sequence and a second mapping sequence; the first fully connected mapping and the second fully connected mapping correspond to different mapping parameter information, and the mapping parameter information is pre-trained;

[0322] Determine a similarity matrix between the first mapping sequence and the first search feature sequence based on the first mapping sequence and the first search feature sequence;

[0323] Based on the similarity matrix and the second mapping sequence, the first matching feature sequence is determined.

[0324] In a possible implementation, the object information includes: category information of the object in the image to be detected, object semantic information corresponding to the object, and position information of a detection frame corresponding to the object.

[0325] In a possible implementation manner, the second determination module 803, when performing relationship prediction based on the first feature sequence to determine the relationship information of the to-be-detected image, is configured to:

[0326] The following steps are executed repeatedly until the preset number of cycles is reached:

[0327] Based on a pre-trained second search feature sequence for searching for relational features in the image to be detected, searching from the first feature sequence for a second matching feature sequence that matches the second search feature sequence;

[0328] Based on the second matching feature sequence, searching for a third matching feature sequence matching the second matching feature sequence from the first matching feature sequence matching the current number of cycles;

[0329] Merging the third matching feature sequence, the second matching feature sequence, and the second search feature sequence as a new second search feature sequence, and returning to the step of searching for the second matching feature sequence;

[0330] Based on the second matching feature sequence and the third matching feature sequence found last time, the relationship information of the image to be detected is determined.

[0331] In a possible implementation manner, the relationship information includes:

[0332] A relationship vector, relationship category information corresponding to the relationship vector, and first object semantic information and second object semantic information of a relationship corresponding to the relationship vector;

[0333] The relationship vector is a vector pointing from the location of the first object where the relationship corresponding to the relationship category information occurs to the location of the second object where the relationship corresponding to the relationship category information occurs.

[0334] In a possible implementation manner, the third determination module 804, when matching the object information with the relationship information to determine the relationship between the target objects in the image to be detected, is configured to:

[0335] For any relationship vector in the relationship information, based on the position information of the detection frames corresponding to each object in the object information and the relationship vector, determine the image distance between any two detection frames and the relationship vector; and,

[0336] Determine a semantic distance between the arbitrary two detection frames and the relationship vector based on the object semantic information corresponding to the arbitrary two detection frames and the first object semantic information and the second object semantic information corresponding to the relationship vector;

[0337] Based on the semantic distance and the image distance corresponding to each relationship vector under the any two detection frames, the relationship between the target objects in the to-be-detected image is determined.

[0338] In a possible implementation manner, the third determination module 804, when determining the relationship between the target objects in the to-be-detected image based on the semantic distance and the image distance corresponding to each relationship vector under the any two detection frames, is configured to:

[0339] The relationship between the target objects in the image to be detected is determined based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames and the confidence of the category information of the different objects.

[0340] In a possible implementation manner, the third determination module 804, when determining the relationship between the target objects in the image to be detected based on the semantic distance and the image distance corresponding to each relationship vector under the any two detection frames and the confidence of the category information of the different objects, is used to:

[0341] Determine the matching cost of each relationship vector under the arbitrary two detection frames based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames and the confidence of the category information of the different objects;

[0342] For any relationship vector, determine the two target detection boxes with the minimum matching cost corresponding to the relationship vector;

[0343] Based on the confidence of the object in the target detection box corresponding to each relationship vector and the confidence of the relationship category information corresponding to each relationship vector, the relationship between the target objects in the image to be detected is determined.

[0344] In a possible implementation manner, after acquiring the image to be detected, the step of determining the relationship between the target objects in the image to be detected may be performed by a neural network;

[0345] The neural network is trained by calculating the following loss values ​​during training:

[0346] A first loss value for representing an object prediction loss, a second loss value for representing a relationship prediction loss, and a third loss value for representing a semantic embedding information prediction loss;

[0347] The first loss value includes the loss between the position information of the detection box corresponding to the predicted object and the position information of the marked detection box, and the loss between the category information of the predicted object and the category information of the marked object;

[0348] The second loss value includes the loss between the predicted relationship vector and the labeled relationship vector, and the confidence of the predicted relationship category information;

[0349] The third loss value includes a difference value between semantic embedding information of each object in the object information, and a similarity value between the relationship information and semantic embedding information of the same object in the object information.

[0350] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0351] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Fig. 9 As shown, it is a schematic diagram of the structure of a computer device 900 provided in an embodiment of the present disclosure, including a processor 901, a memory 902, and a bus 903. Among them, the memory 902 is used to store execution instructions, including a memory 9021 and an external memory 9022; the memory 9021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 901, and the data exchanged with the external memory 9022 such as a hard disk. The processor 901 exchanges data with the external memory 9022 through the memory 9021. When the computer device 900 is running, the processor 901 communicates with the memory 902 through the bus 903, so that the processor 901 executes the following instructions:

[0352] Acquire the image to be detected;

[0353] Extracting features from the image to be detected to determine a first feature sequence corresponding to the image to be detected;

[0354] Based on the first feature sequence, performing object prediction to determine object information in the image to be detected; and based on the first feature sequence, performing relationship prediction to determine relationship information of the image to be detected;

[0355] The object information is matched with the relationship information to determine the relationship between the target objects in the image to be detected.

[0356] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the relationship detection method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0357] The present disclosure also provides a computer program product that carries a program code. The program code includes instructions that can be used to execute the steps of the relationship detection method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0358] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0359] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0360] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0361] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0362] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0363] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A relationship detection method, characterized in that: include: Acquire the image to be detected; Extracting features from the image to be detected to obtain an intermediate feature map corresponding to the image to be detected; For each feature point in the intermediate feature map, based on feature values ​​of other feature points in the intermediate feature map except the feature point, adjusting the feature value of the feature point to obtain a first feature sequence; Based on the first feature sequence, performing object prediction to determine object information in the image to be detected; and based on the first feature sequence, performing relationship prediction to determine relationship information of the image to be detected; The object information is matched with the relationship information to determine the relationship between the target objects in the image to be detected.

2. The method according to claim 1, characterized in that The performing object prediction based on the first feature sequence to determine the object information in the image to be detected includes: The following steps are executed repeatedly until the preset number of cycles is reached: Based on a pre-trained first search feature sequence for searching for object features in the image to be detected, searching from the first feature sequence for a first matching feature sequence that matches the first search feature sequence; Merging the first matching feature sequence with the first search feature sequence as a new first search feature sequence, and returning to the step of searching for the first matching feature sequence; Based on the first matching feature sequence found last time, object information in the image to be detected is determined.

3. The method according to claim 2, characterized in that The step of searching for a first matching feature sequence matching the first search feature sequence from the first feature sequence based on a pre-trained first search feature sequence for object features in the image to be detected includes: Performing a first fully connected mapping and a second fully connected mapping on the first feature sequence respectively to obtain a first mapping sequence and a second mapping sequence; the first fully connected mapping and the second fully connected mapping correspond to different mapping parameter information, and the mapping parameter information is pre-trained; Determine a similarity matrix between the first mapping sequence and the first search feature sequence based on the first mapping sequence and the first search feature sequence; Based on the similarity matrix and the second mapping sequence, the first matching feature sequence is determined.

4. The method according to claim 1, characterized in that: The object information includes: category information of the object in the image to be detected, object semantic information corresponding to the object, and position information of a detection frame corresponding to the object.

5. The method according to claim 2, characterized in that: The performing relationship prediction based on the first feature sequence to determine the relationship information of the image to be detected includes: The following steps are executed repeatedly until the preset number of cycles is reached: Based on a pre-trained second search feature sequence for searching for relational features in the image to be detected, searching from the first feature sequence for a second matching feature sequence that matches the second search feature sequence; Based on the second matching feature sequence, searching for a third matching feature sequence matching the second matching feature sequence from the first matching feature sequence matching the current number of cycles; Merging the third matching feature sequence, the second matching feature sequence, and the second search feature sequence as a new second search feature sequence, and returning to the step of searching for the second matching feature sequence; Based on the second matching feature sequence and the third matching feature sequence found last time, the relationship information of the image to be detected is determined.

6. The method according to claim 1, characterized in that The relationship information includes: A relationship vector, relationship category information corresponding to the relationship vector, and first object semantic information and second object semantic information of the relationship corresponding to the relationship vector; The relationship vector is a vector pointing from the location of the first object where the relationship corresponding to the relationship category information occurs to the location of the second object where the relationship corresponding to the relationship category information occurs.

7. The method according to claim 1, characterized in that The matching the object information with the relationship information to determine the relationship between the target objects in the image to be detected includes: For any relationship vector in the relationship information, based on the position information of the detection frames corresponding to each object in the object information and the relationship vector, determine the image distance between any two detection frames and the relationship vector; and, Determine a semantic distance between the arbitrary two detection frames and the relationship vector based on the object semantic information corresponding to the arbitrary two detection frames and the first object semantic information and the second object semantic information corresponding to the relationship vector; Based on the semantic distance and the image distance corresponding to each relationship vector under the any two detection frames, the relationship between the target objects in the to-be-detected image is determined.

8. The method according to claim 7, characterized in that The determining the relationship between the target objects in the to-be-detected image based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames includes: Based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames, and the confidence of the category information of different objects, the relationship between the target objects in the to-be-detected image is determined.

9. The method according to claim 8, characterized in that The determining the relationship between the target objects in the image to be detected based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames and the confidence of the category information of the different objects includes: Determine the matching cost of each relationship vector under the arbitrary two detection frames based on the semantic distance and the image distance corresponding to each relationship vector under the arbitrary two detection frames and the confidence of the category information of the different objects; For any relationship vector, determine the two target detection boxes with the minimum matching cost corresponding to the relationship vector; Based on the confidence of the object in the target detection box corresponding to each relationship vector and the confidence of the relationship category information corresponding to each relationship vector, the relationship between the target objects in the image to be detected is determined.

10. The method according to claim 1, characterized in that After acquiring the image to be detected, the step of determining the relationship between the target objects in the image to be detected is performed by a neural network; The neural network is trained by calculating the following loss values ​​during training: A first loss value for representing an object prediction loss, a second loss value for representing a relationship prediction loss, and a third loss value for representing a semantic embedding information prediction loss; The first loss value includes the loss between the position information of the detection box corresponding to the predicted object and the position information of the marked detection box, and the loss between the category information of the predicted object and the category information of the marked object; The second loss value includes the loss between the predicted relationship vector and the labeled relationship vector, and the confidence of the predicted relationship category information; The third loss value includes a difference value between semantic embedding information of each object in the object information, and a similarity value between the relationship information and semantic embedding information of the same object in the object information.

11. A relationship detection method, characterized in that: The neural networks used include: feature extraction subnetwork, object prediction subnetwork, relationship prediction subnetwork, and matching network; The feature extraction subnetwork is used to extract features from the image to be detected to obtain an intermediate feature map corresponding to the image to be detected; for each feature point in the intermediate feature map, based on the feature values ​​of other feature points in the intermediate feature map except the feature point, the feature value of the feature point is adjusted to obtain a first feature sequence, and the first feature sequence is transmitted to the object prediction subnetwork and the relationship prediction subnetwork respectively; The object prediction subnetwork is used to perform object prediction based on the first feature sequence, determine the object information in the image to be detected, and transmit the object information to the matching network; and perform relationship prediction on the first feature sequence based on the relationship prediction subnetwork, determine the relationship information of the image to be detected, and transmit the relationship information to the matching network; The matching network is used to match the object information with the relationship information to determine the relationship between the target objects in the image to be detected.

12. The method according to claim 11, characterized in that The relationship prediction subnetwork obtains feature information from the object prediction subnetwork through an information fusion module, and performs relationship prediction in combination with the obtained feature information and the first feature sequence to determine the relationship information of the image to be detected.

13. A relationship detection device, characterized in that: include: An acquisition module, used for acquiring an image to be detected; A first determination module is used to extract features from the image to be detected to obtain an intermediate feature map corresponding to the image to be detected; for each feature point in the intermediate feature map, based on feature values ​​of other feature points in the intermediate feature map except the feature point, the feature value of the feature point is adjusted to obtain a first feature sequence; A second determination module is used to perform object prediction based on the first feature sequence to determine the object information in the image to be detected; and perform relationship prediction based on the first feature sequence to determine the relationship information of the image to be detected; The third determination module is used to match the object information with the relationship information to determine the relationship between the target objects in the image to be detected.

14. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the relationship detection method according to any one of claims 1 to 10 are performed.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the relationship detection method according to any one of claims 1 to 10 are executed.

Images