Machine learning model training method and device, visual relationship detection method and device

Abstract

The present disclosure provides a machine learning model training method and device, a visual relationship detection method and device, and relates to the field of artificial intelligence. The machine learning model training method comprises: processing a sample image by using a first machine learning model to obtain semantic features and spatial features of a target subject and a target object in a target relationship triple, a predicate probability distribution result, and visual features of a target region comprising the target subject and the target object; determining a first loss function according to the predicate probability distribution result; predicting a first semantic vector of the target subject, a first spatial vector of the target subject, a first semantic vector of the target object, and a first spatial vector of the target object by using a second machine learning model; determining a second loss function according to the prediction result; determining a first target loss function according to the first loss function and the second loss function; and training the first machine learning model and the second machine learning model by using the first target loss function.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a machine learning model training method and device, and a visual relationship detection method and device. BACKGROUND

[0002] In the prior art, the visual relationship between the subject and the object is detected by means of semantic features and spatial features of the subject and the object, and visual features of a target region including the subject and the object. That is, the visual relationship prediction problem is converted into a multi-modal feature fusion classification problem. SUMMARY

[0003] The inventor has noticed that the existing visual relationship detection scheme does not utilize internal supervision signals derived from image structured semantic understanding, and thus cannot obtain a stable performance visual relationship detection result.

[0004] Accordingly, the present disclosure provides a machine learning model training scheme that can obtain a stable performance visual relationship detection result.

[0005] According to a first aspect of embodiments of the present disclosure, a machine learning model training method is provided, comprising: processing a sample image by using a first machine learning model to obtain semantic features and spatial features of a target subject in a target relationship triple, semantic features and spatial features of a target object, a predicate probability distribution result, and visual features of a target region including the target subject and the target object; determining a first loss function according to the predicate probability distribution result and a predicate label result; predicting, by using a second machine learning model, a first semantic vector of the target subject according to the spatial features of the target subject, a first spatial vector of the target subject according to the semantic features of the target subject, a first semantic vector of the target object according to the spatial features of the target object, and a first spatial vector of the target object according to the semantic features of the target object; determining a second loss function according to the prediction result; determining a first target loss function according to the first loss function and the second loss function; and training the first machine learning model and the second machine learning model by using the first target loss function.

[0006] In some embodiments, the determining the second loss function according to the prediction result comprises: determining a first sub-loss function according to the first spatial vector of the target subject and a spatial label result of the target subject, and the first spatial vector of the target object and a spatial label result of the target object; determining a second sub-loss function according to the first semantic vector of the target subject and a semantic label result of the target subject, and the first semantic vector of the target object and a semantic label result of the target object; and determining the second loss function according to the first sub-loss function and the second sub-loss function.

[0007] In some embodiments, the first sub-loss function is positively related to a sum of a deviation of the first spatial vector of the target subject and the spatial annotation result of the target subject and a deviation of the first spatial vector of the target object and the spatial annotation result of the target object; and the second sub-loss function is negatively related to a sum of a cross-entropy of the first semantic vector of the target subject and the semantic annotation result of the target subject and a cross-entropy of the first semantic vector of the target object and the semantic annotation result of the target object.

[0008] In some embodiments, the second loss function is a weighted sum of the first sub-loss function and the second sub-loss function.

[0009] In some embodiments, the predicting the first semantic vector of the target subject according to the spatial feature of the target subject comprises: fusing the spatial feature of the target subject and the visual feature to obtain a first fused feature; compressing the first fused feature to obtain a first compressed feature; and processing the first compressed feature by using a multi-layer perception to obtain the first semantic vector of the target subject.

[0010] In some embodiments, the predicting the first spatial vector of the target subject according to the semantic feature of the target subject comprises: performing reconstruction processing on the semantic feature of the target subject and the visual feature to obtain the first spatial vector of the target subject.

[0011] In some embodiments, the predicting the first semantic vector of the target object according to the spatial feature of the target object comprises: fusing the spatial feature of the target object and the visual feature to obtain a second fused feature; compressing the second fused feature to obtain a second compressed feature; and processing the second compressed feature by using a multi-layer perception to obtain the first semantic vector of the target object.

[0012] In some embodiments, the obtaining the first spatial vector of the target object according to the semantic feature of the target object comprises: performing reconstruction processing on the semantic feature of the target object and the visual feature to obtain the first spatial vector of the target object.

[0013] In some embodiments, the first loss function is negatively related to a cross-entropy of the predicate probability distribution result and the predicate annotation result.

[0014] In some embodiments, the first target loss function is a weighted sum of the first loss function and the second loss function.

[0015] In some embodiments, a predicate feature of the target relation triple is determined according to the predicate probability distribution result.

[0016] In some embodiments, the method further comprises: performing multi-modal fusion of the semantic feature and the spatial feature of the target subject, the visual feature, by using the second machine learning model to obtain a first object feature; performing inter-object reconstruction according to the predicate feature and the first object feature by using the second machine learning model to obtain a second semantic vector and a second spatial vector of the target subject; performing multi-modal fusion of the semantic feature and the spatial feature of the target subject, the visual feature, by using the second machine learning model to obtain a second object feature; performing inter-object reconstruction according to the predicate feature and the second object feature by using the second machine learning model to obtain a second semantic vector and a second spatial vector of the target object; determining a third loss function according to the inter-object reconstruction result; determining a second target loss function according to the first loss function, the second loss function and the third loss function; and training the first machine learning model and the second machine learning model by using the second target loss function.

[0017] In some embodiments, determining the third loss function according to the inter-object reconstruction result comprises: determining a third sub-loss function according to the second spatial vector of the target subject and the spatial annotation result of the target subject, the second spatial vector of the target object and the spatial annotation result of the target object; determining a fourth sub-loss function according to the second semantic vector of the target subject and the semantic annotation result of the target subject, the second semantic vector of the target object and the semantic annotation result of the target object; and determining the third loss function according to the third sub-loss function and the fourth sub-loss function.

[0018] In some embodiments, the third sub-loss function is positively correlated with the sum of the deviation of the second spatial vector of the target subject and the spatial annotation result of the target subject and the deviation of the second spatial vector of the target object and the spatial annotation result of the target object; and the fourth sub-loss function is negatively correlated with the sum of the cross-entropy of the second semantic vector of the target subject and the semantic annotation result of the target subject and the cross-entropy of the second semantic vector of the target object and the semantic annotation result of the target object.

[0019] In some embodiments, the third loss function is a weighted sum of the third sub-loss function and the fourth sub-loss function.

[0020] In some embodiments, the second target loss function is a weighted sum of the first loss function, the second loss function and the third loss function.

[0021] In some embodiments, the first machine learning model is used to extract semantic features, spatial features, predicate features and visual features of other relation triples from the sample image in addition to the target relation triple; the second machine learning model is used to perform inter-relation reconstruction according to the semantic features, spatial features, predicate features and visual features of the other relation triples, the spatial features of the target subject and the spatial features of the target object, to obtain a third semantic vector of the target subject, a third semantic vector of the target object, and a predicate probability prediction distribution result; a fourth loss function is determined according to the inter-relation reconstruction result; a third target loss function is determined according to the first loss function, the second loss function, the third loss function and the fourth loss function; and the first machine learning model and the second machine learning model are trained using the third target loss function.

[0022] In some embodiments, the fourth loss function is determined according to the inter-relation reconstruction result includes: a fifth sub-loss function is determined according to the third semantic vector of the target subject and the semantic annotation result, the third semantic vector of the target object and the semantic annotation result; a sixth sub-loss function is determined according to the predicate probability prediction distribution result and the predicate annotation result; and the fourth loss function is determined according to the fifth sub-loss function and the sixth sub-loss function.

[0023] In some embodiments, the fifth sub-loss function is negatively related to the sum of the cross-entropy of the third semantic vector of the target subject and the semantic annotation result of the target subject and the cross-entropy of the third semantic vector of the target object and the semantic annotation result of the target object; and the sixth sub-loss function is negatively related to the cross-entropy of the predicate probability prediction distribution result and the predicate annotation result.

[0024] In some embodiments, the fourth loss function is a weighted sum of the fifth sub-loss function and the sixth sub-loss function.

[0025] In some embodiments, the third target loss function is a weighted sum of the first loss function, the second loss function, the third loss function and the fourth loss function.

[0026] In some embodiments, the processing of the sample image by the first machine learning model includes: processing the sample image by the first machine learning model to obtain semantic features and spatial features of a target subject in a target relation triple, semantic features and spatial features of a target object, and visual features of a target region; and determining the predicate probability distribution result using the semantic features and spatial features of the target subject, the semantic features and spatial features of the target object, and the visual features of the target region.

[0027] In some embodiments, the determining the predicate probability distribution result comprises: fusing the semantic feature and the spatial feature of the target subject to obtain a third fused feature; compressing the visual feature of the target region to obtain a third compressed feature; fusing the third fused feature and the third compressed feature to obtain a fourth fused feature; processing the fourth fused feature by using a multilayer perception to obtain the predicate probability distribution result.

[0028] According to a second aspect of the embodiments of the present disclosure, a machine learning model training apparatus is provided, comprising: a first training module configured to process a sample image by using a first machine learning model to obtain a semantic feature and a spatial feature of a target subject in a target relationship triple, a semantic feature and a spatial feature of a target object, a predicate probability distribution result, a visual feature of a target region comprising the target subject and the target object, and determine a first loss function according to the predicate probability distribution result and a predicate label result; a second training module configured to predict a first semantic vector of the target subject according to the spatial feature of the target subject by using a second machine learning model, predict a first spatial vector of the target subject according to the semantic feature of the target subject, predict a first semantic vector of the target object according to the spatial feature of the target object, predict a first spatial vector of the target object according to the semantic feature of the target object, and determine a second loss function according to the prediction result; and a third training module configured to determine a first target loss function according to the first loss function and the second loss function, and train the first machine learning model and the second machine learning model by using the first target loss function.

[0029] According to a third aspect of the embodiments of the present disclosure, a machine learning model training apparatus is provided, comprising: a memory configured to store instructions; and a processor coupled to the memory, the processor being configured to execute the instructions stored in the memory to implement the method according to any of the above embodiments.

[0030] According to a fourth aspect of the embodiments of the present disclosure, a visual relationship detection method is provided, comprising: inputting a to-be-processed image into a first machine learning model, so that the first machine learning model outputs a predicate probability distribution result associated with a to-be-processed subject and a to-be-processed object in a to-be-processed relationship triple, wherein the first machine learning model is trained by using the machine learning model training method according to any of the above embodiments; and predicting the predicate probability distribution result to obtain a visual relationship between the to-be-processed subject and the to-be-processed object.

[0031] According to a fifth aspect of the embodiments of the present disclosure, a visual relationship detection apparatus is provided, comprising: a first detection module configured to input a to-be-processed image into a first machine learning model, so that the first machine learning model outputs a predicate probability distribution result associated with a to-be-processed subject and a to-be-processed object in a to-be-processed relationship triple, wherein the first machine learning model is trained by using the machine learning model training method in any of the above embodiments; and a second detection module configured to perform prediction on the predicate probability distribution result to obtain a visual relationship between the to-be-processed subject and the to-be-processed object.

[0032] According to a sixth aspect of the embodiments of the present disclosure, a visual relationship detection apparatus is provided, comprising: a memory configured to store instructions; and a processor coupled to the memory, the processor being configured to execute the method according to any of the above embodiments based on the instructions stored in the memory.

[0033] According to a seventh aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, wherein the computer readable storage medium stores computer instructions, and the instructions are executed by a processor to implement the method according to any of the above embodiments.

[0034] Other features and advantages of the present disclosure will be made clear by the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor under the premise of the drawings.

[0036] Figure 1 Flowchart of a machine learning model training method according to an embodiment of the present disclosure;

[0037] Figure 2 Flowchart of a machine learning model training method according to another embodiment of the present disclosure;

[0038] Figure 3 Flowchart of a machine learning model training method according to another embodiment of the present disclosure;

[0039] Figure 4 Structure diagram of a machine learning model training apparatus according to an embodiment of the present disclosure;

[0040] Figure 5 Structure diagram of a machine learning model training apparatus according to another embodiment of the present disclosure;

[0041] Figure 6 Flowchart of a visual relationship detection method according to an embodiment of the present disclosure;

[0042] Figure 7 Structure diagram of a visual relationship detection device according to an embodiment of the present disclosure;

[0043] Figure 8 Structure diagram of a visual relationship detection device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present disclosure will be clearly and completely described with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. The following description of at least one exemplary embodiment is actually only illustrative, but not as any limitation on the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present disclosure.

[0045] Unless otherwise specified, the relative arrangement, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0046] Meanwhile, it should be understood that the sizes of the various parts shown in the drawings are not drawn in accordance with the actual proportional relationship for the convenience of description.

[0047] The techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered as part of the authorized description when appropriate.

[0048] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0049] It should be noted that similar reference numbers and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0050] Figure 1 Flowchart of a machine learning model training method according to an embodiment of the present disclosure. In some embodiments, the following machine learning model training method is performed by a machine learning model training device.

[0051] In step 101, the first machine learning model is used to process the sample image to obtain the semantic features and spatial features of the target subject in the target relation triple, the semantic features and spatial features of the target object, the predicate probability distribution result, and the visual features of the target region including the target subject and the target object.

[0052] A relation triple can be represented as <subject, predicate, object>. For example, an image includes a car and a stop sign, and the car is in front of the stop sign. If the car is taken as the subject and the stop sign is taken as the object, the predicate is front. The corresponding relation triple can be represented as <car, front, stop sign>. The target region is the minimum joint region including the car and the stop sign.

[0053] In some embodiments, the first machine learning model is used to process the sample image to obtain the semantic features and spatial features of the target subject in the target relation triple, the semantic features and spatial features of the target object, and the visual features of the target region.

[0054] For example, the joint region including the subject and the object is expanded in a small range and then input into a pre-trained CNN (Convolutional Neural Network) to extract visual representation. The obtained visual representation can not only capture the visual appearance of the subject and the object, but also capture the relationship of the surrounding environment. A spatial mask module is used to encode a double-layer spatial mask through a sandglass network to obtain an encoding result, and the double-layer spatial mask is composed of binary masks of the subject and the object. Then, the encoding result and the visual representation are spliced to obtain the visual features.

[0055] For the semantic features of the subject and the object, the names of the subject and the object can be first converted into word vectors (for example, using a Word2Vec model), and then encoded by a GRU (Gated Recurrent Unit) to obtain a high-dimensional (for example, 300-dimensional) feature vector. The high-dimensional feature vector is converted into the semantic features of the subject and the semantic features of the object through a fully connected network.

[0056] For the spatial features of the subject and the object, the 4-dimensional coordinate vector of the subject and the 4-dimensional coordinate vector of the object can be converted into the spatial features of the subject and the spatial features of the object through a fully connected layer, respectively.

[0057] Next, the semantic features and spatial features of the target subject, the semantic features and spatial features of the target object, and the visual features of the target region are used to determine the predicate probability distribution result.

[0058] For example, the semantic feature and the spatial feature of the target subject are fused (for example, spliced) to obtain a fused feature. The visual feature of the target region is compressed (for example, compressed by using an AdaptiveAvgPool layer) to obtain a compression result. Then, the fused feature and the compression result are fused (for example, spliced) to obtain a fusion result. Next, the fusion result is processed by using an MLP (Multi-layer Perceptron) as a predicate classifier to obtain a predicate probability distribution result As shown in formula (1).

[0059]

[0060] wherein, is a semantic feature of the subject, is a spatial feature of the subject, is a semantic feature of the object, is a spatial feature of the object, W f is a fusion transformation matrix, f vis is a visual feature, avgpool is a compression function, E cls is a predicate classifier.

[0061] In step 102, a first loss function is determined according to the predicate probability distribution result and the predicate label result.

[0062] In some embodiments, the first loss function is negatively related to the cross entropy of the predicate probability distribution result and the predicate label result. For example, the first loss function L Base As shown in formula (2).

[0063]

[0064] wherein, is a predicate probability distribution result, p is a predicate label result, and x represents the xth dimension in a vector.

[0065] In step 103, a second machine learning model is used to predict a first semantic vector of the target subject according to the spatial feature of the target subject, to predict a first spatial vector of the target subject according to the semantic feature of the target subject, to predict a first semantic vector of the target object according to the spatial feature of the target object, and to predict a first spatial vector of the target object according to the semantic feature of the target object. That is, vector prediction within an object is performed.

[0066] In some embodiments, the spatial feature and the visual feature of the target subject are fused (e.g., stitched) to obtain a first fused feature. The first fused feature is compressed to obtain a first compressed feature. The first compressed feature is processed by a multi-layer perception to obtain a first semantic vector of the target subject.

[0067] In some embodiments, the semantic feature and the visual feature of the target subject are reconstructed to obtain a first spatial vector of the target subject.

[0068] In some embodiments, the spatial feature and the visual feature of the target object are fused (e.g., stitched) to obtain a second fused feature. The second fused feature is compressed to obtain a second compressed feature. The second compressed feature is processed by a multi-layer perception to obtain a first semantic vector of the target object.

[0069] In some embodiments, the semantic feature and the visual feature of the target object are reconstructed to obtain a first spatial vector of the target object.

[0070] For example, the first spatial vector of the target subject and the first spatial vector of the target object are 7x7x5 dimensions, where the image is divided into a 7x7 grid, and 5 dimensions represent 4-dimensional coordinate information plus 1 dimension of confidence

[0071] At step 104, a second loss function is determined according to the prediction result.

[0072] In some embodiments, the step of determining the second loss function includes the following:

[0073] 1) A first sub-loss function is determined according to the first spatial vector of the target subject and the spatial annotation result of the target subject, the first spatial vector of the target object and the spatial annotation result of the target object.

[0074] For example, the first sub-loss function is positively correlated with the sum of the deviation of the first spatial vector of the target subject and the spatial annotation result of the target subject and the deviation of the first spatial vector of the target object and the spatial annotation result of the target object.

[0075] Let the first spatial vector of the target subject be the first spatial vector of the target object be the spatial annotation result of the target subject be b s and the spatial annotation result of the target object be b o Then the first sub-loss function loss1 is shown in equation (3).

[0076]

[0077] For example, the first spatial vector of the target subject is 4-dimensional coordinate information The spatial annotation result of the target subject is 4-dimensional coordinate information (x, y, w, h), and As shown in formula (4).

[0078]

[0079] wherein represents whether the real target appears in the grid cell, the confidence c of the grid cell containing the target value is 1, and i represents the i-th grid cell.

[0080] Correspondingly, Formula (4) can also be used for calculation.

[0081] 2) Determine the second sub-loss function according to the first semantic vector of the target subject and the semantic annotation result of the target subject, and the first semantic vector of the target object and the semantic annotation result of the target object.

[0082] For example, the second sub-loss function is negatively related to the sum of the cross-entropy of the first semantic vector of the target subject and the semantic annotation result of the target subject and the cross-entropy of the first semantic vector of the target object and the semantic annotation result of the target object.

[0083] Let the first semantic vector of the target subject be The semantic annotation result of the target subject is n s , the first semantic vector of the target object is The semantic annotation result of the target object is n o , and the second sub-loss function loss2 is shown in formula (5).

[0084]

[0085] 3) Determine the second loss function according to the first sub-loss function and the second sub-loss function.

[0086] In some embodiments, the second loss function is a weighted sum of the first sub-loss function and the second sub-loss function.

[0087] For example, the second loss function L Intra-object As shown in formula (6).

[0088] L Intra-object = loss1 + loss2 (6)

[0089] In step 105, determine the first target loss function according to the first loss function and the second loss function.

[0090] In some embodiments, the first target loss function is a weighted sum of the first loss function and the second loss function.

[0091] For example, the first target loss function is shown in formula (7).

[0092] L1 = L Base + L Intra-object (7)

[0093] In step 106, the first machine learning model and the second machine learning model are trained by using the first target loss function.

[0094] In the machine learning model training method provided by the above-mentioned embodiments of the present disclosure, on the basis of processing the sample image by the first machine learning model to obtain the relevant features of the target relation triple, the vector prediction within the object is performed by using the second machine learning model, and the first machine learning model and the second machine learning model are trained according to the prediction result. Since the internal supervision signal is used in the training process, the performance stable visual relation detection result can be effectively obtained.

[0095] Figure 2 The flowchart of the machine learning model training method of another embodiment of the present disclosure is shown. In some embodiments, the following machine learning model training method is performed by the machine learning model training device.

[0096] In step 201, the sample image is processed by using the first machine learning model to obtain the semantic features and spatial features of the target subject, the semantic features and spatial features of the target object, the predicate probability distribution result, and the visual features of the target region including the target subject and the target object in the target relation triple.

[0097] In step 202, the first loss function is determined according to the predicate probability distribution result and the predicate annotation result.

[0098] For example, the first loss function is shown in formula (2) above.

[0099] In step 203, the predicate feature of the target relation triple is determined according to the predicate probability distribution result.

[0100] For example, the predicate feature is identified in the form of One Hot Encoding.

[0101] In step 204, the vector prediction within the object is performed.

[0102] That is, the first semantic vector of the target subject is predicted according to the spatial feature of the target subject, the first spatial vector of the target subject is predicted according to the semantic feature of the target subject, the first semantic vector of the target object is predicted according to the spatial feature of the target object, and the first spatial vector of the target object is predicted according to the semantic feature of the target object by using the second machine learning model.

[0103] In step 205, a second loss function is determined according to the prediction result.

[0104] For example, the second loss function is shown in formula (6) as described above.

[0105] In step 206, inter-object vector reconstruction is performed.

[0106] That is, the semantic features and spatial features, visual features of the target subject are fused by the second machine learning model to obtain the first object features. The inter-object reconstruction is performed by the second machine learning model according to the predicate features and the first object features to obtain the second semantic vector and the second spatial vector of the target subject. The semantic features and spatial features, visual features of the target subject are fused by the second machine learning model to obtain the second object features. The inter-object reconstruction is performed by the second machine learning model according to the predicate features and the second object features to obtain the second semantic vector and the second spatial vector of the target subject.

[0107] In step 207, a third loss function is determined according to the inter-object reconstruction result.

[0108] In some embodiments, the step of determining the third loss function includes the following contents:

[0109] 1) A third sub-loss function is determined according to the second spatial vector of the target subject and the spatial annotation result of the target subject, the second spatial vector of the target subject and the spatial annotation result of the target subject.

[0110] For example, the third sub-loss function is positively correlated with the sum of the deviation of the second spatial vector of the target subject and the spatial annotation result of the target subject and the deviation of the second spatial vector of the target subject and the spatial annotation result of the target subject.

[0111] Let the second spatial vector of the target subject be the second spatial vector of the target subject be the spatial annotation result of the target subject be b s and the spatial annotation result of the target subject be b o Then the third sub-loss function loss3 is shown in formula (8).

[0112]

[0113] wherein, and can be calculated by using the above formula (4).

[0114] 2) A fourth sub-loss function is determined according to the second semantic vector of the target subject and the semantic annotation result of the target subject, the second semantic vector of the target subject and the semantic annotation result of the target subject.

[0115] For example, the fourth sub-loss function is negatively related to the sum of the cross entropy of the second semantic vector of the target subject and the semantic annotation result of the target subject and the cross entropy of the second semantic vector of the target object and the semantic annotation result of the target object.

[0116] Let the second semantic vector of the target subject be The semantic annotation result of the target subject is n s Let the second semantic vector of the target object be The semantic annotation result of the target object is n o The fourth sub-loss function loss4 is shown in formula (9).

[0117]

[0118] 3) Determine the third loss function according to the third sub-loss function and the fourth sub-loss function.

[0119] In some embodiments, the third loss function is a weighted sum of the third sub-loss function and the fourth sub-loss function.

[0120] For example, the third loss function L Inter-object is shown in formula (10).

[0121] L Inter-object = loss3 + loss4 (10)

[0122] In step 208, determine the second target loss function according to the first loss function, the second loss function and the third loss function.

[0123] In some embodiments, the second target loss function is a weighted sum of the first loss function, the second loss function and the third loss function.

[0124] For example, the second target loss function is shown in formula (11).

[0125] L2 = L Base + L Intra-object + L Inter-object (11)

[0126] In step 209, train the first machine learning model and the second machine learning model using the second target loss function.

[0127] In the machine learning model training method provided by the above-mentioned embodiments of the present disclosure, on the basis of processing a sample image by a first machine learning model to obtain relevant features of a target relation triple, vector prediction within an object and vector prediction between objects are performed by using a second machine learning model, and the first machine learning model and the second machine learning model are trained according to the prediction results. Since the multi-granularity internal supervision signal is used in the training process, the performance-stable visual relation detection result can be effectively obtained.

[0128] Figure 3 The flowchart of the machine learning model training method of another embodiment of the present disclosure is shown. In some embodiments, the following machine learning model training method is performed by a machine learning model training device.

[0129] In step 301, a sample image is processed by using a first machine learning model to obtain semantic features and spatial features of a target subject, semantic features and spatial features of a target object, a predicate probability distribution result, and visual features of a target region including the target subject and the target object in a target relation triple.

[0130] In step 302, a first loss function is determined according to the predicate probability distribution result and a predicate annotation result.

[0131] For example, the first loss function is shown in the above-mentioned formula (2).

[0132] In step 303, a predicate feature of the target relation triple is determined according to the predicate probability distribution result.

[0133] For example, the predicate feature is identified in the form of hot encoding (One Hot Encoding).

[0134] In step 304, vector prediction within an object is performed.

[0135] That is, a first semantic vector of the target subject is predicted according to the spatial features of the target subject, a first spatial vector of the target subject is predicted according to the semantic features of the target subject, a first semantic vector of the target object is predicted according to the spatial features of the target object, and a first spatial vector of the target object is predicted according to the semantic features of the target object by using the second machine learning model.

[0136] In step 305, a second loss function is determined according to the prediction result.

[0137] For example, the second loss function is shown in the above-mentioned formula (6).

[0138] In step 306, vector reconstruction between objects is performed.

[0139] That is, the semantic features and spatial features, visual features of the target subject are fused by the second machine learning model to obtain the first object features. The inter-object reconstruction is performed by the second machine learning model according to the predicate features and the first object features to obtain the second semantic vector and the second spatial vector of the target subject. The semantic features and spatial features, visual features of the target subject are fused by the second machine learning model to obtain the second object features. The inter-object reconstruction is performed by the second machine learning model according to the predicate features and the second object features to obtain the second semantic vector and the second spatial vector of the target subject.

[0140] In step 307, a third loss function is determined according to the inter-object reconstruction result.

[0141] For example, the third loss function L Inter-object As shown in formula (10) above.

[0142] In step 308, vector reconstruction between relations is performed.

[0143] First, the semantic features, spatial features, predicate features and visual features of the other relation triplets except the target relation triplet are extracted from the sample image by the first machine learning model.

[0144] For example, the image includes a car and a parking sign, and the car is in front of the parking sign, so the corresponding relation triplet is <car, front, parking sign>. In addition, the image also includes objects such as a house and the ground, and the corresponding relation triplets can include <car, on, ground>, <parking sign, front, house>, etc. By using these relation triplets, a more complete scene graph can be constructed to facilitate the model to understand the scene.

[0145] Next, the inter-relation reconstruction is performed by the second machine learning model according to the semantic features, spatial features, predicate features and visual features of the other relation triplets, the spatial features of the target subject and the spatial features of the target object to obtain the third semantic vector of the target subject, the third semantic vector of the target object, and the predicate probability prediction distribution result.

[0146] In step 309, a fourth loss function is determined according to the inter-relation reconstruction result.

[0147] In some embodiments, the step of determining the fourth loss function includes the following contents.

[0148] 1) A fifth sub-loss function is determined according to the third semantic vector of the target subject and the semantic annotation result, and the third semantic vector of the target object and the semantic annotation result.

[0149] For example, the fifth sub-loss function is negatively related to the cross entropy of the third semantic vector of the target subject and the semantic annotation result of the target subject and the cross entropy of the third semantic vector of the target object and the semantic annotation result of the target object.

[0150] Suppose that the sample image includes M relationship triplets, and the target relationship triplet is the i-th relationship triplet R i In R i , the third semantic vector of the target subject is The semantic annotation result of the target subject is n si The third semantic vector of the target object is The semantic annotation result of the target object is n oi The fifth sub-loss function loss4 is shown in formula (12).

[0151]

[0152] 2) Determine the sixth sub-loss function according to the predicate probability prediction distribution result and the predicate annotation result.

[0153] For example, the sixth sub-loss function is negatively related to the cross entropy of the predicate probability prediction distribution result and the predicate label result.

[0154] Suppose that the sample image includes M relationship triplets, and the target relationship triplet is the i-th relationship triplet R i In R i , the predicate probability prediction distribution result is The predicate label result is p si The sixth sub-loss function loss6 is shown in formula (13).

[0155]

[0156] 3) Determine the fourth loss function according to the fifth sub-loss function and the sixth sub-loss function.

[0157] In some embodiments, the fourth loss function is a weighted sum of the fifth sub-loss function and the sixth sub-loss function.

[0158] For example, the fourth loss function L Inter-relation is shown in formula (14).

[0159] L Inter-relation = loss5 + loss6 (14)

[0160] In step 310, determine the third target loss function according to the first loss function, the second loss function, the third loss function, and the fourth loss function.

[0161] In some embodiments, the third target loss function is a weighted sum of the first loss function, the second loss function, the third loss function, and the fourth loss function.

[0162] For example, the third target loss function is shown in formula (15).

[0163] L3 = L Base + L Intra-object + L Inter-object + L Inter-relation (15)

[0164] In step 311, the first machine learning model and the second machine learning model are trained using the fourth target loss function.

[0165] In the machine learning model training method provided by the above-mentioned embodiments of the present disclosure, on the basis of processing the sample image by the first machine learning model to obtain the relevant features of the target relation triple, the second machine learning model is used to perform intra-object vector prediction, inter-object vector prediction, and inter-relation vector prediction, and the first machine learning model and the second machine learning model are trained according to the prediction results. Since the multi-granularity internal supervision signal is used in the training process, the performance-stable visual relation detection result can be effectively obtained.

[0166] Figure 4 The structure diagram of the machine learning model training device of an embodiment of the present disclosure is shown in FIG. 4. As shown in FIG. 4, the machine learning model training device includes a first training module 41, a second training module 42, and a third training module 43. Figure 4

[0167] The first training module 41 is configured to process the sample image by the first machine learning model to obtain the semantic features and spatial features of the target subject, the semantic features and spatial features of the target object, the predicate probability distribution result, the visual features of the target region including the target subject and the target object in the target relation triple, and determine the first loss function according to the predicate probability distribution result and the predicate annotation result.

[0168] In some embodiments, the first training module 41 processes the sample image by the first machine learning model to obtain the semantic features and spatial features of the target subject, the semantic features and spatial features of the target object, and the visual features of the target region in the target relation triple. The predicate probability distribution result is determined using the semantic features and spatial features of the target subject, the semantic features and spatial features of the target object, and the visual features of the target region.

[0169] ​In some embodiments, the first training module 41 fuses the semantic feature and the spatial feature of the target subject to obtain a third fused feature, compresses the visual feature of the target region to obtain a third compressed feature, fuses the third fused feature and the third compressed feature to obtain a fourth fused feature, and processes the fourth fused feature by using a multi-layer perception to obtain a predicate probability distribution result.

[0170] In some embodiments, the first loss function is negatively related to the cross entropy of the predicate probability distribution result and a predicate label result. For example, the first loss function is shown in the above formula (2).

[0171] The second training module 42 is configured to predict a first semantic vector of the target subject according to the spatial feature of the target subject, predict a first spatial vector of the target subject according to the semantic feature of the target subject, predict a first semantic vector of the target object according to the spatial feature of the target object, predict a first spatial vector of the target object according to the semantic feature of the target object, and determine a second loss function according to the prediction results.

[0172] In some embodiments, the second training module 42 fuses the spatial feature and the visual feature of the target subject to obtain a first fused feature, compresses the first fused feature to obtain a first compressed feature, and processes the first compressed feature by using a multi-layer perception to obtain the first semantic vector of the target subject.

[0173] In some embodiments, the second training module 42 reconstructs the semantic feature and the visual feature of the target subject to obtain the first spatial vector of the target subject.

[0174] In some embodiments, the second training module 42 fuses the spatial feature and the visual feature of the target object to obtain a second fused feature, compresses the second fused feature to obtain a second compressed feature, and processes the second compressed feature by using a multi-layer perception to obtain the first semantic vector of the target object.

[0175] In some embodiments, the second training module 42 reconstructs the semantic feature and the visual feature of the target object to obtain the first spatial vector of the target object.

[0176] In some embodiments, the second training module 42 determines a first sub-loss function according to the first spatial vector of the target subject and a spatial label result of the target subject, and according to the first spatial vector of the target object and a spatial label result of the target object; determines a second sub-loss function according to the first semantic vector of the target subject and a semantic label result of the target subject, and according to the first semantic vector of the target object and a semantic label result of the target object; and determines the second loss function according to the first sub-loss function and the second sub-loss function.

[0177] In some embodiments, the first sub-loss function is positively related to a sum of a deviation of the first spatial vector of the target subject and the spatial annotation result of the target subject and a deviation of the first spatial vector of the target object and the spatial annotation result of the target object. The second sub-loss function is negatively related to a sum of a cross-entropy of the first semantic vector of the target subject and the semantic annotation result of the target subject and a cross-entropy of the first semantic vector of the target object and the semantic annotation result of the target object.

[0178] In some embodiments, the second loss function is a weighted sum of the first sub-loss function and the second sub-loss function. For example, the second loss function is shown in the above formula (6).

[0179] The third training module 43 is configured to determine a first target loss function according to the first loss function and the second loss function, and train the first machine learning model and the second machine learning model by using the first target loss function.

[0180] In some embodiments, the first target loss function is a weighted sum of the first loss function and the second loss function. For example, the first target loss function is shown in the above formula (7).

[0181] In some embodiments, the first training module 41 determines the predicate feature of the target relation triple according to the predicate probability distribution result.

[0182] In some embodiments, the second training module 42 performs multi-modal fusion on the semantic feature and the spatial feature and the visual feature of the target subject by using the second machine learning model to obtain a first object feature, performs inter-object reconstruction according to the predicate feature and the first object feature by using the second machine learning model to obtain a second semantic vector and a second spatial vector of the target subject, performs multi-modal fusion on the semantic feature and the spatial feature and the visual feature of the target subject by using the second machine learning model to obtain a second object feature, performs inter-object reconstruction according to the predicate feature and the second object feature by using the second machine learning model to obtain a second semantic vector and a second spatial vector of the target object, and determines a third loss function according to the inter-object reconstruction result.

[0183] In some embodiments, the second training module 42 determines a third sub-loss function according to the second spatial vector of the target subject and the spatial annotation result of the target subject, and the second spatial vector of the target object and the spatial annotation result of the target object, determines a fourth sub-loss function according to the second semantic vector of the target subject and the semantic annotation result of the target subject, and the second semantic vector of the target object and the semantic annotation result of the target object, and determines the third loss function according to the third sub-loss function and the fourth sub-loss function.

[0184] In some embodiments, the third sub-loss function is positively related to a sum of a deviation of the second spatial vector of the target subject and the spatial annotation result of the target subject and a deviation of the second spatial vector of the target object and the spatial annotation result of the target object. The fourth sub-loss function is negatively related to a sum of a cross entropy of the second semantic vector of the target subject and the semantic annotation result of the target subject and a cross entropy of the second semantic vector of the target object and the semantic annotation result of the target object.

[0185] In some embodiments, the third loss function is a weighted sum of the third sub-loss function and the fourth sub-loss function. For example, the third loss function is shown in the above formula (10).

[0186] In some embodiments, the third training module 43 determines a second target loss function according to the first loss function, the second loss function and the third loss function, and trains the first machine learning model and the second machine learning model by using the second target loss function.

[0187] In some embodiments, the second target loss function is a weighted sum of the first loss function, the second loss function and the third loss function. For example, the second target loss function is shown in the above formula (11).

[0188] In some embodiments, the first training module 41 extracts semantic features, spatial features, predicate features and visual features of other relation triplets except the target relation triplet from the sample image by using the first machine learning model.

[0189] The second training module 42 performs inter-relation reconstruction according to the semantic features, spatial features, predicate features and visual features of the other relation triplets, the spatial features of the target subject and the spatial features of the target object by using the second machine learning model, to obtain a third semantic vector of the target subject, a third semantic vector of the target object and a predicate probability prediction distribution result, and determines a fourth loss function according to the inter-relation reconstruction result.

[0190] In some embodiments, the second training module 42 determines a fifth sub-loss function according to the third semantic vector of the target subject and the semantic annotation result of the target subject and the third semantic vector of the target object and the semantic annotation result of the target object, determines a sixth sub-loss function according to the predicate probability prediction distribution result and the predicate annotation result, and determines the fourth loss function according to the fifth sub-loss function and the sixth sub-loss function.

[0191] In some embodiments, the fifth sub-loss function is negatively related to a sum of a cross entropy of the third semantic vector of the target subject and the semantic annotation result of the target subject and a cross entropy of the third semantic vector of the target object and the semantic annotation result of the target object. The sixth sub-loss function is negatively related to a cross entropy of the predicate probability prediction distribution result and the predicate annotation result.

[0192] In some embodiments, the fourth loss function is a weighted sum of the fifth and sixth sub-loss functions. For example, the fourth loss function is shown in Equation (14) above.

[0193] In some embodiments, the third training module 43 determines a third target loss function based on the first loss function, the second loss function, the third loss function, and the fourth loss function, and uses the third target loss function to train the first machine learning model and the second machine learning model.

[0194] In some embodiments, the third objective loss function is a weighted sum of the first, second, third, and fourth loss functions. For example, the third objective loss function is shown in formula (15) above.

[0195] Figure 5 This is a schematic diagram of the structure of a machine learning model training apparatus according to another embodiment of this disclosure. Figure 5 As shown, the machine learning model training device includes a memory 51 and a processor 52.

[0196] Memory 51 is used to store instructions, and processor 52 is coupled to memory 51. Processor 52 is configured to execute instructions based on memory storage, as shown in the example below. Figures 1-3 The method involved in any of the embodiments.

[0197] like Figure 5 As shown, the machine learning model training device also includes a communication interface 53 for exchanging information with other devices. Additionally, the machine learning model training device includes a bus 54, through which the processor 52, communication interface 53, and memory 51 communicate with each other.

[0198] The memory 51 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. The memory 51 may also be a memory array. The memory 51 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0199] Furthermore, processor 52 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0200] This disclosure also relates to a computer-readable storage medium storing computer instructions that, when executed by a processor, implement... Figures 1-3 The method involved in any of the embodiments.

[0201] Figure 6A flowchart of a visual relation detection method according to an embodiment of the present disclosure is shown. In some embodiments, the visual relation detection method is performed by a visual relation detection apparatus.

[0202] In step 601, the image to be processed is input into the first machine learning model, so that the first machine learning model outputs a predicate probability distribution result associated with the subject to be processed and the object to be processed in the relation triple to be processed. The first machine learning model is trained by the machine learning model training method according to any one of the embodiments. Figures 1-3 The machine learning model training method according to any one of the embodiments is used to train the first machine learning model.

[0203] In step 602, the predicate probability distribution result is predicted to obtain the visual relation between the subject to be processed and the object to be processed.

[0204] It should be noted that, since the multi-granularity internal supervision signal is used in the training process, the trained first machine learning model can output stable visual relation detection results.

[0205] Figure 7 A structure diagram of a visual relation detection apparatus according to an embodiment of the present disclosure is shown. As shown in Figure 7 , the visual relation detection apparatus includes a first detection module 71 and a second detection module 72.

[0206] The first detection module 71 is configured to input the image to be processed into the first machine learning model, so that the first machine learning model outputs a predicate probability distribution result associated with the subject to be processed and the object to be processed in the relation triple to be processed. The first machine learning model is trained by the machine learning model training method according to any one of the embodiments. Figures 1-3 The machine learning model training method according to any one of the embodiments is used to train the first machine learning model.

[0207] The second detection module 72 is configured to predict the predicate probability distribution result to obtain the visual relation between the subject to be processed and the object to be processed.

[0208] Figure 8 A structure diagram of a visual relation detection apparatus according to another embodiment of the present disclosure is shown. As shown in Figure 8 , the visual relation detection apparatus includes a memory 81, a processor 82, a communication interface 83 and a bus 84. Figure 8 Different from Figure 5 , in the embodiment shown in Figure 8 , the processor 82 is configured to perform the method according to any one of the embodiments based on the instructions stored in the memory. Figure 6 The machine learning model training method according to any one of the embodiments is used to train the first machine learning model.

[0209] The present disclosure also relates to a computer readable storage medium, wherein the computer readable storage medium stores computer instructions, and the instructions are executed by a processor to implement the method according to any one of the embodiments. Figure 6Any one of the embodiments relates to a method.

[0210] In some embodiments, the functional unit modules described above can be implemented as a general purpose processor, a programmable logic controller (PLC), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any appropriate combination thereof, for performing the functions described in the present disclosure.

[0211] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program instructing relevant hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0212] The description of the present disclosure is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the present disclosure to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are chosen and described in order to best explain the principles of the present disclosure and its practical application, and to enable others skilled in the art to understand the present disclosure in order to design various embodiments with various modifications for specific use.

Claims

1. A method for training a machine learning model, comprising: The sample images are processed using the first machine learning model to obtain the semantic and spatial features of the target subject, the semantic and spatial features of the target object, the predicate probability distribution results, and the visual features of the target region including the target subject and the target object in the target relation triplet. The first loss function is determined based on the predicate probability distribution results and the predicate annotation results; The second machine learning model is used to predict the first semantic vector of the target subject based on the spatial features of the target subject, predict the first spatial vector of the target subject based on the semantic features of the target subject, predict the first semantic vector of the target object based on the spatial features of the target object, and predict the first spatial vector of the target object based on the semantic features of the target object. Determine the second loss function based on the prediction results; The first target loss function is determined based on the first loss function and the second loss function; The first machine learning model and the second machine learning model are trained using the first objective loss function; The step of determining the second loss function based on the prediction results includes: The first sub-loss function is determined based on the first spatial vector of the target subject and the spatial annotation result of the target subject, and the first spatial vector of the target object and the spatial annotation result of the target object; The second sub-loss function is determined based on the first semantic vector of the target subject and the semantic annotation result of the target subject, and the first semantic vector of the target object and the semantic annotation result of the target object; The second loss function is determined based on the first sub-loss function and the second sub-loss function.

2. The method according to claim 1, wherein, The deviation of the first sub-loss function from the first spatial vector of the target subject and the spatial annotation result of the target subject is positively correlated with the sum of the deviations of the first spatial vector of the target object and the spatial annotation result of the target object; The cross-entropy between the second sub-loss function and the first semantic vector of the target subject and the semantic annotation result of the target subject is negatively correlated with the sum of the cross-entropy between the first semantic vector of the target object and the semantic annotation result of the target object.

3. The method according to claim 1, wherein, The second loss function is a weighted sum of the first sub-loss function and the second sub-loss function.

4. The method according to claim 1, wherein, The step of predicting the first semantic vector of the target subject based on the spatial features of the target subject includes: The spatial features and visual features of the target subject are fused to obtain a first fused feature; The first fused feature is compressed to obtain the first compressed feature; The first compressed feature is processed using a multilayer perceptron to obtain the first semantic vector of the target subject.

5. The method according to claim 1, wherein, The step of predicting the first spatial vector of the target subject based on the semantic features of the target subject includes: The semantic features and visual features of the target subject are used for reconstruction to obtain the first spatial vector of the target subject.

6. The method according to claim 1, wherein, The step of predicting the first semantic vector of the target object based on the spatial features of the target object includes: The spatial features and visual features of the target object are fused to obtain a second fused feature; The second fused feature is compressed to obtain the second compressed feature; The second compressed feature is processed using a multilayer perceptron to obtain the first semantic vector of the target object.

7. The method according to claim 1, wherein, The step of obtaining the first spatial vector of the target object based on the semantic features of the target object includes: The semantic features and visual features of the target object are used for reconstruction to obtain the first spatial vector of the target object.

8. The method according to claim 1, wherein, The first loss function is negatively correlated with the cross-entropy of the predicate probability distribution result and the predicate labeling result.

9. The method according to claim 1, wherein, The first target loss function is a weighted sum of the first loss function and the second loss function.

10. The method according to claim 1, further comprising: The predicate features of the target relation triple are determined based on the predicate probability distribution results.

11. The method of claim 10, further comprising: The second machine learning model is used to perform multimodal fusion of the semantic features, spatial features, and visual features of the target object to obtain the first object features; The second machine learning model is used to reconstruct objects based on the predicate features and the first object features to obtain the second semantic vector and the second spatial vector of the target subject; The second machine learning model is used to perform multimodal fusion of the semantic features, spatial features, and visual features of the target subject to obtain the second object features; The second machine learning model is used to reconstruct objects based on the predicate features and the second object features to obtain the second semantic vector and the second spatial vector of the target object; The third loss function is determined based on the reconstruction results between objects; The second target loss function is determined based on the first loss function, the second loss function, and the third loss function; The first machine learning model and the second machine learning model are trained using the second objective loss function.

12. The method according to claim 11, wherein, The process of determining the third loss function based on the reconstruction results between objects includes: The third sub-loss function is determined based on the second spatial vector of the target subject and the spatial annotation result of the target subject, and the second spatial vector of the target object and the spatial annotation result of the target object; The fourth sub-loss function is determined based on the second semantic vector of the target subject and the semantic annotation result of the target subject, and the second semantic vector of the target object and the semantic annotation result of the target object; The third loss function is determined based on the third sub-loss function and the fourth sub-loss function.

13. The method according to claim 12, wherein, The deviation of the third sub-loss function from the second spatial vector of the target subject and the spatial annotation result of the target subject is positively correlated with the sum of the deviations of the second spatial vector of the target object and the spatial annotation result of the target object; The cross-entropy between the fourth sub-loss function and the second semantic vector of the target subject and the semantic annotation result of the target subject is negatively correlated with the sum of the cross-entropy between the second semantic vector of the target object and the semantic annotation result of the target object.

14. The method according to claim 12, wherein, The third loss function is a weighted sum of the third sub-loss function and the fourth sub-loss function.

15. The method according to claim 11, wherein, The second target loss function is a weighted sum of the first loss function, the second loss function, and the third loss function.

16. The method of claim 11, further comprising: The first machine learning model is used to extract semantic features, spatial features, predicate features, and visual features of other relation triples besides the target relation triple from the sample image; The second machine learning model is used to reconstruct relations based on the semantic features, spatial features, predicate features, and visual features of the other relation triples, the spatial features of the target subject, and the spatial features of the target object, so as to obtain the third semantic vector of the target subject, the third semantic vector of the target object, and the predicate probability prediction distribution results. The fourth loss function is determined based on the reconstruction results between relationships; The third target loss function is determined based on the first loss function, the second loss function, the third loss function, and the fourth loss function; The first machine learning model and the second machine learning model are trained using the third objective loss function.

17. The method according to claim 16, wherein, The determination of the fourth loss function based on the reconstruction results between relations includes: The fifth sub-loss function is determined based on the third semantic vector and semantic annotation results of the target subject and the third semantic vector and semantic annotation results of the target object; The sixth sub-loss function is determined based on the predicate probability prediction distribution results and the predicate annotation results; The fourth loss function is determined based on the fifth sub-loss function and the sixth sub-loss function.

18. The method according to claim 17, wherein, The cross-entropy between the fifth sub-loss function and the third semantic vector of the target subject and the semantic annotation result of the target subject is negatively correlated with the sum of the cross-entropy between the third semantic vector of the target object and the semantic annotation result of the target object; The sixth sub-loss function is negatively correlated with the cross-entropy of the predicate probability prediction distribution and the predicate labeling results.

19. The method according to claim 18, wherein, The fourth loss function is a weighted sum of the fifth and sixth sub-loss functions.

20. The method of claim 16, wherein, The third objective loss function is a weighted sum of the first loss function, the second loss function, the third loss function, and the fourth loss function.

21. The method according to any one of claims 1-20, wherein, The process of processing the sample images using the first machine learning model includes: The sample images are processed using a first machine learning model to obtain the semantic and spatial features of the target subject, the semantic and spatial features of the target object, and the visual features of the target region in the target relation triplet. The predicate probability distribution result is determined by utilizing the semantic and spatial features of the target subject, the semantic and spatial features of the target object, and the visual features of the target region.

22. The method according to claim 21, wherein, The determination of the predicate probability distribution includes: The semantic and spatial features of the target entity are fused to obtain a third fused feature; The visual features of the target region are compressed to obtain a third compressed feature; The third fusion feature and the third compression feature are fused together to obtain the fourth fusion feature; The fourth fusion feature is processed using a multilayer perceptron to obtain the predicate probability distribution result.

23. A machine learning model training device, comprising: The first training module is configured to process sample images using a first machine learning model to obtain semantic and spatial features of the target subject, semantic and spatial features of the target object, predicate probability distribution results, and visual features of the target region including the target subject and the target object in the target relation triplet, and to determine a first loss function based on the predicate probability distribution results and the predicate annotation results. The second training module is configured to use a second machine learning model to predict a first semantic vector of the target subject based on the spatial features of the target subject, predict a first spatial vector of the target subject based on the semantic features of the target subject, predict a first semantic vector of the target object based on the spatial features of the target object, predict a first spatial vector of the target object based on the semantic features of the target object, and determine a second loss function based on the prediction results. Specifically, a first sub-loss function is determined based on the first spatial vector of the target subject and the spatial annotation result of the target subject, and the first spatial vector of the target object and the spatial annotation result of the target object. A second sub-loss function is determined based on the first sub-loss function and the second sub-loss function. The third training module is configured to determine a first target loss function based on the first loss function and the second loss function, and to train the first machine learning model and the second machine learning model using the first target loss function.

24. A machine learning model training device, comprising: The memory is configured to store instructions; A processor, coupled to a memory, configured to implement the method as described in any one of claims 1-22 based on memory-stored instruction execution.

25. A visual relationship detection method, comprising: The image to be processed is input into the first machine learning model so that the first machine learning model outputs the predicate probability distribution results associated with the subject and object to be processed in the triplet of the relation to be processed, wherein the first machine learning model is trained using the machine learning model training method of any one of claims 1-22; The predicate probability distribution results are predicted to obtain the visual relationship between the subject to be processed and the object to be processed.

26. A visual relationship detection device, comprising: The first detection module is configured to input the image to be processed into the first machine learning model, so that the first machine learning model outputs the predicate probability distribution results associated with the subject and object to be processed in the triplet of the relation to be processed, wherein the first machine learning model is trained using the machine learning model training method of any one of claims 1-22; The second detection module is configured to predict the predicate probability distribution results to obtain the visual relationship between the subject to be processed and the object to be processed.

27. A visual relationship detection device, comprising: The memory is configured to store instructions; A processor, coupled to a memory, is configured to implement the method of claim 25 based on memory-stored instruction execution.

28. A computer-readable storage medium, wherein, A computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-22 and 25.

Images