Machine learning model training method and device, scene graph generation method and device

Abstract

The present disclosure provides a machine learning model training method and device, a scene graph generation method and device, and relates to the field of artificial intelligence. The machine learning model training method comprises: extracting first target feature information in a sample image; performing information transmission processing on the first target feature information and a preset reweighting coefficient by using a first machine learning model to obtain second target feature information of the sample image; generating a first scene graph by using the second target feature information of the sample image; generating a first loss function by using the first scene graph and a label result of the sample image; and training the first machine learning model by using the first loss function.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and in particular to a machine learning model training method and apparatus, and a scene graph generation method and apparatus. Background Technology

[0002] Scene graph generation is a technique that uses a graph to represent objects and the relationships between them in an image or video. A scene graph consists of multiple triples. Each triple consists of two nodes (representing objects) and one edge (representing a relationship). For example, if an image includes a person and a bicycle, a possible relationship is that the person is riding a bicycle.

[0003] Existing scene graph generation methods mainly include scene graph methods that model contextual information. These methods use contextual information passing based on graph convolutional networks to improve the accuracy of object and relationship recognition. Summary of the Invention

[0004] The inventors noted that graph convolution-based context information transfer methods tend to produce a greater correlation response for targets that are spatially close. For example, targets A and B may be semantically different, but spatially close, making it impossible to accurately distinguish between them, thus leading to target representation confusion.

[0005] Accordingly, this disclosure provides a machine learning model training scheme that effectively solves the problem of target representation confusion by denoising image signals.

[0006] According to a first aspect of the present disclosure, a machine learning model training method is provided, comprising: extracting first target feature information from a sample image; using a first machine learning model to perform information transfer processing on the first target feature information and preset reweighting coefficients to obtain second target feature information of the sample image; generating a first scene map using the second target feature information of the sample image; generating a first loss function using the annotation results of the first scene map and the sample image; and training the first machine learning model using the first loss function.

[0007] In some embodiments, the information transmission processing of the first target feature information and the preset reweighting coefficients using the first machine learning model includes: generating a target correlation matrix using the first target feature information and the reweighting coefficients; and generating the second target feature information using the first target feature information and the target correlation matrix.

[0008] In some embodiments, the target association matrix is ​​associated with the Hadamard product of the reweighted coefficients and the attention score of the first target feature information; the second target feature information is associated with the sum of the first target feature information, the product of the target association matrix and the first target feature information, and preset target feature information.

[0009] In some embodiments, the target correlation matrix A (k+1) for:

[0010] A (k+1) =Normalize(Ω) (k) ⊙H(Y (k) ))

[0011] Where Normalize is the normalization function, Ω (k) Y is the reweighting coefficient. (k) The first target feature information is H, where H is the attention scoring function and ⊙ is the Hadamard multiplication operator; the second target feature information is Y. (k+1) for:

[0012] Y (k+1) =AF1(a(Y (k) +A (k+1) Y (k) +Y (0) ))

[0013] Where AF1 is the activation function, a is the coefficient, and Y is the activation function. (0) The preset target feature information.

[0014] In some embodiments, the method further includes: processing the second target feature information using the second machine learning model to generate a relationship prediction matrix, wherein the relationship prediction matrix includes relationship prediction features between every two targets in the sample image; determining the maximum rank of the relationship prediction matrix; determining a cross-entropy loss function for the relationship prediction results of each target in the sample image based on the relationship prediction matrix; generating a second loss function based on the cross-entropy loss function and the maximum rank of the relationship prediction matrix; and training the first machine learning model and the second machine learning model using the second loss function.

[0015] In some embodiments, the second feature information includes the target features of each target in the sample image, and the joint features of every two features in the sample image.

[0016] In some embodiments, the relationship prediction feature between the i-th target and the j-th target in the sample image is associated with the target feature of the i-th target, the target feature of the j-th target, and the joint feature of the i-th target and the j-th target, where 1≤i,j≤M, and i≠j.

[0017] In some embodiments, the relationship prediction feature P between the i-th target and the j-th target is... ij for:

[0018] P ij =AF2(W r (y i *y j *u ij )+f ij )

[0019] Where AF2 is the activation function, W r Let y be the relation prediction function. i Let y be the target feature of the i-th target. j Let u be the target feature of the j-th target. ij f is the joint feature of the i-th target and the j-th target. ij Let represent the probability distribution of the relationship category between the i-th target and the j-th target, and * represent the feature fusion operator.

[0020] In some embodiments, the feature fusion operator * satisfies the following condition:

[0021] x*y=AF3(W x x+W y y)-(W x xW y y)⊙(W x xW y y)

[0022] Where x is the first feature vector, y is the second feature vector, AF3 is the activation function, and W... x Let W be the projection dimension of x. y Let y be the projection dimension, and ⊙ be the Hadamard multiplication operator.

[0023] In some embodiments, determining the maximum rank of the relationship prediction matrix includes: rank L of the relationship prediction matrix. 2,1 The regularization term of the norm serves as the maximum rank of the relation prediction matrix.

[0024] In some embodiments, the L of the relationship prediction matrix 2,1 Regular terms of the norm ||P|| 2,1 for

[0025]

[0026] Where N and R are the dimensions of the relationship prediction matrix.

[0027] In some embodiments, the second loss function is the difference between the cross-entropy loss function and the maximum rank of the relation prediction matrix.

[0028] In some embodiments, the second loss function L e for

[0029]

[0030] in, Let P be the cross-entropy loss function, l be the weight coefficient, B be the number of groups in the mini-batch, and P be the weight coefficient. b Let M be the relation prediction matrix for group b. B Let N be the prediction vector for the mini-batch. b Predict the features for the b-th group of relationships.

[0031] According to a second aspect of the present disclosure, a machine learning model training apparatus is provided, comprising: a first training module configured to extract first target feature information from a sample image; a second training module configured to perform information transfer processing on the first target feature information and preset reweighting coefficients using a first machine learning model to obtain second target feature information of the sample image; a third training module configured to generate a first scene map using the second target feature information of the sample image; and a fourth training module configured to generate a first loss function using the annotation results of the first scene map and the sample image, and train the first machine learning model using the first loss function.

[0032] According to a third aspect of the present disclosure, a machine learning model training apparatus is provided, comprising: a memory configured to store instructions; and a processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the method as described in any of the above embodiments.

[0033] According to a fourth aspect of the present disclosure, a scene graph generation method is provided, comprising: extracting third target feature information from an image to be processed; performing information transfer processing on the third target feature information and preset reweighting coefficients using a first machine learning model to obtain fourth target feature information of the image to be processed, wherein the first machine learning model is trained using the machine learning model training method described in any of the above embodiments; and generating a second scene graph using the fourth target feature information.

[0034] In some embodiments, the second machine learning model is used to process the fourth target feature information to generate a relationship prediction matrix, wherein the relationship prediction matrix includes relationship prediction features between every two targets in the image to be processed, and the second machine learning model is trained using the machine learning model training method described in any of the above embodiments; a third scene map is generated using the fourth target feature information and the relationship prediction matrix.

[0035] According to a fifth aspect of the present disclosure, a scene graph generation apparatus is provided, comprising: a first processing module configured to extract third target feature information from an image to be processed; a second processing module configured to perform information transfer processing on the third target feature information and preset reweighting coefficients using a first machine learning model to obtain fourth target feature information of the image to be processed, wherein the first machine learning model is trained using the machine learning model training method described in any of the above embodiments; and a third processing module configured to generate a second scene graph using the fourth target feature information.

[0036] According to a sixth aspect of the present disclosure, a scene graph generation apparatus is provided, comprising: a memory configured to store instructions; and a processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the method as described in any of the above embodiments.

[0037] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any of the above embodiments.

[0038] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of an image to be processed according to an embodiment of the present disclosure;

[0041] Figure 2 This is a flowchart illustrating a machine learning model training method according to an embodiment of the present disclosure;

[0042] Figure 3This is a flowchart illustrating a machine learning model training method according to another embodiment of the present disclosure;

[0043] Figure 4 This is a schematic diagram of the structure of a machine learning model training apparatus according to an embodiment of the present disclosure;

[0044] Figure 5 This is a schematic diagram of the structure of a machine learning model training apparatus according to another embodiment of the present disclosure;

[0045] Figure 6 This is a schematic diagram of the structure of a machine learning model training apparatus according to yet another embodiment of the present disclosure;

[0046] Figure 7 This is a schematic flowchart of a scene graph generation method according to an embodiment of the present disclosure;

[0047] Figure 8 This is a schematic diagram of an image to be processed according to another embodiment of the present disclosure;

[0048] Figure 9 This is a schematic diagram of a scene according to an embodiment of the present disclosure;

[0049] Figure 10 This is a schematic diagram of a scene according to another embodiment of the present disclosure;

[0050] Figure 11 This is a schematic flowchart of a scene graph generation method according to another embodiment of the present disclosure;

[0051] Figure 12 This is a schematic diagram of a scenario according to yet another embodiment of the present disclosure;

[0052] Figure 13 This is a schematic diagram of the structure of a scene graph generation apparatus according to an embodiment of the present disclosure;

[0053] Figure 14 This is a schematic diagram of the structure of a scene graph generation apparatus according to another embodiment of the present disclosure;

[0054] Figure 15 This is a schematic diagram of the structure of a scene graph generation apparatus according to yet another embodiment of the present disclosure. Detailed Implementation

[0055] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0056] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0057] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0058] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0059] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0060] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0061] The inventors noted that graph convolution-based context information transfer methods tend to produce larger associated responses for targets that are spatially close. For example... Figure 1 As shown, although people and skateboards are semantically easily distinguishable, their close spatial proximity leads to a tendency for these two objects to learn a larger attention coefficient during information transmission. This results in the inability to correctly distinguish between the skateboard and the person in the image, a problem known as target representation confusion.

[0062] Accordingly, this disclosure effectively solves the problem of target representation confusion by denoising the image signal.

[0063] Figure 2 This is a schematic flowchart illustrating a machine learning model training method according to an embodiment of the present disclosure. In some embodiments, the following machine learning model training method is performed by a machine learning model training apparatus.

[0064] In step 201, the first target feature information is extracted from the sample image.

[0065] In some embodiments, target feature information in sample images is obtained by utilizing a target detection model such as Faster RCNN (Region Convolutional Neural Network).

[0066] For example, the first target feature information includes the target features of each target in the sample image, as well as the joint features of every two features in the sample image.

[0067] In step 202, the first machine learning model is used to process the first target feature information and the preset reweighting coefficients to obtain the second target feature information of the sample image.

[0068] It should be noted that the essence of information transmission processing is image signal denoising processing. Correspondingly, the second target feature information also includes the target features of each target in the sample image, as well as the joint features of every two features in the sample image.

[0069] In some embodiments, a target correlation matrix is ​​generated using the first target feature information and the reweighting coefficients, and a second target feature information is generated using the first target feature information and the target correlation matrix.

[0070] For example, the target association matrix is ​​associated with the Hadamard product of the reweighted coefficients and the attention score of the first target feature information, and the second target feature information is associated with the first target feature information, the product of the target association matrix and the first target feature information, and the sum of the preset target feature information.

[0071] For example, the target correlation matrix A (k+1) As shown in formula (1):

[0072] A (k+1) =Normalize(Ω) (k) ⊙H(Y (k) )) (1)

[0073] Where Normalize is the normalization function, Ω (k) Y is the reweighted coefficient. (k) Here, H represents the first target feature information, H is the attention scoring function, and ⊙ is the Hadamard multiplication operator.

[0074] Second target feature information Y (k+1) As shown in formula (2):

[0075] Y (k+1) =AF1(a(Y (k) +A (k+1) Y (k) +Y (0) (2)

[0076] Where AF1 is the activation function, a is the coefficient, and Y is the activation function. (0) Preset target feature information.

[0077] For example, AF1 is a ReLU function, and a is 1 / 3.

[0078] In step 203, the second target feature information of the sample image is used to generate the first scene image.

[0079] In step 204, a first loss function is generated using the annotation results of the first scene map and the sample images.

[0080] In step 205, the first machine learning model is trained using the first loss function.

[0081] In the machine learning model training method provided in the above embodiments of this disclosure, by denoising the image signal and then using the denoised target feature information to train the first machine learning model, the target recognition accuracy of the first machine learning model can be improved. Processing images using the trained first machine learning model can effectively solve the problem of target representation confusion.

[0082] The inventors also noted that insufficient relation labeling leads to a lack of diversity in target relation predictions. For example, in an image, the first person is standing on the street, the second person is sitting on the street, the third person is lying on the street, and the fourth person is walking on the street. Because the pre-set relation labeling only shows people standing on the street, the relationships between the second to fourth people in the image and the street cannot be accurately predicted.

[0083] To this end, this disclosure improves the diversity of relation predictions by introducing the constraint of the maximum rank of the relation prediction matrix.

[0084] Figure 3 This is a schematic flowchart illustrating a machine learning model training method according to another embodiment of the present disclosure. In some embodiments, the following machine learning model training method is performed by a machine learning model training apparatus.

[0085] In step 301, the first target feature information is extracted from the sample image.

[0086] In some embodiments, target feature information in sample images is obtained by utilizing a target detection model such as Faster RCNN.

[0087] For example, the first target feature information includes the target features of each target in the sample image, as well as the joint features of every two features in the sample image.

[0088] In step 302, the first machine learning model is used to process the first target feature information and the preset reweighting coefficients to obtain the second target feature information of the sample image.

[0089] It should be noted that the essence of information transmission processing is image signal denoising processing. Correspondingly, the second target feature information also includes the target features of each target in the sample image, as well as the joint features of every two features in the sample image.

[0090] In some embodiments, a target correlation matrix is ​​generated using the first target feature information and the reweighting coefficients, and a second target feature information is generated using the first target feature information and the target correlation matrix.

[0091] For example, the target association matrix is ​​associated with the Hadamard product of the reweighted coefficients and the attention score of the first target feature information, and the second target feature information is associated with the first target feature information, the product of the target association matrix and the first target feature information, and the sum of the preset target feature information.

[0092] For example, the target correlation matrix A (k+1) As shown in formula (1), the second target feature information Y (k+1) As shown in formula (2).

[0093] In step 303, the second target feature information is processed using a second machine learning model to generate a relationship prediction matrix, wherein the relationship prediction matrix includes relationship prediction features between every two targets in the sample image.

[0094] In some embodiments, the relationship prediction feature between the i-th target and the j-th target in the sample image is associated with the target feature of the i-th target, the target feature of the j-th target, and the joint feature of the i-th target and the j-th target, where 1≤i,j≤M, and i≠j.

[0095] For example, the predictive feature P of the relationship between the i-th target and the j-th target. ij As shown in formula (3).

[0096] P ij =AF2(W r (y i *y j *u ij )+f ij (3)

[0097] Where AF2 is the activation function, W r Let y be the relation prediction function. i Let y be the target feature of the i-th target. j Let u be the target feature of the j-th target. ij f represents the joint feature of the i-th objective and the j-th objective. ij Let represent the probability distribution of the relationship categories between the i-th target and the j-th target, and * represent the feature fusion operator.

[0098] For example, AF2 is the softmax function. The feature fusion operator * satisfies the conditions shown in formula (4).

[0099] x*y=AF3(W x x+W y y)-(W x xW y y)⊙(W x xW y y) (4)

[0100] Where x is the first feature vector, y is the second feature vector, AF3 is the activation function, and W... x Let W be the projection dimension of x. y Let y be the projection dimension, and ⊙ be the Hadamard multiplication operator.

[0101] For example, AF3 is a ReLU function.

[0102] Furthermore, if the relation prediction feature P∈R N×R If the relationship prediction feature P satisfies the following conditions, as shown in formulas (5) and (6).

[0103]

[0104]

[0105] In step 304, the maximum rank of the relation prediction matrix is ​​determined.

[0106] It should be noted that since the row vectors of the relation prediction matrix are linearly independent when predicting different relation categories, the maximum rank of the relation prediction matrix can be used to improve the diversity of predictions. However, maximizing the rank of a matrix is ​​an NP-hard problem; therefore, in some embodiments, the rank of the relation prediction matrix is... 2,1 The regularization term of the norm serves as the maximum rank of the relation prediction matrix.

[0107] For example, the L of the relation prediction matrix 2,1 Regular terms of the norm ||P|| 2,1 As shown in formula (7).

[0108]

[0109] Where N and R are the dimensions of the relation prediction matrix. Formula (7) can promote column sparsity of matrix P, thereby helping to improve the diversity of relation prediction.

[0110] In step 305, the cross-entropy loss function of the prediction results of each target relationship in the sample image is determined based on the relationship prediction matrix.

[0111] It should be noted that, since determining the cross-entropy loss function is not the inventive point of this disclosure, it will not be described in detail here.

[0112] In step 306, a second loss function is generated based on the cross-entropy loss function and the maximum rank of the relation prediction matrix.

[0113] In some embodiments, the second loss function is the difference between the cross-entropy loss function and the maximum rank of the relation prediction matrix.

[0114] For example, the second loss function L e As shown in formula (8).

[0115]

[0116] in, Let P be the cross-entropy loss function, 1 be the weight coefficient, B be the number of groups in the mini-batch, and P be the weight coefficient. b Let M be the relation prediction matrix for group b. B N is the prediction vector for the mini-batch. b Predict the features for the b-th group of relationships.

[0117] In step 307, the first machine learning model and the second machine learning model are trained using the second loss function.

[0118] Figure 4 This is a schematic diagram of the structure of a machine learning model training apparatus according to an embodiment of this disclosure. Figure 4 As shown, the machine learning model training device includes a first training module 41, a second training module 42, a third training module 43, and a fourth training module 44.

[0119] The first training module 41 is configured to extract the first target feature information from the sample image.

[0120] In some embodiments, target feature information in sample images is obtained by utilizing a target detection model such as Faster RCNN.

[0121] For example, the first target feature information includes the target features of each target in the sample image, as well as the joint features of every two features in the sample image.

[0122] The second training module 42 is configured to use the first machine learning model to perform information transmission processing on the first target feature information and the preset reweighting coefficients to obtain the second target feature information of the sample image.

[0123] It should be noted that the essence of information transmission processing is image signal denoising processing. Correspondingly, the second target feature information also includes the target features of each target in the sample image, as well as the joint features of every two features in the sample image.

[0124] In some embodiments, a target correlation matrix is ​​generated using the first target feature information and the reweighting coefficients, and a second target feature information is generated using the first target feature information and the target correlation matrix.

[0125] For example, the target association matrix is ​​associated with the Hadamard product of the reweighted coefficients and the attention score of the first target feature information, and the second target feature information is associated with the first target feature information, the product of the target association matrix and the first target feature information, and the sum of the preset target feature information.

[0126] For example, the target correlation matrix A (k+1) As shown in formula (1), the second target feature information Y (k+1) As shown in formula (2).

[0127] The third training module 43 is configured to generate a first scene map using the second target feature information of the sample images.

[0128] The fourth training module 44 is configured to generate a first loss function using the annotation results of the first scene map and sample images, and to train the first machine learning model using the first loss function.

[0129] Figure 5 This is a schematic diagram of the structure of a machine learning model training apparatus according to another embodiment of the present disclosure. Figure 5 and Figure 4 The difference is that, in Figure 5 In the embodiment shown, the machine learning model training device further includes a fifth training module 45.

[0130] The fifth training module 45 uses the second machine learning model to process the feature information of the second target to generate a relationship prediction matrix, wherein the relationship prediction matrix includes the relationship prediction features between every two targets in the sample image.

[0131] In some embodiments, the relationship prediction feature between the i-th target and the j-th target in the sample image is associated with the target feature of the i-th target, the target feature of the j-th target, and the joint feature of the i-th target and the j-th target, where 1≤i,j≤M, and i≠j.

[0132] For example, the predictive feature P of the relationship between the i-th target and the j-th target. ij As shown in formula (3).

[0133] Next, the fifth training module 45 determines the maximum rank of the relation prediction matrix.

[0134] For example, the L of the relation prediction matrix 2,1 Regular terms of the norm ||P|| 2,1 As shown in formula (7).

[0135] Next, the fifth training module 45 determines the cross-entropy loss function of the prediction results of each target relationship in the sample image based on the relationship prediction matrix, and generates a second loss function based on the cross-entropy loss function and the maximum rank of the relationship prediction matrix.

[0136] In some embodiments, the second loss function is the difference between the cross-entropy loss function and the maximum rank of the relation prediction matrix.

[0137] For example, the second loss function L e As shown in formula (8).

[0138] Next, the fifth training module 45 uses the second loss function to train the first machine learning model and the second machine learning model.

[0139] Figure 6 This is a schematic diagram of the structure of a machine learning model training apparatus according to yet another embodiment of this disclosure. Figure 6 As shown, the machine learning model training device includes a memory 61 and a processor 62.

[0140] Memory 61 is used to store instructions, and processor 62 is coupled to memory 61. Processor 62 is configured to execute instructions based on the memory storage, as shown in the example. Figure 2 or Figure 3 The method involved in any of the embodiments.

[0141] like Figure 6 As shown, the machine learning model training device also includes a communication interface 63 for exchanging information with other devices. Additionally, the machine learning model training device includes a bus 64, through which the processor 62, communication interface 63, and memory 61 communicate with each other.

[0142] Memory 61 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. Memory 61 may also be a memory array. Memory 61 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0143] Furthermore, processor 62 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0144] This disclosure also relates to a computer-readable storage medium storing computer instructions that, when executed by a processor, implement as follows: Figure 2 or Figure 3 The method involved in any of the embodiments.

[0145] Figure 7 This is a schematic flowchart illustrating a scene graph generation method according to an embodiment of the present disclosure. In some embodiments, the following scene graph generation method is performed by a scene graph generation apparatus.

[0146] In step 701, the third target feature information is extracted from the image to be processed.

[0147] In some embodiments, target feature information in the image to be processed is obtained by utilizing a target detection model such as Faster RCNN.

[0148] For example, the third target feature information includes the target features of each target in the image to be processed, as well as the joint features of every two features in the image to be processed.

[0149] In step 702, the first machine learning model is used to process the third target feature information and the preset reweighting coefficients to obtain the fourth target feature information of the image to be processed.

[0150] The first machine learning model utilizes Figure 2 The machine learning model is trained using the training method shown in any of the embodiments.

[0151] Accordingly, the fourth target feature information also includes the target features of each target in the image to be processed, as well as the joint features of every two features in the image to be processed.

[0152] In step 703, the second scene map is generated using the fourth target feature information.

[0153] like Figure 8 As shown, the image contains a man and a woman; the woman is sitting inside a car, and the man is standing beside the car. The scene graph generated using existing scene graph generation schemes is as follows: Figure 9 As shown. Clearly, because the man and woman are close together, the existing scene graph generation scheme incorrectly identifies that both the man and woman are sitting in the car.

[0154] And in Figure 7 In the provided scene graph generation method, the trained first machine learning model can reduce noise in the image signal, thereby effectively improving the accuracy of target recognition.

[0155] For example, using Figure 7 The provided embodiments are for Figure 8 The resulting scene image is as follows: Figure 10 As shown. Obviously, since the first machine learning model trained is able to denoise the image signal, it can accurately distinguish the man and woman in the picture, and thus identify that the woman is sitting in the car and the man is next to the car.

[0156] Figure 11 This is a schematic flowchart illustrating a scene graph generation method according to an embodiment of the present disclosure. In some embodiments, the following scene graph generation method is performed by a scene graph generation apparatus.

[0157] In step 1101, the third target feature information is extracted from the image to be processed.

[0158] In some embodiments, target feature information in the image to be processed is obtained by utilizing a target detection model such as Faster RCNN.

[0159] For example, the third target feature information includes the target features of each target in the image to be processed, as well as the joint features of every two features in the image to be processed.

[0160] In step 1102, the first machine learning model is used to process the third target feature information and the preset reweighting coefficients to obtain the fourth target feature information of the image to be processed.

[0161] The first machine learning model utilizes Figure 3 The machine learning model is trained using the training method shown in any of the embodiments.

[0162] Accordingly, the fourth target feature information also includes the target features of each target in the image to be processed, as well as the joint features of every two features in the image to be processed.

[0163] In step 1103, the second machine learning model is used to process the feature information of the fourth target to generate a relationship prediction matrix. This matrix includes relationship prediction features between every two targets in the image to be processed. The second machine learning model utilizes... Figure 3 The machine learning model is trained using the training method shown in any of the embodiments.

[0164] In step 1104, the third scene map is generated using the fourth target feature information and the relationship prediction matrix.

[0165] like Figure 8 As shown in the image, the woman has one hand on the car door and is holding a mobile phone in the other. Due to insufficient relationship annotation, the relationship between the phone and the scene cannot be determined when generating the scene graph. Figure 8 What relationships exist between the other objectives within it? For example, in Figure 9or Figure 10 The generated scene image only shows the relationship between the woman and her hand, but not the relationship between the phone and the woman.

[0166] This disclosure effectively improves the diversity of relation predictions by introducing the constraint of the maximum rank of the relation prediction matrix when training the first machine learning model and the second machine learning model.

[0167] For example, using Figure 11 The provided embodiments are for Figure 8 The resulting scene image is as follows: Figure 12 As shown in the image, by simultaneously training the first and second machine learning models, the system can accurately distinguish between men and women in the image and also identify that the woman is holding a mobile phone. This effectively solves the problem of low relation diversity caused by insufficient relation labeling.

[0168] Figure 13 This is a schematic diagram of the structure of a scene graph generation apparatus according to an embodiment of this disclosure. Figure 13 As shown, the scene graph generation device includes a first processing module 131, a second processing module 132, and a third processing module 133.

[0169] The first processing module 133 is configured to extract third target feature information from the image to be processed.

[0170] In some embodiments, target feature information in the image to be processed is obtained by utilizing a target detection model such as Faster RCNN.

[0171] For example, the third target feature information includes the target features of each target in the image to be processed, as well as the joint features of every two features in the image to be processed.

[0172] The second processing module 132 is configured to use the first machine learning model to perform information transmission processing on the third target feature information and the preset reweighting coefficients to obtain the fourth target feature information of the image to be processed.

[0173] The first machine learning model utilizes Figure 2 or Figure 3 The machine learning model is trained using the training method shown in any of the embodiments.

[0174] Accordingly, the fourth target feature information also includes the target features of each target in the image to be processed, as well as the joint features of every two features in the image to be processed.

[0175] The third processing module 133 is configured to generate a third scene map using the feature information of the fourth target.

[0176] Figure 14 This is a schematic diagram of the structure of a scene graph generation apparatus according to another embodiment of the present disclosure. Figure 14 and Figure 13 The difference is that, in Figure 14 In the embodiment shown, the scene graph generation device further includes a fourth processing module 134.

[0177] The fourth processing module 134 uses the second machine learning model to process the feature information of the fourth target to generate a relationship prediction matrix. The relationship prediction matrix includes the relationship prediction features between every two targets in the image to be processed. The second machine learning model utilizes... Figure 3 The machine learning model is trained using the training method shown in any of the embodiments. The fourth processing module 134 generates a third scene graph using the fourth target feature information and the relationship prediction matrix.

[0178] Figure 15 This is a schematic diagram of the structure of a scene graph generation apparatus according to yet another embodiment of this disclosure. Figure 15 As shown, the scene graph generation device includes a memory 151, a processor 152, a communication interface 153, and a bus 154. Figure 15 and Figure 6 The difference is that, in Figure 15 In the illustrated embodiment, processor 152 is configured to implement memory-based instruction execution as follows: Figure 11 The method involved in any of the embodiments.

[0179] This disclosure also relates to a computer-readable storage medium storing computer instructions that, when executed by a processor, implement as follows: Figure 11 The method involved in any of the embodiments.

[0180] In some embodiments, the functional unit modules described above may be implemented as general-purpose processors, programmable logic controllers (PLCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any suitable combination thereof for performing the functions described herein.

[0181] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0182] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for training a machine learning model, comprising: Extract the first target feature information from the sample image; The first machine learning model is used to process the first target feature information and the preset reweighting coefficients to obtain the second target feature information of the sample image. A first scene image is generated using the second target feature information of the sample image; A first loss function is generated using the annotation results of the first scene map and the sample images; The first machine learning model is trained using the first loss function; The step of using a first machine learning model to perform information transmission processing on the first target feature information and the preset reweighting coefficients includes: A target association matrix is ​​generated using the first target feature information and the reweighted coefficients. The target association matrix is ​​associated with the Hadamard product of the reweighted coefficients and the attention score result of the first target feature information. The second target feature information is generated using the first target feature information and the target association matrix. The second target feature information is associated with the sum of the first target feature information, the product of the target association matrix and the first target feature information, and preset target feature information.

2. The method according to claim 1, wherein, The target correlation matrix for: in, For normalization function, The reweighting coefficients are... The first target feature information, For attention scoring function, It is the Hadamard multiplication operator; Second target feature information for: Where AF1 is the activation function. For coefficients, The preset target feature information.

3. The method according to any one of claims 1-2, further comprising: The second target feature information is processed using a second machine learning model to generate a relationship prediction matrix, wherein the relationship prediction matrix includes relationship prediction features between every two targets in the sample image; Determine the maximum rank of the relation prediction matrix; The cross-entropy loss function for each target relationship prediction result in the sample image is determined based on the relationship prediction matrix. A second loss function is generated based on the cross-entropy loss function and the maximum rank of the relation prediction matrix; The first machine learning model and the second machine learning model are trained using the second loss function.

4. The method according to claim 3, wherein, The second target feature information includes the target features of each target in the sample image, and the joint features of every two features in the sample image.

5. The method according to claim 4, wherein, The relationship prediction feature between the i-th target and the j-th target in the sample image is associated with the target feature of the i-th target, the target feature of the j-th target, and the joint feature of the i-th target and the j-th target, where 1≤i, j≤M, and i≠j.

6. The method according to claim 5, wherein, Predictive features of the relationship between the i-th target and the j-th target for: Where AF2 is the activation function. For relation prediction functions, The target features of the i-th target are... The target features of the j-th target are... The joint features of the i-th target and the j-th target, Let be the probability distribution of the relationship category between the i-th target and the j-th target. This is a feature fusion operator.

7. The method according to claim 6, wherein, Feature fusion operator The following conditions must be met: in, The first eigenvector, This is the second feature vector. For activation function, for The projection dimension, for The projection dimension, This is the Hadamard multiplication operator.

8. The method according to claim 3, wherein, Determining the maximum rank of the relation prediction matrix includes: The relationship prediction matrix The regularization term of the norm serves as the maximum rank of the relation prediction matrix.

9. The method according to claim 8, wherein, The relationship prediction matrix Regular terms of norm for Where N and R are the dimensions of the relationship prediction matrix.

10. The method according to claim 3, wherein, The second loss function is the difference between the cross-entropy loss function and the maximum rank of the relation prediction matrix.

11. The method of claim 10, wherein, The second loss function for in, Let the cross-entropy loss function be... Here, B is the weighting factor, and B is the number of groups within the mini-batch. This is the relation prediction matrix for group b. Let be the prediction vector for the small batch. Predict the features for the b-th group of relationships.

12. A machine learning model training device, comprising: The first training module is configured to extract the first target feature information from the sample image; The second training module is configured to use a first machine learning model to perform information transfer processing on the first target feature information and preset reweighting coefficients to obtain the second target feature information of the sample image. The first target feature information and the reweighting coefficients are used to generate a target association matrix. The target association matrix is ​​associated with the Hadamard product of the reweighting coefficients and the attention score result of the first target feature information. The first target feature information and the target association matrix are used to generate the second target feature information. The second target feature information is associated with the sum of the first target feature information, the product of the target association matrix and the first target feature information, and preset target feature information. The third training module is configured to generate a first scene map using the second target feature information of the sample image; The fourth training module is configured to generate a first loss function using the annotation results of the first scene map and the sample image, and to train the first machine learning model using the first loss function.

13. A machine learning model training device, comprising: The memory is configured to store instructions; A processor, coupled to a memory, configured to implement the method as described in any one of claims 1-11 based on memory-stored instruction execution.

14. A method for generating a scene graph, comprising: Extract the third target feature information from the image to be processed; The first machine learning model is used to process the third target feature information and the preset reweighting coefficients to obtain the fourth target feature information of the image to be processed, wherein the first machine learning model is trained using the machine learning model training method of any one of claims 1-11. The second scene map is generated using the fourth target feature information.

15. The method of claim 14, further comprising: The second machine learning model is used to process the feature information of the fourth target to generate a relationship prediction matrix, wherein the relationship prediction matrix includes relationship prediction features between every two targets in the image to be processed, and the second machine learning model is trained using the machine learning model training method of any one of claims 3-11; The third scene map is generated using the fourth target feature information and the relationship prediction matrix.

16. A scene graph generation apparatus, comprising: The first processing module is configured to extract the third target feature information from the image to be processed; The second processing module is configured to use a first machine learning model to perform information transmission processing on the third target feature information and preset reweighting coefficients to obtain the fourth target feature information of the image to be processed, wherein the first machine learning model is trained using the machine learning model training method of any one of claims 1-11. The third processing module is configured to generate a second scene map using the fourth target feature information.

17. A scene graph generation apparatus, comprising: The memory is configured to store instructions; A processor, coupled to memory, configured to implement the method as described in any one of claims 14-15 based on memory-stored instruction execution.

18. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-11 and 14-15.

Images