End-to-end relationship identification method, model training method, device, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to an end-to-end relationship identification method and device, a model training method and device, and a medium. The method obtains a sample image and a label vector thereof, inputs the sample image into a feature extraction model to obtain sample features, inputs the sample features into a prediction model to obtain N prediction vectors, matches the N prediction vectors with preset label vectors respectively by using a preset matching algorithm, determines a prediction vector with the maximum matching degree as a reference vector, calculates an overall matching error according to the reference vector and the label vector, trains the feature extraction model and the prediction model according to the overall matching error, and determines that a trained feature extraction model and a trained prediction model constitute a trained relationship identification model. The end-to-end relationship identification model reduces the influence of human subjective factors, improves the fitting effect of the model, and further improves the accuracy of the relationship identification model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an end-to-end relationship recognition method, model training method, apparatus, device and medium. Background Technology

[0002] With the development of artificial intelligence technology, the accuracy of relationship recognition technology in identifying relationships between objects is getting higher and higher. Relationship recognition technology is now widely used in scenarios such as public security and user recommendation, generating great social and commercial value. As the accuracy of relationship recognition technology increases, it can also be gradually applied to scenarios such as intelligent security and precise targeting.

[0003] However, current relationship recognition models still employ a two-stage approach: first, the bounding boxes of the objects to be identified in the image are determined, and then relationship recognition is performed based on these bounding boxes. This two-stage architecture necessitates manual parameter tuning during training, making it difficult to achieve optimal recognition results and consequently leading to low accuracy in the trained relationship recognition model. Therefore, improving the accuracy of relationship recognition models has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an end-to-end relationship recognition method, model training method, apparatus, device, and medium to solve the problem of low accuracy of relationship recognition models.

[0005] In a first aspect, embodiments of the present invention provide a training method for an end-to-end relationship recognition model, the training method comprising:

[0006] A sample image and its corresponding at least one label vector are obtained. The sample image is input into a feature extraction model for feature extraction to obtain sample features. The label vector includes a first label bounding box, a second label bounding box, and a label relationship category.

[0007] The sample features are input into the prediction model to obtain N prediction vectors. The prediction vectors include a first sample bounding box, a second sample bounding box, and a sample relationship category, where N is an integer greater than zero.

[0008] A preset matching algorithm is used to match the N prediction vectors with the at least one label vector, and the prediction vector with the highest matching degree with any label vector is determined as the reference vector of the corresponding label vector.

[0009] Iterate through each label vector and calculate the category matching error between the sample relationship category in the reference vector and the label relationship category in the label vector;

[0010] Calculate the first bounding box matching error between the first sample bounding box in the reference vector and the first label bounding box in the label vector, and calculate the second bounding box matching error between the second sample bounding box in the reference vector and the second label bounding box in the label vector;

[0011] Based on the category matching error, the first bounding box matching error, and the second bounding box matching error, the overall matching error of the label vector is determined. Based on the overall matching error of all label vectors, the feature extraction model and the prediction model are trained, and the trained feature extraction model and the trained prediction model are combined to form a trained relationship recognition model.

[0012] Secondly, embodiments of the present invention provide an end-to-end relationship identification method, the end-to-end relationship identification method comprising:

[0013] Acquire an image to be identified, wherein the image to be identified contains at least two objects to be identified;

[0014] The image to be identified is input into the trained relationship recognition model to obtain P relationship recognition results. The relationship recognition results include the bounding boxes of the two objects to be identified and the recognition relationship category between the two objects to be identified, where P is an integer greater than zero. The trained relationship recognition model is obtained based on the training method of the end-to-end relationship recognition model according to any one of claims 1-6.

[0015] Thirdly, embodiments of the present invention provide a training apparatus for an end-to-end relationship recognition model, the training apparatus comprising:

[0016] The feature extraction module is used to acquire a sample image and its corresponding at least one label vector, input the sample image into the feature extraction model for feature extraction, and obtain sample features. The label vector includes a first label bounding box, a second label bounding box, and a label relationship category.

[0017] The vector prediction module is used to input the sample features into the prediction model to obtain N prediction vectors. The prediction vectors include a first sample bounding box, a second sample bounding box, and a sample relationship category, where N is an integer greater than zero.

[0018] The vector matching module is used to match the N prediction vectors with the at least one label vector using a preset matching algorithm, and determine the prediction vector with the highest matching degree with any label vector as the reference vector of the corresponding label vector.

[0019] The first calculation module is used to traverse each label vector and calculate the category matching error between the sample relationship category in the reference vector and the label relationship category in the label vector;

[0020] The second calculation module is used to calculate the first bounding box matching error between the first sample bounding box in the reference vector and the first label bounding box in the label vector, and to calculate the second bounding box matching error between the second sample bounding box in the reference vector and the second label bounding box in the label vector.

[0021] The model training module is used to determine the overall matching error based on the category matching error, the first bounding box matching error, and the second bounding box matching error, and to train the feature extraction model and the prediction model based on the overall matching error of all label vectors, so as to determine that the trained feature extraction model and the trained prediction model constitute a trained relationship recognition model.

[0022] Fourthly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training method as described in the first aspect.

[0023] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method as described in the first aspect.

[0024] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0025] Obtain sample images and their corresponding at least one label vector. Input the sample images into a feature extraction model for feature extraction to obtain sample features. Input the sample features into a prediction model to obtain N prediction vectors. Use a preset matching algorithm to match each of the N prediction vectors with at least one label vector. Determine the prediction vector with the highest matching degree to any label vector as the reference vector for that label vector. Iterate through each label vector and calculate the category matching error between the sample relationship category in the reference vector and the label relationship category in the label vector. Calculate the first bounding box matching error between the first bounding box of the reference vector and the first bounding box of the label vector, and calculate the second bounding box matching error between the second bounding box of the reference vector and the second bounding box of the label vector. Based on the category matching error, the first bounding box matching error, and the second bounding box matching error, the overall matching error of the label vector is determined. Based on the overall matching error of all label vectors, the feature extraction model and the prediction model are trained. The trained feature extraction model and the trained prediction model together form a trained relationship recognition model. The category matching error and bounding box matching error are calculated between the predicted vector and the label vector of the corresponding sample image, improving the accuracy of spatial association recognition. Based on this, a matching reference vector is determined for calculating the loss function. The relationship recognition model is trained in an end-to-end manner, avoiding manual model tuning, reducing the influence of subjective factors, improving the fitting effect of the relationship recognition model, and thus improving the accuracy of the relationship recognition model. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of an application environment for a training method of an end-to-end relationship recognition model provided in Embodiment 1 of the present invention;

[0028] Figure 2 This is a flowchart illustrating a training method for an end-to-end relationship recognition model provided in Embodiment 1 of the present invention.

[0029] Figure 3 This is a flowchart illustrating a training method for an end-to-end relationship recognition model provided in Embodiment 2 of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of a training device for an end-to-end relationship recognition model provided in Embodiment 4 of the present invention;

[0031] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 5 of the present invention. Detailed Implementation

[0032] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0033] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0034] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0035] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0036] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0038] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0039] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0040] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0041] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0042] The training method for an end-to-end relationship recognition model provided in Embodiment 1 of this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server. The client includes, but is not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The client can be applied to scenarios such as image classification, intelligent security, and user recommendation. The client receives relationship recognition requests, which can be sent by the user or automatically generated at preset time points. For example, in an intelligent security scenario, relationship recognition requests can be automatically generated at preset time points to detect abnormal intrusions in real time based on monitoring images. That is, based on the identified social relationships, cases where individuals do not meet the preset social relationships but simultaneously enter the access control system are identified as abnormal intrusions. The server connects to image acquisition devices in application scenarios. For example, in a smart security scenario, the image acquisition device could be a surveillance camera, while in a user recommendation scenario, it could be a camera, to acquire images captured by the image acquisition device. In this embodiment, one relationship recognition result corresponds to two objects to be identified in the image. When multiple objects exist in the image, the relationship between multiple objects to be identified can be determined based on the pairwise relationship recognition results. The objects to be identified are generally people. For example, in a user recommendation scenario, if the pairwise relationship recognition results of three objects to be identified are all family members, then it is determined that the objects to be identified can be family product audiences, thereby providing accurate recommendations.

[0043] See Figure 2 This is a flowchart illustrating a training method for an end-to-end relationship recognition model provided in Embodiment 1 of the present invention. The training method described above can be applied to... Figure 1 The client-side computer connects to the server to obtain sample images and their label vectors. The client-side deploys a feature extraction model and a prediction model. The feature extraction model extracts feature information from the sample images, while the prediction model predicts social relationship categories based on these feature information. Figure 2 As shown, the training method may include the following steps:

[0044] Step S201: Obtain the sample image and its corresponding at least one label vector, input the sample image into the feature extraction model for feature extraction, and obtain the sample features.

[0045] The label vector includes a first label bounding box, a second label bounding box, and a label relationship category. The sample image can be an image containing the object to be identified. In order to facilitate model learning, there are at least two objects to be identified in the sample image in this embodiment.

[0046] Feature extraction models can combine visual semantic models and Transformer models. That is, the sample image is input into the visual semantic model for processing to obtain semantic feature vectors, and then the semantic feature vectors are input into the Transformer model to obtain sample features.

[0047] Visual semantic models can use residual network models, which can be multi-layer convolutional models based on residual connections. Residual connections mean that the output of each convolutional layer is combined with the output of the previous convolutional layer and used as the input of the next convolutional layer. The Transformer model can act as an attention mechanism to extract important features from the semantic feature vector.

[0048] Optionally, the feature extraction model includes a semantic extraction layer, an encoder, and a decoder;

[0049] The sample image is input into the feature extraction model for feature extraction, and the sample features are obtained as follows:

[0050] The sample image is input into the semantic extraction layer to obtain the semantic feature vector. The first vector and the semantic feature vector are multiplied by a preset first vector to obtain the value vector.

[0051] The sample image is encoded using a preset encoding method to obtain a positional encoding. The semantic feature vector is added to the positional encoding point by point. The sum is then multiplied by the preset second vector and the preset third vector to obtain the query vector and the key value vector.

[0052] Input the value vector, query vector, and key vector into the encoder to obtain the global information feature vector;

[0053] The preset learnable vectors are initialized, and the initialized learnable vectors, global information feature vectors, and positional codes are input into the decoder to obtain sample features.

[0054] The semantic extraction layer can use a visual backbone network, such as the encoder of a semantic segmentation model. The preset first vector, second vector, and third vector can be learnable weight vectors. The position encoding can refer to the position information encoding of each sub-image block in the image. The position information encoding can be obtained by one-hot encoding, sine and cosine encoding, or by processing the position encoding part of the BERT model. The preset initialization can be done by random initialization.

[0055] Value vectors, query vectors, and key-value vectors can be used to transform low-dimensional semantic feature vectors into high-dimensional spaces. Learnable vectors use dimension as a hyperparameter, meaning the dimension of the learnable vector is preset by the implementer. It should be noted that the dimension of the learnable vector needs to be greater than the maximum number of relations in the sample image.

[0056] Specifically, in this embodiment, the sample image is divided into K sub-image blocks of the same size. Each sub-image block is input into the semantic extraction layer for semantic feature extraction, resulting in K semantic feature vectors. Each sub-image block is assigned a sequence number, resulting in K sub-image block numbers. Position encoding is performed based on the K sub-image block numbers. Position encoding can be one-hot encoding, resulting in K position codes. The dimension of the position codes is the same as that of the semantic feature vectors. The semantic feature vector of the corresponding sub-image block is added to the position code of the corresponding sub-image block and concatenated to obtain a vector of size K*D as the addition result, where D can refer to the dimension of the semantic feature vector.

[0057] In this embodiment, query vectors, key vectors, and value vectors are constructed using weight vectors with learnable parameters. This effectively extracts the contextual information of sub-image patches in the sample image, which facilitates subsequent weighting of features based on the contextual information. This achieves the effect of an attention mechanism, improves the efficiency of feature integration, and avoids the loss of important features.

[0058] Optionally, the input of the value vector, query vector, and key-value vector into the encoder includes:

[0059] Multiply the query vector by the transpose of the key vector, divide the result of the multiplication by a preset coefficient, and determine the result of the division as the attention vector;

[0060] The attention vector is normalized using a normalized exponential function. The normalized result is multiplied by the value vector, and the result of the multiplication is input into the encoder.

[0061] Among them, the preset coefficients can be used to scale the multiplication result to avoid the multiplication result being too large. The attention vector can be a vector containing weighted parameters. The normalization process can be used to normalize the weighted parameters into probability values, that is, the sum of the normalized values ​​of each column in the attention vector is 1.

[0062] Specifically, in this embodiment, the preset coefficient can be set to... That is, the square root of the dimension of the semantic feature vector.

[0063] In this embodiment, the value vector is weighted by the normalization result of the attention vector, thereby giving attention to the features so that subsequent global feature processing focuses more on important features and ensures the effectiveness of the global information feature vector.

[0064] Optionally, the sample image is input into the semantic extraction layer to obtain a semantic feature vector, including:

[0065] The sample image is input into the semantic extraction layer to obtain the initial feature vector;

[0066] The initial feature vector is reduced in dimensionality using a pre-defined convolutional layer. The dimensionality reduction result is then flattened, and the result of the flattening operation is determined to be the semantic feature vector.

[0067] The initial feature vector can be a vector obtained by directly extracting features from the sample image, and the convolutional layer can include a convolutional kernel of a preset size of n*n.

[0068] For example, in this embodiment, a 5x5 convolutional kernel is used. Let the initial feature vector size be H*W*C, where H and W are the height and width of the vector, and C is the number of channels. The initial value of C is 2048. After convolution with the 5x5 kernel, C is reduced to 256. Flattening can refer to compression, which compresses a high-dimensional vector to a size of 1*M, where M can be equal to H*W*C. Flattening converts the high-dimensional semantic feature vector into a low-dimensional representation, which can then be viewed as a word embedding vector in a regular Transformer, facilitating subsequent multiplication with positional encodings.

[0069] This embodiment performs dimensionality reduction on the feature vectors, thereby reducing the computational load of subsequent processing, improving computational efficiency, enabling social relationship recognition to be applied in real-time scenarios, and enhancing the applicability of the relationship recognition model.

[0070] The steps described above—obtaining sample images and at least one corresponding label vector, inputting the sample images into a feature extraction model for feature extraction, and obtaining sample features—obtain sample features with strong representational capabilities through the feature extraction model, thereby improving the fitting accuracy and efficiency of the subsequent model training process, and thus improving the accuracy of the trained relationship recognition model.

[0071] Step S202: Input the sample features into the prediction model to obtain N prediction vectors.

[0072] The prediction vector includes a first sample bounding box, a second sample bounding box, and a sample relationship category. N is a positive integer. The prediction model can include a classification branch and two regression branches. Each branch can be implemented using a fully connected layer. The input of the classification branch is the sample features, and the output of the classification branch is the sample relationship category. The input of the two regression branches is the sample features, and the outputs of the two regression branches are the first sample bounding box and the second sample bounding box, respectively.

[0073] The first sample bounding box can refer to the location box of an object to be identified in the sample image, and the second sample bounding box can refer to the location box of another object to be identified in the sample image. The sample relationship category can refer to the relationship recognition result between the object to be identified corresponding to the first sample bounding box and the object to be identified corresponding to the second sample bounding box. The relationship recognition result can include kinship relationship category, friend relationship category, neighbor relationship category, etc.

[0074] Specifically, the classification branch outputs the predicted value for each relation category. After normalizing the predicted value for each relation category, the predicted probability for each relation category is obtained. The relation category corresponding to the maximum predicted probability among all relation categories is determined as the sample relation category.

[0075] The regression branch outputs the bounding box coordinates, which can include the coordinates of the top-left corner and the bottom-right corner of the bounding box.

[0076] Since the sample image contains at least two objects to be identified, and each pair of objects corresponds to a prediction vector, the output of the prediction model is N prediction vectors. For example, when there are Q objects to be identified, the value of N is the same as the factorial of Q-1.

[0077] The above steps of inputting sample features into the prediction model to obtain N prediction vectors adopt an end-to-end model architecture. It directly outputs the first bounding box, the second bounding box, and the sample relationship category based on the sample features of the input sample image, which improves the convenience of model training and use. As a result, it can be more widely applied to specific scenarios. Under the premise of ensuring model accuracy, the implementer only needs to directly input the acquired data into the model to obtain the corresponding results, making the model highly applicable and convenient.

[0078] Step S203: Using a preset matching algorithm, the N prediction vectors are matched with at least one label vector respectively, and the prediction vector with the highest matching degree with any label vector is determined as the reference vector of the corresponding label vector.

[0079] The preset matching algorithm can be the Hungarian algorithm, the KM algorithm, etc. In this embodiment, each predicted vector is matched with the label vector to obtain N matching pairs.

[0080] In this embodiment, for any matching pair, the matching loss can be calculated based on the first and second sample bounding boxes of the predicted vector within the matching pair and the first and second label bounding boxes in the label vector. The loss calculation method between bounding boxes can be the generalized intersection-union ratio.

[0081] The generalized intersection-union ratio (OCR) is calculated as follows: for two bounding boxes, first obtain the minimum bounding rectangle of the two bounding boxes, and calculate the ratio of the area of ​​the minimum bounding rectangle minus the area of ​​the union of the two bounding boxes to the area of ​​the minimum bounding rectangle. Subtract the above ratio from the OCR of the two bounding boxes to obtain the generalized OCR. Using the generalized OCR can more accurately describe the positional relationship between the bounding boxes.

[0082] The smaller the generalized intersection-union ratio (GUIR) of the bounding boxes in a matching pair, the greater the matching loss and the smaller the matching degree. Conversely, the larger the GUIR of the bounding boxes in a matching pair, the smaller the matching loss and the greater the matching degree. The predicted vector in the matching pair with the highest matching degree is determined as the reference vector.

[0083] The above-described steps involve using a preset matching algorithm to match N predicted vectors with at least one label vector, determining the predicted vector with the highest degree of matching to any label vector as the reference vector for the corresponding label vector, and determining the label based on the matching results. This approach can effectively adapt to sample images with multiple objects to be identified, avoid situations where the label vector and the predicted vector do not correspond, leading to incorrect fitting of the relationship recognition model, and thus improve the accuracy of the relationship recognition model fitting.

[0084] Step S204: Traverse each label vector and calculate the category matching error between the sample relationship category in the reference vector and the label relationship category in the label vector.

[0085] The label relationship category can be a pre-set relationship category, which can include spousal relationship, friend relationship, family relationship, etc. The relationship category can be represented by one-hot encoding. For example, if the total number of relationship categories is 3, the one-hot encoding of the spousal relationship category can be [1, 0, 0]. The category matching error can be used to characterize the difference between the sample relationship category and the label relationship category.

[0086] Specifically, the category matching error can be calculated using classification error, such as the cross-entropy loss function. The larger the category matching error, the greater the difference between the sample relationship category and the label relationship category. The smaller the category matching error, the smaller the difference between the sample relationship category and the label relationship category. Since the category matching error is calculated using the reference vector, the subsequent training process will adjust the model parameters to make the output reference vector and the label vector close enough. Accordingly, the sample relationship category and the label relationship category output by the adjusted model will be as consistent as possible.

[0087] The steps described above—traversing each label vector and calculating the class matching error between the sample relationship category in the reference vector and the label relationship category in the label vector—use the class matching error as supervision during subsequent training to ensure that the sample relationship category output by the model is as consistent as possible with the label relationship category in the label vector. This allows the model to be trained in an end-to-end manner, thereby improving the training efficiency and recognition accuracy of the model.

[0088] Step S205: Calculate the first bounding box matching error between the first sample bounding box in the reference vector and the first label bounding box in the label vector, and calculate the second bounding box matching error between the second sample bounding box in the reference vector and the second bounding box in the label vector.

[0089] The sample bounding box and the label bounding box are both represented by bounding box positioning points. The bounding box positioning points can include the upper left corner and the lower right corner of the bounding box. The bounding box matching error can be used to characterize the distance difference between the sample bounding box and the label bounding box.

[0090] Specifically, the bounding box matching error can be calculated using the generalized intersection-union ratio (CIU) method described above. In one embodiment, it can also be calculated using regression error, such as the mean squared error loss function. The larger the bounding box matching error, the farther the distance between the sample bounding box and the label bounding box. The smaller the bounding box matching error, the closer the distance between the sample bounding box and the label bounding box. Since the matching loss is calculated using the reference vector, the subsequent training process will adjust the model parameters so that the output reference vector is close enough to the label vector. Correspondingly, the distance between the sample bounding box and the label bounding box output by the adjusted model will also be close enough.

[0091] The steps described above—calculating the first bounding box matching error between the first sample bounding box in the reference vector and the first label bounding box in the label vector, and calculating the second bounding box matching error between the second sample bounding box in the reference vector and the second bounding box in the label vector—enable the learning of the location of the object to be identified through a bounding box matching error supervision model. This provides feature information for the relationship recognition task and facilitates improving the accuracy of the model in relationship recognition.

[0092] Step S206: Determine the overall matching error of the label vectors based on the category matching error, the first bounding box matching error, and the second bounding box matching error. Train the feature extraction model and the prediction model based on the overall matching error of all label vectors. The trained feature extraction model and the trained prediction model are then combined to form the trained relationship recognition model.

[0093] The trained feature extraction model can be used to extract features from the acquired image to be identified, and the trained prediction model can be used to map the feature tensor corresponding to the image to be identified to the output space. In this embodiment, the output space can refer to the relationship category space and the bounding box positioning point space to obtain the relationship recognition result between the objects to be identified in the image to be identified.

[0094] Specifically, the loss function used for training is:

[0095]

[0096] Where L represents the predicted loss, L match (g i p σ(i) ) represents the overall matching error, g i Let p represent the i-th label vector. σ(i) Let σ(i) represent the predicted vector that matches the i-th label vector, and let σ(i) represent the index of the predicted vector that matches the i-th label vector.

[0097] Optionally, the category matching error corresponds to the first weight, and the first bounding box matching error and the second bounding box matching error correspond to the second weight;

[0098] Based on the category matching error, the first bounding box matching error, and the second bounding box matching error, the overall matching error of the label vector is determined to include:

[0099] The first matching error is obtained by multiplying the category matching error by the first weight.

[0100] The first bounding box matching error and the second bounding box matching error are multiplied by the second weight respectively, and then added together to obtain the second matching error;

[0101] The sum of the first matching error and the second matching error is determined as the overall matching error.

[0102] The first weight can be used to control the influence of category matching error on the overall matching error, and the second weight can be used to control the influence of bounding box matching error on the overall matching error.

[0103] Specifically, the overall matching error is:

[0104]

[0105] in, Let r represent the class matching error of the j-th sample relation category, and r represent the number of sample relation categories. Let represent the matching error of the kth bounding box, h be the number of bounding boxes, β1 represent the first weight of the category matching error, and β2 represent the second weight of the matching error of the kth bounding box.

[0106] Specifically, in this embodiment, the value of h is 2, and the relationship categories can include r categories such as friends, spouses, and relatives.

[0107] In this embodiment, both the first weight and the second weight are set to 0.5. It should be noted that the implementer can adjust the values ​​of the first weight and the second weight according to the actual situation. For example, the first weight can be adjusted to 0.3 and the second weight can be adjusted to 0.7, so as to pay more attention to the accuracy of category classification.

[0108] This embodiment uses a weighted approach to calculate the overall matching error by combining category matching error and bounding box matching error. This provides an effective fitting direction for training the relationship recognition model and improves the accuracy of the trained relationship recognition model.

[0109] The above steps involve determining the overall matching error based on the category matching error, the first bounding box matching error, and the second bounding box matching error. Then, based on the overall matching error of all label vectors, the feature extraction model and the prediction model are trained. The trained feature extraction model and the trained prediction model are combined to form the trained relationship recognition model. The feature extraction model and the prediction model are jointly trained end-to-end without human intervention. During training, the learning parameters of the feature extraction model and the prediction model are adjusted, avoiding accuracy errors caused by manual adjustments. This results in more accurate model fitting and, consequently, more accurate relationship recognition.

[0110] This embodiment calculates the category matching error and bounding box matching error between the predicted vector and the label vector of the corresponding sample image, which improves the recognition accuracy of spatial domain association. Based on this, the matching reference vector is determined to calculate the loss function, and the relationship recognition model is trained in an end-to-end manner to avoid manual intervention in model tuning, reduce the influence of subjective factors, improve the fitting effect of the relationship recognition model, and thus improve the accuracy of the relationship recognition model.

[0111] See Figure 3 This is a flowchart illustrating a training method for an end-to-end relationship recognition model provided in Embodiment 2 of the present invention. In this training method, a preset matching algorithm is used to match N prediction vectors with at least one label vector, and the prediction vector with the highest matching degree to any label vector is determined as the reference vector for the corresponding label vector, including:

[0112] Step S301: Using the Hungarian algorithm, the N predicted vectors are matched with at least one label vector to obtain N matching pairs;

[0113] Step S302: For any matching pair, calculate the overall matching error of the matching pair, and iterate through N matching pairs to obtain N overall matching errors;

[0114] Step S303: For any label vector, determine the prediction vector corresponding to the minimum value of the overall matching error of the matching pair containing the label vector. This prediction vector is the one with the highest degree of matching with the label vector. The prediction vector with the highest degree of matching is used as the reference vector of the corresponding label vector.

[0115] Among them, the Hungarian algorithm can be used for binary matching. In order to improve the matching efficiency, after obtaining N matching pairs, the prediction vector and label vector within the matching pair are directly used to calculate the overall matching error, and N overall matching errors are obtained.

[0116] The smaller the matching error, the higher the degree of matching between the predicted vector and the label vector. The minimum value among the N overall matching errors can indicate that the predicted vector used for calculation is close enough to the label vector. Therefore, the predicted vector corresponding to the minimum value among the N overall matching errors is directly taken as the predicted vector with the highest degree of matching, which is also the reference vector.

[0117] Specifically, when calculating the overall matching error, the second weight of the bounding box matching error can be set to a larger value, for example, 0.8 in this embodiment, so that the matching process pays more attention to the distance between the sample bounding box and the label bounding box.

[0118] This embodiment uses the Hungarian algorithm to match N predicted vectors with at least one label vector, and directly calculates the overall matching loss based on the matched pairs to determine the overall matching loss used for training the relation recognition model. This simplifies the relation recognition model training process, improves the efficiency of relation recognition model training, and ensures the accuracy of relation recognition model training.

[0119] Embodiment 3 of the present invention provides an end-to-end relationship identification method, which includes:

[0120] Acquire the image to be identified, which contains at least two objects to be identified;

[0121] The image to be identified is input into the trained relationship recognition model to obtain P relationship recognition results. The relationship recognition results include the bounding boxes of the two objects to be identified and the recognition relationship category between the two objects to be identified.

[0122] Where P is an integer greater than zero, the trained relationship recognition model is an end-to-end model, that is, after the image to be recognized is acquired, it is directly input into the trained relationship recognition model to obtain the relationship recognition result. The trained relationship recognition model is obtained based on the training method of the end-to-end relationship recognition model provided in the above embodiment one.

[0123] For example, in targeted advertising scenarios, the relationship categories identified through image recognition can determine the corresponding content to be advertised. For instance, if the relationship category of two individuals is identified as "married couple," it can assist in pushing couple-related products for precise targeting. In smart security scenarios, the relationship categories identified through image recognition, combined with pre-deployed sensors, can enable security strategies such as access control follow-up recognition and automatic access control opening and closing. For example, if the relationship category of two individuals is identified as "relatives" or "married couple," and the sensor detects that one individual has already passed through the access control while the other has not, the access control can be opened directly without a verification process. Conversely, if the relationship category of two individuals is not a preset category, after the sensor detects that one individual has already passed through the access control, the access control will be automatically closed, waiting for the other individual to complete the verification process before opening the access control.

[0124] This embodiment uses an end-to-end trained relationship recognition model to process the image to be recognized, eliminating the need for preprocessing to obtain bounding boxes before relationship recognition. This improves the deployment efficiency and recognition efficiency of the trained relationship recognition model, thus enhancing its practicality.

[0125] Corresponding to the training method of the end-to-end relationship recognition model in the above embodiment, Figure 4 This diagram illustrates the structural block diagram of the training device for the end-to-end relationship recognition model provided in Embodiment 4 of the present invention. The training device is applied to a client, and the computer device corresponding to the client connects to the server to obtain sample images and their label vectors. The client deploys a feature extraction model and a prediction model. The feature extraction model can be used to extract feature information from the sample images, and the prediction model can be used to predict social relationship categories based on the feature information of the sample images. For ease of explanation, only the parts relevant to the embodiments of the present invention are shown.

[0126] See Figure 4 The training device includes:

[0127] Feature extraction module 41 is used to obtain sample images and at least one corresponding label vector. The sample images are input into the feature extraction model for feature extraction to obtain sample features. The label vector includes a first label bounding box, a second label bounding box, and a label relationship category.

[0128] The vector prediction module 42 is used to input sample features into the prediction model to obtain N prediction vectors. The prediction vectors include a first sample bounding box, a second sample bounding box, and a sample relationship category, where N is an integer greater than zero.

[0129] The vector matching module 43 is used to match N prediction vectors with at least one label vector using a preset matching algorithm, and determine the prediction vector with the highest matching degree with any label vector as the reference vector of the corresponding label vector.

[0130] The first calculation module 44 is used to traverse each label vector and calculate the category matching error between the sample relationship category in the reference vector and the label relationship category in the label vector;

[0131] The second calculation module 45 is used to calculate the first bounding box matching error between the first sample bounding box in the reference vector and the first label bounding box in the label vector, and to calculate the second bounding box matching error between the second sample bounding box in the reference vector and the second label bounding box in the label vector.

[0132] The model training module 46 is used to determine the overall matching error of the label vector based on the category matching error, the first bounding box matching error, and the second bounding box matching error. Based on the overall matching error of all label vectors, the feature extraction model and the prediction model are trained, and the trained feature extraction model and the trained prediction model are combined to form the trained relationship recognition model.

[0133] Optionally, the feature extraction model includes a semantic extraction layer, an encoder, and a decoder;

[0134] The aforementioned feature extraction module 41 includes:

[0135] The semantic extraction unit is used to input the sample image into the semantic extraction layer to obtain the semantic feature vector. The first vector and the semantic feature vector are multiplied together to obtain the value vector.

[0136] The vector calculation unit is used to encode the sample image using a preset encoding method to obtain the position code, add the semantic feature vector to the position code point by point, and multiply the sum by the preset second vector and the preset third vector respectively to obtain the query vector and the key value vector.

[0137] The global extraction unit is used to input the value vector, query vector, and key vector into the encoder to obtain the global information feature vector.

[0138] The feature decoding unit is used to initialize the preset learnable vector, and input the initialized learnable vector, global information feature vector and position code into the decoder to obtain sample features.

[0139] Optionally, the aforementioned global extraction unit includes:

[0140] The attention subunit is used to multiply the query vector by the transpose of the key vector, divide the result of the multiplication by a preset coefficient, and determine the result of the division as the attention vector.

[0141] The normalization subunit is used to normalize the attention vector using a normalization exponential function, multiply the normalization result by the value vector, and input the multiplication result into the encoder.

[0142] Optionally, the semantic extraction unit mentioned above includes:

[0143] The initial extraction subunit is used to input the sample image into the semantic extraction layer to obtain the initial feature vector;

[0144] The dimensionality reduction subunit is used to reduce the dimensionality of the initial feature vector using a preset convolutional layer, and then flatten the dimensionality reduction result to determine the result of the flattening operation as a semantic feature vector.

[0145] Optionally, the vector matching module 43 mentioned above includes:

[0146] The matching pair acquisition unit is used to use the Hungarian algorithm to match N prediction vectors with at least one label vector to obtain N matching pairs;

[0147] The error traversal unit is used to calculate the overall matching error of any matching pair. It traverses N matching pairs to obtain N overall matching errors.

[0148] The reference vector determination unit is used to determine, for any label vector, the prediction vector corresponding to the minimum value of the overall matching error of the matching pair containing the label vector, which is the prediction vector with the highest degree of matching with the label vector, and the prediction vector with the highest degree of matching is used as the reference vector of the corresponding label vector.

[0149] Optionally, the category matching error corresponds to the first weight, and the first bounding box matching error and the second bounding box matching error correspond to the second weight;

[0150] The above model training module 46 includes:

[0151] The first weighting unit is used to multiply the category matching error by the first weight to obtain the first matching error;

[0152] The second weighting unit is used to multiply the first bounding box matching error and the second bounding box matching error by the second weight respectively and then add them together to obtain the second matching error;

[0153] The summation calculation unit is used to determine the sum of the first matching error and the second matching error as the overall matching error.

[0154] It should be noted that the information interaction and execution process between the above modules, units, and sub-units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0155] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 5 of the present invention. Figure 5 As shown, the computer device of this embodiment includes: at least one processor ( Figure 5 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, which, when executed by the processor, implements the steps in any of the above-described training method embodiments.

[0156] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0157] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0158] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0159] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0160] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0161] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0162] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0163] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0164] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A training method for an end-to-end relationship recognition model, characterized in that, The training method includes: A sample image and its corresponding at least one label vector are obtained. The sample image is input into a feature extraction model for feature extraction to obtain sample features. The label vector includes a first label bounding box, a second label bounding box, and a label relationship category. The sample features are input into the prediction model to obtain N prediction vectors. The prediction vectors include a first sample bounding box, a second sample bounding box, and a sample relationship category, where N is an integer greater than zero. A preset matching algorithm is used to match the N prediction vectors with the at least one label vector, and the prediction vector with the highest matching degree with any label vector is determined as the reference vector of the corresponding label vector. Iterate through each label vector and calculate the category matching error between the sample relationship category in the reference vector and the label relationship category in the label vector; Calculate the first bounding box matching error between the first sample bounding box in the reference vector and the first label bounding box in the label vector, and calculate the second bounding box matching error between the second sample bounding box in the reference vector and the second label bounding box in the label vector; Based on the category matching error, the first bounding box matching error, and the second bounding box matching error, the overall matching error of the label vector is determined. Based on the overall matching error of all label vectors, the feature extraction model and the prediction model are trained, and the trained feature extraction model and the trained prediction model are combined to form a trained relationship recognition model.

2. The training method according to claim 1, characterized in that, The feature extraction model includes a semantic extraction layer, an encoder, and a decoder; The step of inputting the sample image into the feature extraction model for feature extraction to obtain sample features includes: The sample image is input into the semantic extraction layer to obtain a semantic feature vector. The semantic feature vector is multiplied by a preset first vector to obtain a value vector. The sample image is encoded using a preset encoding method to obtain a position code. The semantic feature vector is added to the position code point by point. The addition result is multiplied by a preset second vector and a preset third vector to obtain a query vector and a key value vector. The value vector, the query vector, and the key vector are input into the encoder to obtain the global information feature vector; The preset learnable vector is initialized, and the initialized learnable vector, the global information feature vector, and the position encoding are input into the decoder to obtain the sample features.

3. The training method according to claim 2, characterized in that, The step of inputting the value vector, the query vector, and the key vector into the encoder includes: Multiply the query vector by the transpose of the key value vector, divide the result of the multiplication by a preset coefficient, and determine the result of the division as the attention vector; The attention vector is normalized using a normalized exponential function, the normalized result is multiplied by the value vector, and the multiplication result is input into the encoder.

4. The training method according to claim 2, characterized in that, The step of inputting the sample image into the semantic extraction layer to obtain the semantic feature vector includes: The sample image is input into the semantic extraction layer to obtain the initial feature vector; The initial feature vector is reduced in dimensionality using a preset convolutional layer, and the dimensionality reduction result is flattened to determine the result of the flattening operation as the semantic feature vector.

5. The training method according to claim 1, characterized in that, The step of using a preset matching algorithm to match the N predicted vectors with the at least one label vector, and determining the predicted vector with the highest matching degree to any label vector as the reference vector for the corresponding label vector, includes: The Hungarian algorithm is used to match the N predicted vectors with the at least one label vector to obtain N matching pairs; For any matching pair, calculate the overall matching error of the matching pair, and iterate through the N matching pairs to obtain N overall matching errors; For any label vector, determine the prediction vector corresponding to the minimum value of the overall matching error of the matching pair containing the label vector, which is the prediction vector with the highest matching degree with the label vector, and use the prediction vector with the highest matching degree as the reference vector corresponding to the label vector.

6. The training method according to claim 1, characterized in that, The category matching error corresponds to the first weight, and the first bounding box matching error and the second bounding box matching error correspond to the second weight; The step of determining the overall matching error of the label vector based on the category matching error, the first bounding box matching error, and the second bounding box matching error includes: The first matching error is obtained by multiplying the category matching error by the first weight; The first bounding box matching error and the second bounding box matching error are multiplied by the second weight respectively, and then added together to obtain the second matching error; The sum of the first matching error and the second matching error is determined as the overall matching error.

7. A method for identifying end-to-end relationships, characterized in that, The end-to-end relationship identification method includes: Acquire an image to be identified, wherein the image to be identified contains at least two objects to be identified; The image to be identified is input into the trained relation recognition model to obtain P relation recognition results. The relation recognition results include the bounding boxes of the two objects to be identified and the recognition relation category between the two objects to be identified, where P is an integer greater than zero. The trained relation recognition model is obtained based on the training method of the end-to-end relation recognition model according to any one of claims 1-6.

8. A training device for an end-to-end relationship recognition model, characterized in that, The training device includes: The feature extraction module is used to acquire a sample image and its corresponding at least one label vector, input the sample image into the feature extraction model for feature extraction, and obtain sample features. The label vector includes a first label bounding box, a second label bounding box, and a label relationship category. The vector prediction module is used to input the sample features into the prediction model to obtain N prediction vectors. The prediction vectors include a first sample bounding box, a second sample bounding box, and a sample relationship category, where N is an integer greater than zero. The vector matching module is used to match the N prediction vectors with the at least one label vector using a preset matching algorithm, and determine the prediction vector with the highest matching degree with any label vector as the reference vector of the corresponding label vector. The first calculation module is used to traverse each label vector and calculate the category matching error between the sample relationship category in the reference vector and the label relationship category in the label vector; The second calculation module is used to calculate the first bounding box matching error between the first sample bounding box in the reference vector and the first label bounding box in the label vector, and to calculate the second bounding box matching error between the second sample bounding box in the reference vector and the second label bounding box in the label vector. The model training module is used to determine the overall matching error based on the category matching error, the first bounding box matching error, and the second bounding box matching error, and to train the feature extraction model and the prediction model based on the overall matching error of all label vectors, so as to determine that the trained feature extraction model and the trained prediction model constitute a trained relationship recognition model.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method as described in any one of claims 1 to 7.

Images