Image-based five-element feature detection model training method, system and device

Abstract

The application discloses an image-based five-element feature detection model training method, system and device. The method comprises the following steps: acquiring a to-be-trained image; inputting the to-be-trained image into an initial model to perform visual feature extraction processing on the to-be-trained image, and obtaining initial three-element features of each frame of to-be-trained picture in the to-be-trained image; performing element construction based on the initial three-element features to obtain initial five-element features; performing space-time aggregation processing on the initial five-element features to obtain target five-element features of the corresponding to-be-trained picture; determining a prediction confidence of the target five-element features, and adjusting the initial model based on the prediction confidence and the target five-element features to obtain a target model. The problems that the binary-element feature relationship output by the existing model is difficult to fully reflect the complex operation relationship that a subject uses a tool to operate an object in an image, and the recognition accuracy of the element feature relationship of the model for the picture is reduced are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a five-element feature detection model training method, system and device based on images. BACKGROUND

[0002] With the development of computer vision technology, video-based scene understanding has become an important research direction. When traditional models identify image features, they usually only consider the binary element feature relationship between the "subject-object" to reflect the relationship between the subject and the object in the image.

[0003] However, when there is a dynamic scene in the image in which the subject uses a tool to operate on the object, the binary element feature relationship output by the model is difficult to fully reflect the complex operation relationship in the dynamic scene, reducing the recognition accuracy of the model for the element feature relationship of the picture. SUMMARY

[0004] In order to overcome the problem that the binary element feature relationship output by the existing model is difficult to fully reflect the complex operation relationship in which the subject uses a tool to operate on the object in the image, and reduce the recognition accuracy of the model for the element feature relationship of the picture, the present application provides a five-element feature detection model training method, system and device based on images.

[0005] In a first aspect, to solve the above technical problems, the present application provides a five-element feature detection model training method based on images, comprising:

[0006] obtaining a to-be-trained image;

[0007] inputting the to-be-trained image into an initial model to perform visual feature extraction processing on the to-be-trained image, and obtaining an initial three-element feature of each frame of to-be-trained picture in the to-be-trained image, the elements of the initial three-element feature including an initial subject, an initial tool and an initial object;

[0008] performing element construction based on the initial three-element feature to obtain an initial five-element feature, the elements of the initial five-element feature including the initial subject, the initial tool, the initial object, an initial action and an initial element relationship;

[0009] performing spatio-temporal aggregation processing on the initial five-element feature to obtain a target five-element feature corresponding to the to-be-trained picture, the elements of the target five-element feature including a target subject, a target tool, a target object, a target action and a target element relationship;

[0010] determining a prediction confidence of the target five-element feature, and adjusting the initial model based on the prediction confidence and the target five-element feature to obtain a target model.

[0011] Further, the element construction based on the initial three-element feature to obtain the initial five-element feature comprises:

[0012] Detect the initial ternary feature by using a preset ternary interaction action detection method to obtain an initial action formed by the element interaction in the ternary feature;

[0013] Input the initial ternary feature into a preset multi-head attention mechanism for processing to obtain an initial element relationship between the elements in the ternary feature;

[0014] Form an initial pentuple feature based on the initial ternary feature, the initial action and the initial element relationship.

[0015] Further, the initial pentuple feature is subjected to spatio-temporal aggregation processing to obtain a target pentuple feature of the corresponding training picture, including:

[0016] Based on the initial pentuple feature, the spatial context aggregation processing is performed to obtain a fusion pentuple feature of the corresponding training picture;

[0017] Based on the fusion pentuple feature, the temporal context aggregation processing is performed to obtain a target pentuple feature of the training picture.

[0018] Further, based on the initial pentuple feature, the spatial context aggregation processing is performed to obtain a fusion pentuple feature of the corresponding training picture, including:

[0019] The feature similarity of each element in the initial pentuple feature of the current frame training picture is calculated by using the initial pentuple feature of the current frame training picture and the previous frame training picture respectively;

[0020] Based on the feature similarity, the dynamic weight of the corresponding element is calculated;

[0021] The initial pentuple feature is weighted and aggregated by using the dynamic weight to obtain a transition pentuple feature;

[0022] The transition pentuple feature is subjected to multi-scale feature fusion to obtain a fusion pentuple feature of the current frame training picture.

[0023] Further, the transition pentuple feature is subjected to multi-scale feature fusion to obtain a fusion pentuple feature of the current frame training picture, including:

[0024] The transition pentuple feature is input into a preset multi-scale attention mechanism to extract the scale feature of each element in the transition pentuple feature at each scale;

[0025] Based on the scale feature, the scale attention weight of the corresponding element is calculated;

[0026] Based on the scale attention weight and the scale feature, the fusion feature of the corresponding element is obtained by weighted aggregation;

[0027] The five fusion features of the transition five-element feature are aggregated to obtain a fusion five-element feature of the current frame to-be-trained picture.

[0028] Further, the fusion five-element feature is subjected to time context aggregation processing to obtain a target five-element feature of the to-be-trained picture, including:

[0029] The fusion five-element feature of the current frame to-be-trained picture and the plurality of fusion five-element features of the previous frame to-be-trained picture are respectively subjected to pair combination to obtain a plurality of pairs of five-element feature groups;

[0030] A pair matching confidence corresponding to the five-element feature group is obtained.

[0031] The feature groups corresponding to a preset number of pair matching confidences in a front rank are determined as target feature groups, and the ranking represents descending order sorting of the plurality of pair matching confidences.

[0032] The fusion five-element feature of the previous frame to-be-trained picture in the target feature group is used for feature fusion of the fusion five-element feature of the current frame to-be-trained picture to obtain a target five-element feature of the to-be-trained picture.

[0033] Further, a prediction confidence of the target five-element feature is determined, including:

[0034] A prediction class and a class probability of the prediction class of an element are obtained by using a preset classification head to predict the class of the element in the target five-element feature.

[0035] Based on the five class probabilities of the target five-element feature, a prediction confidence of the target five-element feature is calculated.

[0036] Further, based on the prediction confidence and the target five-element feature, an initial model is adjusted to obtain a target model, including:

[0037] A classification loss of the corresponding element is calculated by using a preset classification loss function based on the prediction class corresponding to the prediction confidence and a real class corresponding to the prediction class.

[0038] A prediction bounding box of the target subject, the target tool and the target object in the target five-element feature is regressed to obtain a prediction bounding box of the target five-element feature in the corresponding to-be-trained picture.

[0039] A regression loss of the target five-element feature is calculated by using a preset bounding box regression loss function based on the prediction bounding box and a real bounding box.

[0040] A target loss of the target five-element feature is calculated based on the classification loss and the regression loss.

[0041] A target model is obtained by adjusting network parameters in an initial model based on the target loss.

[0042] In a second aspect, the present application also provides an image-based five-element feature detection model training system, comprising:

[0043] An acquisition module is configured to acquire a to-be-trained image.

[0044] A feature extraction module is configured to input the to-be-trained image into an initial model, perform visual feature extraction processing on the to-be-trained image, and obtain initial three-element features of each frame of to-be-trained picture in the to-be-trained image, wherein elements of the initial three-element features include an initial subject, an initial tool and an initial object.

[0045] An element construction module is configured to perform element construction based on the initial three-element features, and obtain initial five-element features, wherein elements of the initial five-element features include the initial subject, the initial tool, the initial object, an initial action and an initial element relationship.

[0046] A space-time aggregation module is configured to perform space-time aggregation processing on the initial five-element features, and obtain target five-element features of the corresponding to-be-trained picture, wherein elements of the target five-element features include a target subject, a target tool, a target object, a target action and a target element relationship.

[0047] A model adjustment module is configured to determine a prediction confidence of the target five-element features, and adjust the initial model based on the prediction confidence and the target five-element features, to obtain a target model.

[0048] In a third aspect, the present application also provides a computing device, comprising a memory, a processor and a program stored in the memory and running on the processor, wherein the processor implements the steps of the image-based five-element feature detection model training method when executing the program.

[0049] In a fourth aspect, the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores instructions, and when the instructions run on a terminal device, the terminal device performs the steps of the image-based five-element feature detection model training method.

[0050] The beneficial effects of the present application are: first, by performing visual feature processing on the to-be-trained image in the initial model, obtaining the initial ternary features of each frame of to-be-trained picture in the to-be-trained image, and performing element construction based on the initial ternary features, obtaining the initial pentad features, the elements of the initial pentad features include initial subject, initial tool, initial object, initial action and initial element relationship. Second, the spatio-temporal aggregation processing of the initial pentad features can improve the expression ability of the obtained target pentad features to the context information. Then, the initial model is adjusted based on the prediction confidence of the target pentad features and the target pentad features. The target model is obtained. In this way, since the expression ability of the target pentad features of the extracted picture to the context information is strong in the model training process, the target model obtained by training can combine the subject, tool, object, action, element relationship and picture context information in the input picture to output the target pentad features which can fully reflect the complex operation relationship of the subject using the tool to operate the object existing in the input picture, thereby improving the recognition accuracy of the target model to the element feature relationship in the picture. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A flowchart of an image-based pentad feature detection model training method according to an example embodiment of the present application is shown.

[0052] Figure 2 A structural diagram of an image-based pentad feature detection model training system according to an example embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] The following examples are further explanations and supplements of the present application and do not constitute any limitation on the present application.

[0054] The following describes an image-based pentad feature detection model training method, system and device according to an embodiment of the present application with reference to the accompanying drawings.

[0055] The image-based pentad feature detection model training method provided by the embodiment of the present application can be specifically executed by a server. It should be noted that the server can be an independent server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms, etc. Basic cloud computing services are not limited here.

[0056] Please refer to Figure 1 , Figure 1 An image-based pentad feature detection model training method according to an example embodiment of the present application is shown. Figure 1As shown, the present application provides an image-based five-element feature detection model training method, comprising:

[0057] S11, acquiring a to-be-trained image;

[0058] S12, inputting the to-be-trained image into an initial model to perform visual feature extraction processing on the to-be-trained image, to obtain initial three-element features of each frame of to-be-trained picture in the to-be-trained image, the elements of the initial three-element features including an initial subject, an initial tool and an initial object;

[0059] S13, performing element construction based on the initial three-element features to obtain initial five-element features, the elements of the initial five-element features including the initial subject, the initial tool, the initial object, an initial action and an initial element relationship;

[0060] S14, performing spatio-temporal aggregation processing on the initial five-element features to obtain target five-element features corresponding to the to-be-trained picture, the elements of the target five-element features including a target subject, a target tool, a target object, a target action and a target element relationship;

[0061] S15, determining a prediction confidence of the target five-element features, and adjusting the initial model based on the prediction confidence and the target five-element features to obtain a target model.

[0062] The image-based five-element feature detection model training method provided in this embodiment of the present application first performs visual feature processing on the to-be-trained image in the initial model to obtain initial three-element features of each frame of to-be-trained picture in the to-be-trained image, and performs element construction based on the initial three-element features to obtain initial five-element features, the elements of the initial five-element features including the initial subject, the initial tool, the initial object, the initial action and the initial element relationship. Secondly, performing spatio-temporal aggregation processing on the initial five-element features can improve the expression ability of the obtained target five-element features to context information. Then, adjusting the initial model based on the prediction confidence of the target five-element features and the target five-element features to obtain a target model. In this way, since the target five-element features of the extracted picture have strong expression ability to context information in the model training process, the target model obtained by training can combine the subject, the tool, the object, the action, the element relationship in the input picture and the picture context information to output the target five-element features that can fully reflect the complex operation relationship of the subject using the tool to operate the object in the input picture, thereby improving the recognition accuracy of the target model to the element feature relationship in the picture.

[0063] In this embodiment, the to-be-trained image is subjected to visual feature extraction processing to obtain initial three-element features of each frame of to-be-trained picture in the to-be-trained image, and the specific steps are as follows:

[0064] visual feature extraction processing is performed on the to-be-trained image to obtain visual features of each frame of to-be-trained picture in the to-be-trained image, which can be denoted as: F = {f_T, f_{T1},..., f_{Tn}}, wherein f_i∈R^{H×W×d_{model}}, H and W represent the height and width of the to-be-trained picture respectively, and d_{model} represents the feature dimension;

[0065] The plurality of visual features are classified to obtain an initial subject set, an initial tool set and an initial object set, the initial subject set containing a plurality of initial subjects, the initial tool set containing a plurality of initial tools, and the initial object set containing a plurality of initial objects.

[0066] Based on the initial subject set, the initial tool set and the initial object set, feature combination is performed to obtain a plurality of initial triple features of the to-be-trained picture, and an element of the initial triple feature is an initial subject in the initial subject set, an initial tool in the initial tool set and an initial object in the initial object set.

[0067] Optionally, based on the initial triple feature, element construction is performed to obtain an initial quintuple feature, which includes:

[0068] The initial triple feature is detected by using a preset triple interaction action detection method to obtain an initial action formed by element interaction in the triple feature;

[0069] The initial triple feature is input into a preset multi-head attention mechanism for processing to obtain an initial element relationship between the elements of the triple feature.

[0070] Based on the initial triple feature, the initial action and the initial element relationship, an initial quintuple feature is formed.

[0071] In the embodiment provided in the present application, the initial triple feature is detected by using a preset triple interaction action detection method to obtain an initial action formed by element interaction in the triple feature, and the initial triple feature is input into a preset multi-head attention mechanism for processing to obtain an initial element relationship between the elements of the triple feature, forming an initial quintuple feature including the initial triple feature, the initial action and the initial element relationship, which can reflect the complex operation relationship of the subject using the tool to operate the object in the corresponding to-be-trained picture, thereby improving the learning ability of the initial model for the complex operation relationship in the training process, and further improving the recognition accuracy of the element feature relationship in the picture by the target model obtained through subsequent training.

[0072] In the embodiment, the specific steps of the preset triple interaction action detection method can be referred to the technical solution of the patent application with the granted announcement of CN117953590B and the subject name of a triple interaction action detection method, system, device and medium.

[0073] The preset multi-head attention mechanism can be represented as: wherein, Multi-Head, , , represent query, key and value vectors respectively, represent the weight matrix of the head.

[0074] The initial five-element feature formed can be represented as wherein, represent the initial five-element feature, represent the initial subject, represent the initial tool, represent the initial object, represent the initial action, represent the initial element relationship.

[0075] Optionally, the initial five-element feature is subjected to spatio-temporal aggregation processing to obtain a target five-element feature of the picture to be trained, including:

[0076] Based on the initial five-element feature, spatial context aggregation processing is performed to obtain a fusion five-element feature of the picture to be trained;

[0077] Based on the fusion five-element feature, temporal context aggregation processing is performed to obtain the target five-element feature of the picture to be trained.

[0078] In the embodiment provided in the present application, first, based on the initial five-element feature, spatial context aggregation processing is performed to learn the context information having a spatial structure relationship with the initial five-element feature, thereby improving the expression ability of the obtained fusion five-element feature to the spatial context information. Secondly, based on the fusion five-element feature, temporal context aggregation processing is performed to learn the context information having a time sequence relationship with the fusion five-element feature, thereby improving the expression ability of the obtained target five-element feature to the temporal context information. In this way, the target five-element feature has strong expression ability to the temporal and spatial context information, so that the model can comprehensively learn the complex operation relationship of the subject using the tool to operate the object in the training process, thereby improving the recognition accuracy of the element feature relationship in the picture by the target model obtained in the subsequent training.

[0079] Optionally, based on the initial five-element feature, spatial context aggregation processing is performed to obtain a fusion five-element feature of the picture to be trained, including:

[0080] The feature similarity of each element in the initial five-element feature of the current frame to-be-trained picture is calculated using the initial five-element feature of the current frame to-be-trained picture and the initial five-element feature of the last frame to-be-trained picture respectively.

[0081] The calculation formula of the feature similarity is as follows:

[0082]

[0083] wherein, represents the feature vector of an element in the initial five-element feature of the last frame to-be-trained picture, represents the feature vector of an element in the initial five-element feature of the current frame to-be-trained picture, and represents the vector norm of ; represents the vector norm of ;

[0084] The dynamic weight of the corresponding element is calculated based on the feature similarity.

[0085] The calculation formula of the dynamic weight is as follows:

[0086] ;

[0087] wherein, represents the dynamic weight of , represents the feature vector of the corresponding element in the initial five-element feature of the first to-be-trained picture,

[0088] The initial five-element feature is weighted and aggregated using the dynamic weight to obtain a transition five-element feature.

[0089] The calculation formula of the feature vector of an element in the transition five-element feature is as follows:

[0090] ;

[0091] wherein, represents the feature vector of an element in the transition five-element feature, represents the feature vector of an element in the initial five-element feature, S represents an initial subject, T represents an initial tool, O represents an initial object, A represents an initial action, and R represents an initial element relationship.

[0092] The transition five-element feature is subjected to multi-scale feature fusion to obtain a fusion five-element feature of the current frame to-be-trained picture.

[0093] ​​In the embodiment provided in the present application, first, the feature similarity of each element in the initial five-element feature of the current frame to be trained picture is calculated using the initial five-element feature of the current frame to be trained picture and the initial five-element feature of the last frame to be trained picture respectively, and the dynamic weight of the corresponding element is calculated based on the feature similarity. The transition five-element feature that can reflect the spatial structure relationship between elements can be obtained by weighting and aggregating the initial five-element feature using the dynamic weight. The expression ability of the transition five-element feature on the spatial context information is enhanced. Then, the transition five-element feature is subjected to multi-scale feature fusion. The feature information of the transition five-element feature in different scale spaces is fully utilized. The expression ability of the obtained fusion five-element feature on the spatial context information is further enhanced. Therefore, the learning ability of the model on the spatial context information of the complex operation relationship in the training process can be improved, and the recognition accuracy of the target model obtained by subsequent training on the feature relationship of elements in the picture can be improved.

[0094] In an example embodiment provided in the present application, first, the feature similarity (which can be cosine similarity) of the same element is calculated using the last frame to be trained picture and the current frame to be trained picture. This can meet the demand for real-time and continuous analysis. In video monitoring, motion capture and other scenes, it is necessary to quickly grasp the content changes of video images. For example, in video monitoring, comparing the feature similarity of the main body (such as pedestrians, vehicles) in the previous frame and the current frame in real time can timely discover the behaviors such as movement and posture change of the main body; in motion capture, by comparing the cosine similarity of the feature vectors of the key parts (such as joints) of the athlete's body in adjacent frames, the action continuity can be judged. Moreover, this method can reduce the computational complexity and delay, and does not need to wait for the next frame data, which is more feasible on resource-limited devices. At the same time, the time interval between adjacent frames is short, and the influence of external interference (such as sudden change of light, noise) is small, so the data is more stable and reliable. This method also conforms to the viewing habit of comparing and understanding the current frame picture in the video image with the last frame picture just watched, and improves the explainability and practicality of the analysis result.

[0095] In actual scenes, the importance of the features of each element in the five-element feature for the final feature representation is not fixed, but changes with the changes of scenes and tasks. Based on this, a dynamic weight is introduced, which can flexibly adjust the contribution degree of each element according to the relationship between the features. The correlation strength between the elements in the five-element group is dynamically measured using a similarity function, and the dynamic weight of feature interaction is adjusted according to the correlation strength, which can improve the expression ability of the context information, so as to better capture the complex spatial dependency relationship between the subjects, tools, objects, actions and relationships in the five-element group.

[0096] Specifically, first, the cosine similarity between the feature vectors is calculated to obtain dynamic weights, and then the five-element features are weighted and aggregated based on these dynamic weights to calculate the transition five-element features. The core role of the dynamic weight is to adaptively adjust the importance of each element feature in the five-element feature. In different scenarios, the contribution of each element feature to the overall feature is different. For example, in the video scene of analyzing the interaction action of the character and the object, if the character action of the current frame changes significantly, the dynamic weight corresponding to the action feature will increase, and the influence of the action feature on the aggregated feature will be more prominent. At the same time, the dynamic weight can also capture the context dependency relationship between the elements in the five-element feature. It reflects the similarity between the current element feature and other element features, so that the aggregated feature can better integrate the information between different elements. Just like in natural language processing, the meaning of a word is influenced by surrounding words, the dynamic weight can reflect this dependency, allowing the aggregated feature to more accurately express the overall semantic. In addition, the dynamic weight can enhance the expression ability of the feature, highlight the key information, suppress irrelevant or secondary information, and make the aggregated transition five-element feature more discriminative and representative, thereby improving the understanding and processing ability of the model for complex scenes and tasks, such as improving the recognition accuracy of target objects and their related actions and attributes in image recognition tasks.

[0097] Optionally, the transition five-element feature is subjected to multi-scale feature fusion to obtain a fusion five-element feature of the current frame of the picture to be trained, comprising:

[0098] The transition five-element feature is input into a preset multi-scale attention mechanism to extract the scale feature of each element in the transition five-element feature at each scale;

[0099] Based on the scale feature, the scale attention weight of the corresponding element is calculated;

[0100] Based on the scale attention weight and the scale feature, the fusion feature of the corresponding element is obtained by weighted aggregation;

[0101] The five fusion features of the transition five-element feature are subjected to feature aggregation to obtain a fusion five-element feature of the current frame of the picture to be trained.

[0102] In the embodiment provided in the present application, first, the transition five-element feature is input into the preset multi-scale attention mechanism, the scale feature of each element in the transition five-element feature at each scale is extracted, and the scale attention weight of the element is calculated based on the scale feature, and the scale features are weighted and aggregated to obtain the fusion feature of the corresponding element, so that the fusion feature of each element can have the expression ability of the scale space context information. The five fusion features of the transition five-element feature are aggregated to make the obtained fusion five-element feature have the expression ability of the scale space context information, so as to improve the learning ability of the model to the spatial context information of the complex operation relationship in the training process, and then improve the recognition accuracy of the target model to the element feature relationship in the picture obtained by subsequent training.

[0103] In an example embodiment provided in the present application, the transition five-element feature is input into the preset multi-scale attention mechanism, for each element in the transition five-element feature, different scale convolution kernels or different receptive field modules (such as a dilated convolution) are used to extract multi-scale features to obtain the feature of the nth scale. Then, the attention weight of each scale n is calculated , first, the correlation score of each scale feature and the global feature or the task-related feature is calculated , and then the is obtained by using the softmax function. Then, the different scale features of one element of the transition five-element feature are weighted and aggregated according to the attention weight to obtain , , wherein represents the feature vector of the element of the transition five-element feature after weighted aggregation by the multi-scale attention mechanism. Finally, the is substituted into the weighted aggregation formula of the original transition five-element feature to obtain:

[0104] ;

[0105] , wherein, represents the fusion five-element feature, n represents the scale index, which is used to distinguish the features of the element at different scales, represents the attention weight of the feature of the element of the transition five-element feature at the nth scale, which is obtained by performing softmax normalization on the correlation score of the different scale features, represents the feature of the element of the transition five-element feature at the nth scale.

[0106] Optionally, the fusion five-element feature is used for time context aggregation processing to obtain the target five-element feature of the picture to be trained, including:

[0107] The fused five-element features of the current frame to-be-trained picture and the plurality of fused five-element features of the previous frame to-be-trained picture are respectively combined in pairs to obtain a plurality of pairs of five-element feature groups;

[0108] A pair matching confidence corresponding to the five-element feature group is obtained;

[0109] The feature group corresponding to the preset number of pair matching confidences in the front of the ranking is determined as a target feature group, and the ranking represents descending sorting of the plurality of pair matching confidences;

[0110] The fused five-element features of the current frame to-be-trained picture are fused with the fused five-element features of the previous frame to-be-trained picture in the target feature group to obtain a target five-element feature of the to-be-trained picture.

[0111] In the embodiment provided in the present application, firstly, the fused five-element features of the current frame to-be-trained picture and the plurality of fused five-element features of the previous frame to-be-trained picture are respectively combined in pairs to obtain a plurality of pairs of five-element feature groups, and a pair matching confidence corresponding to the five-element feature group is obtained. Then, the feature group corresponding to the preset number of pair matching confidences in the front of the ranking is determined as a target feature group, and the fused five-element features of the current frame to-be-trained picture are fused with the fused five-element features of the previous frame to-be-trained picture in the target feature group, so that the feature information of the fused five-element features between the previous time frame and the current time frame that meet the requirement of the reliability degree can be introduced into the fused five-element features of the current time frame. Not only the expression ability of the time context information of the obtained target five-element feature can be improved, but also the accuracy of the feature information of the target five-element feature can be improved, and then the recognition accuracy of the element feature relationship in the picture of the subsequent trained target model can be improved.

[0112] In an example embodiment provided in the present application, when capturing the time dependence across frames, for example, the predicate of “gaze”, “hold” and “drink” which needs multi-frame information, the traditional method relies on explicit track or complex 3D convolution, resulting in redundant calculation and difficult end-to-end optimization, while the Progressively Refined Module (PRM) implicitly aggregates time context through progressive feature screening and interaction, without additional trackers, to achieve efficient time dependence modeling.

[0113] The core idea of the PRM is to dynamically screen high-confidence pair features from the reference frame (previous frame or key frame) and interact with the target frame (current frame) features to gradually enhance the modeling ability of time dependence. First, a spatial context aggregation module (cascade decoder) is used to obtain a pair feature set formed by a plurality of pairs of five-element feature groups of the target frame and the reference frame , and in the i-th step (processing step of time context aggregation processing based on fused five-element features), the pair feature set is screened The function performs sorting and selection, that is, from Filter out the pairs with the highest confidence. (Preset number) target feature groups, and the selected ones The target features in each target feature group interact with the current features of the target frame (current frame) (feature fusion), and the specific calculation process is as follows:

[0114] ;

[0115] ;

[0116] ;

[0117] in, Represents the target's five-element features. Representing a pairwise feature set In the middle, the top results are sorted in descending order based on the pairwise matching confidence scores. A set of high-confidence target features is used to compare the current features of the elements in the target frame (current frame). Interaction (feature fusion) introduces temporal context information from the reference frame (previous frame or keyframe) into the target frame (current frame); This represents the sorting selection function, specifically, in this embodiment, it represents the selection function before sorting in descending order. A high confidence level; This represents the paired features obtained after the target frame (current frame) has been processed by the spatial context aggregation module (cascaded decoder). These features contain the spatial context information of the target frame (current frame) and represent the initial stage of temporal context aggregation. The initial value.

[0118] In summary, PRM progressively refines by dynamically adjusting the number of filters to improve confidence. To balance contextual information and noise, PRM gradually reduces the number of filters. The strategy of (decreasing) simply means that the early steps retain more reference features to capture a wide range of temporal contexts, while the later steps focus on high-confidence features to filter out noise (such as background interference or invalid pairings).

[0119] Optionally, the prediction confidence of the target quintuple feature is determined, including:

[0120] Using a pre-defined classification head, the elements in the target five-element feature are predicted to have a class, and the predicted class of the element and the class probability of the predicted class are obtained.

[0121] The prediction confidence of the target pentagonal feature is calculated based on the five class probabilities of the target pentagonal feature.

[0122] In the embodiment provided in the present application, the preset classification head is used to predict the categories of the elements in the target five-element feature, to obtain the predicted categories and category probabilities of the elements, and based on the five category probabilities of the target five-element feature, the predicted confidence of the target five-element feature is calculated, so as to understand the reliability degree of the target five-element feature, facilitate subsequent model adjustment based on the target five-element feature and the predicted confidence extracted in the training process, improve the reliability of model training, and thus improve the recognition accuracy of the target model obtained in subsequent training on the element feature relationship in the picture.

[0123] In an example embodiment provided in the present application, the preset classification head has five classification branches, including a subject classification head, a tool classification head, an object classification head, an action classification head, and an element relationship classification head, and the category probability includes a subject category probability, a tool category probability, an object category probability, an action category probability, and an element relationship category probability. The relationship between the classification head and the category probability is as follows:

[0124] The subject classification head: input the pair instance feature corresponding to the target five-element feature , and output the subject category probability (cross-entropy loss).

[0125] The action classification head: input the pair relationship feature corresponding to the target five-element feature fused with the space-time information , and output the action category probability (focal loss, processing category imbalance).

[0126] The tool classification head: share the pair instance feature with the subject classification head , and output the tool category probability (supporting the “no tool” category, cross-entropy loss).

[0127] The element relationship classification head: independently model the relationship between the subject / tool and the object, input the target five-element feature, and output the relationship category probability (focal loss).

[0128] The object classification head: same as the design of the subject classification head, input the pair instance feature corresponding to the target five-element feature , and output the object category probability (cross-entropy loss).

[0129] The predicted confidence is obtained by multiplying the five category probabilities: , which can highlight the joint rationality of each element in the target five-element feature.

[0130] In another example embodiment provided in the present application, regression can be performed only for the bounding boxes of the subject and the object, and the coordinate parameters (such as the center point coordinates, the width and height) thereof are output. The regression loss of the regression processing is calculated by weighting and calculating the L1 loss and the GIoU loss, as shown in the following formula (1): The loss is calculated by weighting and calculating the L1 loss and the GIoU loss, as shown in the following formula (1): , to ensure the accuracy of the positioning of the bounding box. In the formula, w1 represents the first preset weight, w1 represents the first preset weight, w2 represents the second preset weight, The loss represents the average value of the absolute error between the predicted value and the true value, and the GIoU loss is a loss function for target detection or instance segmentation, which is improved from the traditional loU (intersection over union). The gradient disappearance problem of loU when there is no overlap is solved by introducing the minimum closed region.

[0131] Optionally, the initial model is adjusted based on the prediction confidence and the target five-element feature to obtain a target model, including:

[0132] A classification loss of each element in the target five-element feature is calculated based on the prediction class corresponding to the prediction confidence and the true class corresponding to the prediction class by using a preset classification loss function;

[0133] The position boundaries of the target subject, the target tool and the target object in the target five-element feature are predicted by regression to obtain a predicted bounding box of the target five-element feature in the corresponding to-be-trained picture;

[0134] A regression loss of the target five-element feature is calculated based on the predicted bounding box and the corresponding true bounding box by using a preset bounding box regression loss function;

[0135] A target loss of the target five-element feature is calculated based on the classification loss and the regression loss;

[0136] The network parameters in the initial model are adjusted based on the target loss to obtain a target model.

[0137] In the embodiment provided in the present application, first, a classification loss of each element in the target five-element feature is calculated based on the prediction class corresponding to the prediction confidence, and a regression loss of the target five-element feature is calculated based on the predicted bounding box obtained by the regression prediction of the target five-element feature. Then, a target loss of the target five-element feature is calculated based on the classification loss and the regression loss, and the network parameters in the initial model are adjusted based on the target loss to obtain a target model. The target model can meet the loss requirement when identifying the five-element feature in the picture, thereby improving the identification accuracy of the target model for the element feature relationship in the picture.

[0138] In an exemplary embodiment provided in this application, the target loss of the target five-element feature is calculated using the matching loss function based on classification loss and regression loss. The matching loss function can optimize the model parameters, enabling the model to have better performance when predicting scene graphs.

[0139] The matching loss function is: ;

[0140] in, Indicates target loss. Represents classification loss. This represents the regression loss weight coefficient. This represents the regression loss.

[0141] In this context, j represents different types of elements. Specifically, s corresponds to the subject, which is the core object in the scene graph that performs actions or has relationships with other elements; r represents the relation, which describes the connection between elements such as the subject, tool, and object; and t represents the tool, which is the item used by the subject when performing actions. The 'o' represents the action, reflecting the subject's behavior; the 'o' represents the object, the recipient of the action. These are the weight coefficients assigned to the classification loss for each element; different The value reflects the difference in importance of the corresponding element in the classification task and can be adjusted according to the specific task requirements and data characteristics. This is the classification loss function, typically using common classification loss metrics such as cross-entropy loss, to measure the deviation between the model's prediction of element j's class and the true label. By weighted summing of the classification losses for these different elements, the model is encouraged to focus more on the classification accuracy of each element during training, thereby improving overall classification performance.

[0142] In the expression, It is a weighting coefficient used to adjust the proportion of regression loss in the total loss. This indicates that the regression loss only applies to the three elements—the subject, the tool, and the object—which have definite spatial locations. This is because accurately locating these elements is crucial in scene graph prediction. It is the bounding box regression loss function, and common ones include smoothing. Loss, IoU (Intersection over Union) loss, etc., are used to measure the difference between the bounding box predicted by the model and the true bounding box. The bounding box regression loss is calculated by summing these three elements and multiplying them by the weights. The model can be guided to learn continuously during the training process to improve the prediction accuracy of the positions of these elements.

[0143] Therefore, the image-based five-element feature detection model training method provided in the application can extract visual relationships from image video sequences and generate detailed scene graphs containing five-element relationships of "subject, relationship, tool, object, and action". Through steps such as visual feature extraction, five-element group generation, spatial and temporal context aggregation, classification and regression, accurate modeling of the five-element relationships in the scene can be achieved, thereby improving the accuracy and efficiency of scene understanding. During model training, the matching loss function is minimized, and the model can simultaneously optimize the classification and regression performance, so that when predicting the scene graph, the model can not only accurately determine the categories of the elements but also accurately determine the positions of the subject, tool, and object, thereby achieving high-quality prediction of the scene graph.

[0144] Referring to Figure 2 , Figure 2 An image-based five-element feature detection model training system according to an example embodiment of the application is shown in Figure 2 As shown in FIG. 2, the image-based five-element feature detection model training system 200 provided in the application includes:

[0145] The acquisition module 201 is configured to acquire a to-be-trained image.

[0146] The feature extraction module 202 is configured to input the to-be-trained image into an initial model, perform visual feature extraction processing on the to-be-trained image, and obtain initial three-element features of each frame of to-be-trained picture in the to-be-trained image. The elements of the initial three-element features include an initial subject, an initial tool, and an initial object.

[0147] The element construction module 203 is configured to perform element construction based on the initial three-element features to obtain initial five-element features. The elements of the initial five-element features include an initial subject, an initial tool, an initial object, an initial action, and an initial element relationship.

[0148] The space-time aggregation module 204 is configured to perform space-time aggregation processing on the initial five-element features to obtain target five-element features corresponding to the to-be-trained picture. The elements of the target five-element features include a target subject, a target tool, a target object, a target action, and a target element relationship.

[0149] The model adjustment module 205 is configured to determine a prediction confidence of the target five-element features and adjust the initial model based on the prediction confidence and the target five-element features to obtain a target model.

[0150] The image-based five-element feature detection model training system 200 of the embodiment provided in the application first performs visual feature processing on the to-be-trained image acquired by the acquisition module 201 in the initial model through the feature extraction module 202, obtains the initial three-element feature of each frame of to-be-trained picture in the to-be-trained image, and performs element construction based on the initial three-element feature by using the element construction module 203 to obtain the initial five-element feature. The elements of the initial five-element feature include the initial subject, the initial tool, the initial object, the initial action and the initial element relationship. Secondly, the initial five-element feature is subjected to spatio-temporal aggregation processing by using the spatio-temporal aggregation module 204, which can improve the expression ability of the obtained target five-element feature to the context information. Then, the initial model is adjusted based on the prediction confidence of the target five-element feature and the target five-element feature by using the model adjustment module 205 to obtain the target model. In this way, since the expression ability of the target five-element feature of the extracted picture to the context information is strong in the model training process, the target model obtained by training can combine the subject, the tool, the object, the action, the element relationship and the picture context information in the input picture to output the target five-element feature which can fully reflect the complex operation relationship that the subject uses the tool to operate the object in the input picture, thereby improving the recognition accuracy of the target model to the element feature relationship in the picture.

[0151] Optionally, the element construction module 203 is specifically used for:

[0152] detecting the initial three-element feature by using a preset three-element interaction action detection method to obtain the initial action formed by the element interaction in the three-element feature;

[0153] inputting the initial three-element feature into a preset multi-head attention mechanism for processing to obtain the initial element relationship between the elements of the three-element feature;

[0154] forming the initial five-element feature based on the initial three-element feature, the initial action and the initial element relationship.

[0155] Optionally, the spatio-temporal aggregation module 204 is specifically used for:

[0156] performing spatial context aggregation processing based on the initial five-element feature to obtain the fusion five-element feature corresponding to the to-be-trained picture;

[0157] performing temporal context aggregation processing based on the fusion five-element feature to obtain the target five-element feature of the to-be-trained picture.

[0158] Optionally, the spatio-temporal aggregation module 204 is specifically used for:

[0159] calculating the feature similarity of each element in the initial five-element feature of the current frame of to-be-trained picture by using the initial five-element feature of the current frame of to-be-trained picture and the last frame of to-be-trained picture respectively;

[0160] The dynamic weight of the corresponding element is calculated based on the feature similarity;

[0161] The initial five-element feature is weighted and aggregated by using the dynamic weight to obtain a transition five-element feature;

[0162] The transition five-element feature is subjected to multi-scale feature fusion to obtain a fusion five-element feature of the current frame of the picture to be trained.

[0163] Optionally, the spatio-temporal aggregation module 204 is specifically configured to:

[0164] The transition five-element feature is input into a preset multi-scale attention mechanism to extract a scale feature of each element in the transition five-element feature at each scale;

[0165] The scale attention weight of the corresponding element is calculated based on the scale feature;

[0166] The fusion feature of the corresponding element is obtained by weighted aggregation based on the scale attention weight and the scale feature;

[0167] The five fusion features of the transition five-element feature are subjected to feature aggregation to obtain a fusion five-element feature of the current frame of the picture to be trained.

[0168] Optionally, the spatio-temporal aggregation module 204 is specifically configured to:

[0169] The fusion five-element feature of the current frame of the picture to be trained and the plurality of fusion five-element features of the previous frame of the picture to be trained are respectively subjected to pairwise combination to obtain a plurality of pairs of five-element feature groups;

[0170] A pairwise matching confidence of the pair of five-element feature groups is obtained;

[0171] A feature group corresponding to a preset number of pairwise matching confidences in the front of the ranking is determined as a target feature group, and the ranking represents descending order sorting of the plurality of pairwise matching confidences;

[0172] The fusion five-element feature of the current frame of the picture to be trained is subjected to feature fusion by using the fusion five-element feature of the previous frame of the picture to be trained in the target feature group to obtain a target five-element feature of the picture to be trained.

[0173] Optionally, the model adjustment module 205 is specifically configured to:

[0174] The elements in the target five-element feature are subjected to category prediction by using a preset classification head to obtain a predicted category of the elements and a category probability of the predicted category;

[0175] A prediction confidence of the target five-element feature is calculated based on the five category probabilities of the target five-element feature.

[0176] Optionally, the model adjustment module 205 is specifically configured to:

[0177] The classification loss of the corresponding element is calculated based on the predicted class corresponding to the prediction confidence and the real class corresponding to the predicted class by using a preset classification loss function;

[0178] The position boundary of the target subject, the target tool and the target object in the target five-element feature is regressively predicted to obtain a predicted bounding box of the target five-element feature in the corresponding to-be-trained picture;

[0179] The regression loss of the target five-element feature is calculated based on the predicted bounding box and the corresponding real bounding box by using a preset bounding box regression loss function;

[0180] The target loss of the target five-element feature is calculated based on the classification loss and the regression loss;

[0181] The network parameters in the initial model are adjusted based on the target loss to obtain a target model.

[0182] It should be noted that the image-based five-element feature detection model training system provided in the above embodiments and the image-based five-element feature detection model training method provided in the above embodiments belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be described here. The image-based five-element feature detection model training system provided in the above embodiments can be used to complete the above-described functions by different functional modules according to the needs, that is, the internal structure of the system is divided into different functional modules to complete all or part of the above-described functions, and this is not limited herein.

[0183] A computing device in an embodiment of the present application includes a memory, a processor, and a program stored on the memory and running on the processor. When the processor executes the program, it implements some or all steps of the above-described image-based five-element feature detection model training method.

[0184] The computing device can be a computer, and the program is computer software. The parameters and steps in the above-described computing device can refer to the parameters and steps in the above-described image-based five-element feature detection model training method, which will not be described here.

[0185] A computer-readable storage medium in an embodiment of the present application stores instructions. When the instructions are executed, the steps of the above-described image-based five-element feature detection model training method are executed.

[0186] The computer-readable storage medium can be a transitory computer-readable storage medium or a non-transitory computer-readable storage medium.

[0187] The technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes one or more instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method of the embodiments of the present disclosure. The aforementioned computer readable storage medium can be a non-transitory computer readable storage medium, including: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes, or can be a transitory computer readable storage medium.

[0188] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment or a part of code, and the module, the program segment or the part of code include one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order from that noted in the drawings. For example, two blocks represented in succession can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0189] Those skilled in the art know that the present application can be implemented as a system, a method or a computer program product. Therefore, the present disclosure can be embodied in the form of a complete hardware, a complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, which is generally referred to as "module" or "system" herein. In addition, in some embodiments, the present application can also be implemented in the form of a computer program product in one or more computer readable media, which includes computer readable program codes. The computer readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof.

[0190] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are contained in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0191] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and the person skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A method for training an image-based five-element feature detection model, characterized in that, The method comprises the following steps: acquiring a to-be-trained image; inputting the to-be-trained image into an initial model to perform visual feature extraction processing on the to-be-trained image, and obtaining an initial triple feature of each frame of to-be-trained picture in the to-be-trained image, wherein elements of the initial triple feature include an initial subject, an initial tool and an initial object; performing element construction based on the initial triple feature to obtain an initial quintuple feature, wherein elements of the initial quintuple feature include the initial subject, the initial tool, the initial object, an initial action and an initial element relationship; calculating feature similarity of each element in the initial quintuple feature of the current frame of to-be-trained picture by using the initial quintuple feature of the current frame of to-be-trained picture and the initial quintuple feature of the last frame of to-be-trained picture respectively; calculating a dynamic weight of the corresponding element based on the feature similarity, and performing weighted aggregation on the initial quintuple feature by using the dynamic weight to obtain a transition quintuple feature; performing multi-scale feature fusion on the transition quintuple feature to obtain a fusion quintuple feature of the current frame of to-be-trained picture; performing time context aggregation processing based on the fusion quintuple feature to obtain a target quintuple feature of the to-be-trained picture, wherein elements of the target quintuple feature include a target subject, a target tool, a target object, a target action and a target element relationship; 2. The method of claim 1, wherein, determining a prediction confidence of the target quintuple feature, and adjusting the initial model based on the prediction confidence and the target quintuple feature to obtain a target model. The element construction based on the initial triple feature to obtain the initial quintuple feature comprises the following steps: detecting the initial triple feature by using a preset triple interaction action detection method to obtain an initial action formed by element interaction in the triple feature; inputting the initial triple feature into a preset multi-head attention mechanism for processing to obtain an initial element relationship between elements of the triple feature; 3. The method of claim 1, wherein, forming the initial quintuple feature based on the initial triple feature, the initial action and the initial element relationship. The multi-scale feature fusion on the transition quintuple feature to obtain the fusion quintuple feature of the current frame of to-be-trained picture comprises the following steps: inputting the transition quintuple feature into a preset multi-scale attention mechanism to extract a scale feature of each element in the transition quintuple feature at each scale; calculating a scale attention weight of the corresponding element based on the scale feature; performing weighted aggregation based on the scale attention weight and the scale feature to obtain a fusion feature of the corresponding element; 4. The method of claim 1, wherein, performing feature aggregation on five fusion features of the transition quintuple feature to obtain the fusion quintuple feature of the current frame of to-be-trained picture. The time context aggregation processing based on the fusion quintuple feature to obtain the target quintuple feature of the to-be-trained picture comprises the following steps: performing pairwise combination on the fusion quintuple feature of the current frame of to-be-trained picture and a plurality of fusion quintuple features of the last frame of to-be-trained picture respectively to obtain a plurality of pairs of quintuple feature groups; obtaining a pairwise matching confidence corresponding to the quintuple feature group; Determine a target feature group corresponding to a preset number of pair matching confidence degrees in a front rank as a target feature group, the ranking representing descending order sorting of the plurality of pair matching confidence degrees; Fuse the fusion five-element features of the last frame of training pictures in the target feature group with the fusion five-element features of the current frame of training pictures to obtain target five-element features of the training pictures.

5. The method according to any one of claims 1 to 4, characterized in that, The determination of the prediction confidence of the target five-element features includes: Using a preset classification head to predict the category of the elements in the target five-element features to obtain the predicted category and the category probability of the predicted category; Based on the five category probabilities of the target five-element features, the prediction confidence of the target five-element features is calculated.

6. The method of claim 5, wherein, Based on the prediction confidence and the target five-element features, the initial model is adjusted to obtain a target model, which includes: Using a preset classification loss function, based on the predicted category corresponding to the prediction confidence and the real category corresponding to the predicted category, the classification loss of the corresponding element is calculated; The position boundary of the target subject, target tool and target object in the target five-element features is predicted to obtain the prediction bounding box of the target five-element features in the corresponding training picture; Using a preset bounding box regression loss function, based on the prediction bounding box and the corresponding real bounding box, the regression loss of the target five-element features is calculated; Based on the classification loss and the regression loss, the target loss of the target five-element features is calculated; Based on the target loss, the network parameters in the initial model are adjusted to obtain a target model.

7. An image-based five-element feature detection model training system, characterized by, It includes: An acquisition module is configured to acquire a training image; A feature extraction module is configured to input the training image into an initial model to perform visual feature extraction processing on the training image to obtain an initial three-element feature of each frame of training picture in the training image, and the elements of the initial three-element feature include an initial subject, an initial tool and an initial object; An element construction module is configured to construct elements based on the initial three-element feature to obtain an initial five-element feature, and the elements of the initial five-element feature include the initial subject, the initial tool, the initial object, an initial action and an initial element relationship; A space-time aggregation module is configured to calculate the feature similarity of each element in the initial five-element feature of the current frame of training picture using the initial five-element features of the current frame of training picture and the last frame of training picture respectively; Based on the feature similarity, the dynamic weight of the corresponding element is calculated; the initial five-element feature is weighted and aggregated using the dynamic weight to obtain a transition five-element feature; and the transition five-element feature is subjected to multi-scale feature fusion to obtain a fusion five-element feature of the current frame of training picture; Based on the fusion five-element feature, a time context aggregation process is performed to obtain a target five-element feature of the training picture, and the elements of the target five-element feature include a target subject, a target tool, a target object, a target action and a target element relationship; A model adjusting module is configured to determine a prediction confidence of the target five-element feature, and adjust the initial model based on the prediction confidence and the target five-element feature to obtain a target model.

8. A computing device comprising a memory, a processor, and a program stored on the memory and running on the processor, wherein, The processor implements the steps of the image-based five-element feature detection model training method according to any one of claims 1 to 6 when executing the program.

Images