Object representation model training method, object representation method and apparatus

Abstract

The application relates to an object representation model training method, an object representation method, an object representation device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: inputting a first training image set into an initial object representation model to obtain object association features corresponding to training images, object features and object fusion features; obtaining an object association classification loss based on the object association features corresponding to the same training image and an object association label, obtaining an object classification loss based on the object features corresponding to the same training image and an object label, and obtaining a fusion loss based on each object fusion feature corresponding to the same training image group; adjusting model parameters of the initial object representation model based on various losses to obtain an intermediate object representation model; performing model iterative training on the intermediate object representation model as the initial object representation model until a first convergence condition is met, and obtaining a target object representation model. The method can improve the representation accuracy of features for objects.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an object representation model training method, object representation method, apparatus, computer device, storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, images are increasingly widely used in daily life and work, such as image retrieval, video retrieval, and video deduplication. In the process of applying images, it is necessary to extract image features to characterize the information contained within the image.

[0003] In traditional techniques, for images containing objects, the image is typically input into a model to extract object features, thus obtaining the corresponding object features in the image. However, simply representing objects in an image based on object features is not accurate enough and cannot effectively distinguish between images that are similar but actually different. Summary of the Invention

[0004] Based on this, it is necessary to provide an object representation model training method, object representation method, device, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of feature representation of objects in addressing the above-mentioned technical problems.

[0005] This application provides a method for training an object representation model. The method includes:

[0006] Obtain a first training image set and input the first training image set into the initial object representation model; the first training image set includes multiple training image groups, each training image group includes multiple training images that are compared with each other, and the objects in each training image have corresponding object labels and object association labels;

[0007] Object association features are extracted from the training images to obtain object association features corresponding to each training image. Object features are extracted from the training images to obtain object features corresponding to each training image. The object features and object association features corresponding to the same training image are fused to obtain object fusion features corresponding to each training image.

[0008] Based on the object association features and object association labels corresponding to the same training image, we obtain the object association classification loss; based on the object features and object labels corresponding to the same training image, we obtain the object classification loss; based on the fusion features of each object corresponding to the same group of training images, we obtain the fusion loss.

[0009] Based on the object association classification loss, the object classification loss, and the fusion loss, the model parameters of the initial object representation model are adjusted to obtain an intermediate object representation model. The intermediate object representation model is then used as the initial object representation model. The steps of obtaining the first training image set and inputting the first training image set into the initial object representation model are repeated until the first convergence condition is met to obtain the target object representation model. The target object representation model is used to extract the object fusion features corresponding to the input image.

[0010] In one embodiment, obtaining the first training image set includes:

[0011] Obtain target object sub-graph sequences corresponding to multiple training object trajectories; the training object trajectories are mined from training videos containing objects, and the object sub-graphs in the target object sub-graph sequences are obtained by object detection on the corresponding video frames in the training videos to which the training object trajectories belong; sample each target object sub-graph sequence to obtain multiple positive sample image pairs; obtain the negative sample image corresponding to the current positive sample image pair from other positive sample image pairs; obtain the object label and object association label corresponding to the object in the training image in the positive sample image pair; form a training image group based on the positive sample image pair, the corresponding negative sample image, the corresponding object label, and the object association label; obtain the first training image set based on each training image group.

[0012] In one embodiment, obtaining the target object subgraph sequence corresponding to the trajectories of multiple training objects includes:

[0013] Multiple training video frame sequences are acquired; video frames in the same training video frame sequence belong to the same training video; object detection is performed on the video frames in the training video frame sequence to obtain object subgraphs in the video frames, and the object subgraphs are input into the initial object representation model to obtain the object fusion features corresponding to the object subgraphs; based on each object subgraph and the corresponding object fusion features in adjacent video frames, object matching is performed on adjacent video frames in the same training video frame sequence to obtain the object matching results corresponding to each group of adjacent video frames; trajectory aggregation is performed based on the object matching results of each group of adjacent video frames in the same training video frame sequence to obtain multiple training object trajectories corresponding to target object subgraph sequences.

[0014] In one embodiment, the step of performing object matching on adjacent video frames in the same training video frame sequence based on each object subgraph and its corresponding object fusion feature in adjacent video frames to obtain object matching results for each group of adjacent video frames includes:

[0015] The process begins by determining a target video frame from the current video frame sequence; sequentially using each object subgraph corresponding to the target video frame as the target object subgraph; based on the feature distance between the object fusion features corresponding to the target object subgraph and the object fusion features corresponding to each object subgraph in the target backward video frame, candidate object subgraphs corresponding to the target object subgraph are determined from each object subgraph corresponding to the target backward video frame; candidate object subgraphs whose positional distance to the target object subgraph is less than a distance threshold are used as the backward matching object subgraphs corresponding to the target object subgraph; the target backward video frame is the adjacent backward video frame of the target video frame; based on each target object subgraph and the corresponding backward matching object subgraph, the object matching result between the target video frame and the target backward video frame is obtained; the next video frame is obtained from the current video frame sequence as the target video frame, and the process of sequentially using each object subgraph corresponding to the target video frame as the target object subgraph is repeated until the object matching results corresponding to each group of adjacent video frames in the current video frame sequence are obtained.

[0016] In one embodiment, the distance threshold is dynamically changing and is obtained by statistically analyzing the positional distance between the updated target object sub-image and the corresponding backward matching object sub-image after completing object matching of a second preset number of adjacent video frames.

[0017] In one embodiment, obtaining the negative sample image corresponding to the current positive sample image pair from other positive sample image pairs besides the current positive sample image pair includes:

[0018] Training images are obtained from the current positive sample image pair as target positive sample images, and training images are obtained from each other positive sample image pair as candidate images. Based on the image distance between the target positive sample image and each candidate image, the candidate images are sorted in ascending order of image distance to obtain a candidate image sequence. A third preset number of candidate images are deleted from the candidate image sequence in ascending order of image distance to obtain an updated image sequence. A fourth preset number of candidate images are obtained from the updated image sequence in ascending order of image distance to serve as negative sample images corresponding to the current positive sample image pair.

[0019] In one embodiment, when the current feature is an object association feature, the current label is an object association label, and the current classification loss is an object association classification loss; when the current feature is an object feature, the current label is an object label, and the current classification loss is an object classification loss.

[0020] The calculation process for the current classification loss includes the following steps:

[0021] Obtain the prototype object feature sets corresponding to various current labels; the prototype object feature sets include multiple prototype object features corresponding to the same current label, and the prototype object features are obtained from the current feature library by selecting the current feature that is closest to the cluster center feature corresponding to the current label. The cluster center feature is obtained by clustering the current features corresponding to the same current label in the current feature library. From each prototype object feature set, select the prototype object feature with the smallest feature distance to the current feature corresponding to the current training image as the reference object feature corresponding to the current training image. Based on the difference between the current feature and the reference object feature corresponding to the same training image, obtain the current classification loss.

[0022] In one embodiment, the process of determining the prototype object feature set includes the following steps:

[0023] Within the initial model iteration rounds, every fifth preset number of batches trained, the prototype object feature sets corresponding to various current labels are determined based on the current feature library; within the subsequent model iteration rounds within the initial model iteration rounds, every sixth preset number of batches trained, the prototype object feature sets corresponding to various current labels are determined based on the current feature library; the sixth preset number is greater than the fifth preset number; for each batch trained, the obtained current features are stored in the current feature library, and the current feature library is cleared every time a prototype object feature set is determined.

[0024] In one embodiment, when the current feature is an object association feature, the current label is an object association label, and the current classification loss is an object association classification loss; when the current feature is an object feature, the current label is an object label, and the current classification loss is an object classification loss.

[0025] The calculation process for the current classification loss includes the following steps:

[0026] The predicted probability of the training image is obtained based on the current features corresponding to the training image; the current classification loss is obtained based on the predicted probability of the same training image and the current label.

[0027] In one embodiment, obtaining the fusion loss based on the fusion features of each object corresponding to the same training image group includes:

[0028] A first difference is obtained based on the difference between the fusion features of each object corresponding to the positive sample image pair in the same training image group; a reference positive sample image is determined from the positive sample image pair; a second difference is obtained based on the difference between the fusion features of the objects corresponding to the reference positive sample image and the negative sample image in the same training image group; and the fusion loss is obtained based on the distance between the first difference and the second difference corresponding to the same training image group and a preset distance.

[0029] This application also provides an object representation model training device. The device includes:

[0030] The training image set acquisition module is used to acquire a first training image set and input the first training image set into the initial object representation model; the first training image set includes multiple training image groups, and each training image group includes multiple training images that are compared with each other, and the objects in each training image have corresponding object labels and object association labels;

[0031] The feature determination module is used to extract object association features from training images to obtain object association features corresponding to each training image, extract object features from training images to obtain object features corresponding to each training image, and fuse object features and object association features corresponding to the same training image to obtain object fusion features corresponding to each training image.

[0032] The loss determination module is used to obtain object association classification loss based on object association features and object association labels corresponding to the same training image, object classification loss based on object features and object labels corresponding to the same training image, and fusion loss based on the fusion features of each object corresponding to the same group of training images.

[0033] The model adjustment module is used to adjust the model parameters of the initial object representation model based on the object association classification loss, the object classification loss, and the fusion loss to obtain an intermediate object representation model. The intermediate object representation model is then used as the initial object representation model. The steps of obtaining the first training image set and inputting the first training image set into the initial object representation model are repeated until the first convergence condition is met to obtain the target object representation model. The target object representation model is used to extract the object fusion features corresponding to the input image.

[0034] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps described in the object representation model training method above.

[0035] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the object representation model training method above.

[0036] A computer program product includes a computer program that, when executed by a processor, implements the steps described in the object representation model training method above.

[0037] The aforementioned object representation model training method, apparatus, computer equipment, storage medium, and computer program product acquire a first training image set and input the first training image set into an initial object representation model. The first training image set includes multiple training image groups, each training image group including multiple mutually contrasting training images. Objects in each training image have corresponding object labels and object association labels. Object association features are extracted from the training images to obtain object association features corresponding to each training image. Object features are extracted from the training images to obtain object features corresponding to each training image. The object features and object association features corresponding to the same training image are fused to obtain object fusion features corresponding to each training image. Based on the same training image... The model extracts object association features and labels from training images to obtain object association classification loss. It then extracts object classification loss based on object features and labels from the same training image and extracts fusion loss based on the fusion features of each object from the same training image group. Based on the object association classification loss, object classification loss, and fusion loss, the model parameters of the initial object representation model are adjusted to obtain an intermediate object representation model. This intermediate model is then used as the initial object representation model. The process of obtaining the first training image set and inputting it into the initial object representation model is repeated until the first convergence condition is met, resulting in the target object representation model. The target object representation model is used to extract object fusion features from the input images. In this way, the model extracts object features and object association features from the images, fuses these features from the training images to obtain object fusion features, which contain feature information of different dimensions, allowing for better representation of objects in the images. Furthermore, there's no need to train a separate model for each feature, nor to train multiple models. During model training, object features, object association features, and object fusion features corresponding to the same training image are extracted from a single model. Object association classification loss is calculated based on the object association features and object association labels corresponding to the same training image; object classification loss is calculated based on the object features and object labels corresponding to the same training image; and fusion loss is calculated based on the fusion features of each object corresponding to the same group of training images. Joint training is then performed based on the object association classification loss, object classification loss, and fusion loss, ultimately resulting in a model that is accurate and reliable in feature extraction for all types of features. The target object representation model obtained after model training can extract accurate and reliable object fusion features corresponding to the input image. Object fusion features can better represent objects in the image and also help improve the accuracy of subsequent data processing.

[0038] This application provides an object representation method. The method includes:

[0039] Obtain the image of the object to be processed;

[0040] The image of the object to be processed is input into the target object representation model to obtain the object fusion feature corresponding to the image of the object to be processed; the object fusion feature corresponding to the image of the object to be processed is used to determine the image that matches the image of the object to be processed.

[0041] The training process of the target object representation model includes the following steps:

[0042] A first training image set is acquired and input into an initial object representation model. The first training image set includes multiple training image groups, each group containing multiple comparative training images. Objects in each training image have corresponding object labels and object association labels. Object association features are extracted from the training images to obtain object association features corresponding to each training image. Object features are extracted from the training images to obtain object features corresponding to each training image. Object features and object association features corresponding to the same training image are fused to obtain object fusion features corresponding to each training image. Based on the object association features and object association labels corresponding to the same training image, an object association classification loss is obtained. Based on the object features and object labels corresponding to the same training image, an object classification loss is obtained. Based on the object fusion features corresponding to each object in the same training image group, a fusion loss is obtained. Based on the object association classification loss, the object classification loss, and the fusion loss, the model parameters of the initial object representation model are adjusted to obtain an intermediate object representation model. The intermediate object representation model is used as the initial object representation model. The process of acquiring the first training image set and inputting it into the initial object representation model is repeated until a first convergence condition is met, resulting in a target object representation model.

[0043] In one embodiment, the object representation method further includes:

[0044] Obtain the video frame sequence corresponding to the video to be processed; perform object detection on the video frames in the video frame sequence to obtain object subgraphs in the video frames, input the object subgraphs into the target object representation model to obtain object fusion features corresponding to the object subgraphs; based on each object subgraph and corresponding object fusion features in the adjacent video frames of the video frame sequence to be processed, obtain the reference object trajectory corresponding to the video to be processed; the reference object trajectory includes the object fusion features of the corresponding object subgraphs; based on each reference object trajectory corresponding to the video to be processed and each candidate object trajectory corresponding to candidate videos in the video library, query candidate videos similar to the video to be processed from the video library to obtain query results.

[0045] In one embodiment, the object representation method further includes:

[0046] When the video to be processed is a video to be uploaded, if the query result is a successful query, the upload suggestion for the video to be processed is to prohibit uploading; if the query result is a failed query, the upload suggestion for the video to be processed is to allow uploading.

[0047] If the video to be processed is an already uploaded video, and the query result is a successful query, a target video is determined from the video to be processed and candidate videos similar to the video to be processed, and the processing suggestion corresponding to the target video is to notify it to be taken down. If the query result is a failed query, the processing suggestion corresponding to the video to be processed and the candidate video is to not process it.

[0048] This application also provides an object representation device. The device includes:

[0049] The image acquisition module is used to acquire images of the object to be processed.

[0050] The image processing module is used to input the image of the object to be processed into the target object representation model to obtain the object fusion features corresponding to the image of the object to be processed; the object fusion features corresponding to the image of the object to be processed are used to determine the image that matches the image of the object to be processed.

[0051] The training process of the target object representation model includes the following steps:

[0052] A first training image set is acquired and input into an initial object representation model. The first training image set includes multiple training image groups, each group containing multiple comparative training images. Objects in each training image have corresponding object labels and object association labels. Object association features are extracted from the training images to obtain object association features corresponding to each training image. Object features are extracted from the training images to obtain object features corresponding to each training image. Object features and object association features corresponding to the same training image are fused to obtain object fusion features corresponding to each training image. Based on the object association features and object association labels corresponding to the same training image, an object association classification loss is obtained. Based on the object features and object labels corresponding to the same training image, an object classification loss is obtained. Based on the object fusion features corresponding to each object in the same training image group, a fusion loss is obtained. Based on the object association classification loss, the object classification loss, and the fusion loss, the model parameters of the initial object representation model are adjusted to obtain an intermediate object representation model. The intermediate object representation model is used as the initial object representation model. The process of acquiring the first training image set and inputting it into the initial object representation model is repeated until a first convergence condition is met, resulting in a target object representation model.

[0053] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps described in the object representation method above.

[0054] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the object representation method above.

[0055] A computer program product includes a computer program that, when executed by a processor, implements the steps described in the object representation method above.

[0056] The aforementioned object representation method, apparatus, computer equipment, storage medium, and computer program product acquire an image of the object to be processed; input the image of the object to be processed into a target object representation model to obtain object fusion features corresponding to the image of the object to be processed; the object fusion features corresponding to the image of the object to be processed are used to determine images that match the image of the object to be processed. Thus, in the model, object features and object association features of the image are extracted, and the object features and object association features of the training images are fused to obtain object fusion features. Object fusion features contain feature information of different dimensions, which can better represent objects in the image. Furthermore, it is not necessary to train a separate model for each feature, nor is it necessary to train multiple models. During model training, object features, object association features, and object fusion features corresponding to the same training image are extracted from a single model. Object association classification loss is calculated based on the object association features and object association labels corresponding to the same training image. Object classification loss is calculated based on the object features and object labels corresponding to the same training image. Fusion loss is calculated based on the fusion features of each object corresponding to the same group of training images. Joint training is performed based on object association classification loss, object classification loss, and fusion loss, ultimately training a model that is accurate and reliable in feature extraction for various features. The target object representation model obtained through model training can extract accurate and reliable object fusion features corresponding to the input image. Object fusion features can better represent the objects in the image and also help improve the accuracy of subsequent data processing, improving the accuracy of image matching when determining the image that matches the image of the object to be processed. Attached Figure Description

[0057] Figure 1 This is a diagram illustrating the object representation model training method and the application environment of the object representation method in one embodiment.

[0058] Figure 2 This is a flowchart illustrating an object representation model training method in one embodiment;

[0059] Figure 3 This is a schematic diagram illustrating the calculation of object classification loss in one embodiment;

[0060] Figure 4 This is a schematic diagram of the structure of an object representation model in one embodiment;

[0061] Figure 5 This is a schematic diagram of the object representation model in another embodiment;

[0062] Figure 6 This is a flowchart illustrating the object representation model training method in another embodiment;

[0063] Figure 7 This is a schematic diagram of the first stage of model training in one embodiment;

[0064] Figure 8 This is a flowchart illustrating the object representation model training method in yet another embodiment;

[0065] Figure 9 This is a schematic diagram of the third stage of training of the model in one embodiment;

[0066] Figure 10 This is a flowchart illustrating an object representation method in one embodiment;

[0067] Figure 11 This is a schematic diagram of the hero representation model in one embodiment;

[0068] Figure 12 This is a schematic diagram illustrating the detection of the hero's location from a video frame in one embodiment;

[0069] Figure 13 This is a structural block diagram of an object representation model training device in one embodiment;

[0070] Figure 14 This is a structural block diagram of an object representation device in one embodiment;

[0071] Figure 15 This is an internal structural diagram of a computer device in one embodiment;

[0072] Figure 16 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0074] It is understood that the terms "first," "second," etc., used in this application may be used to describe various elements herein, but these elements are not limited by these terms. These terms are only used to distinguish one piece of data from another.

[0075] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, image or video retrieval, image or video deduplication, and image or video recommendation.

[0076] The solutions provided in this application involve artificial intelligence technologies such as computer vision and machine learning, and are specifically illustrated through the following embodiments.

[0077] The object representation model training method and object representation method provided in the embodiments of this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on other servers. Terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server, a server cluster consisting of multiple servers, or a cloud server.

[0078] Both the terminal and the server can be used independently to execute the object representation model training method and object representation method provided in the embodiments of this application.

[0079] For example, the server acquires a first training image set and inputs it into an initial object representation model. The first training image set includes multiple training image groups, each containing multiple comparative training images. Objects in each training image have corresponding object labels and object association labels. In the initial object representation model, object association features are extracted from the training images to obtain object association features corresponding to each training image. Object features are extracted from the training images to obtain object features corresponding to each training image. Object features and object association features corresponding to the same training image are fused to obtain object fusion features corresponding to each training image. Based on the object association features and object association labels corresponding to the same training image, an object association classification loss is obtained. Based on the object features and object labels corresponding to the same training image, an object classification loss is obtained. Based on the object fusion features corresponding to each object in the same training image group, a fusion loss is obtained. The server adjusts the model parameters of the initial object representation model based on the object association classification loss, object classification loss, and fusion loss to obtain an intermediate object representation model. This intermediate object representation model is used as the initial object representation model. The steps of acquiring the first training image set and inputting it into the initial object representation model are repeated until the first convergence condition is met, resulting in the target object representation model. The target object representation model is used to extract the object fusion features corresponding to the input image.

[0080] The server acquires the image of the object to be processed, inputs it into the target object representation model, and obtains the object fusion features corresponding to the image of the object to be processed. These object fusion features are used to determine images that match the image of the object to be processed.

[0081] The terminal and server can also work together to execute the object representation model training method and object representation method provided in the embodiments of this application.

[0082] For example, the server obtains a first training image set from the terminal, and the server trains the initial object representation model based on the first training image set to obtain the target object representation model.

[0083] The server retrieves the image of the object to be processed from the terminal. It then inputs this image into a target object representation model to obtain the object fusion features corresponding to the image. Based on these features, the server determines an image that matches the original image. The server then sends this matching image to the terminal. The terminal can then display the matching image.

[0084] In one embodiment, such as Figure 2 As shown, a method for training an object representation model is provided. Taking the application of this method to a computer device as an example, the computer device can be a terminal or a server. The method can be executed independently by the terminal or server, or it can be implemented through interaction between the terminal and the server. (Reference) Figure 2 The object representation model training method includes the following steps:

[0085] Step S202: Obtain the first training image set and input the first training image set into the initial object representation model; the first training image set includes multiple training image groups, each training image group includes multiple training images that are compared with each other, and the objects in each training image have corresponding object labels and object association labels.

[0086] The training image set comprises multiple training image groups. Each training image group is an image triplet, consisting of three training images used for mutual comparison; that is, a training image group includes three mutually contrasting training images. Specifically, a training image group includes positive sample image pairs and their corresponding negative sample images. A positive sample image pair consists of a pair of positive sample images. The training images in a positive sample image pair are similar images. In contrast, a negative sample image is an image that is less similar to the images in the positive sample image pair. That is, the similarity between training images in a positive sample image pair is greater than the similarity between any image in the positive sample image pair and the negative sample image. Using any one training image from a positive sample image pair within a training image group as the baseline image, the other training image in the positive sample image pair is equivalent to the forward reference image corresponding to the baseline image, and the negative sample images in the training image group are equivalent to the reverse reference images corresponding to the baseline image.

[0087] Each training image in the training image set contains an object. The object can be a physical object, such as a vehicle; it can also be a living being, such as an animal; or it can be a virtual character, such as a game character.

[0088] Training images have corresponding object labels and object association labels. Object labels and object association labels are training labels used for model training. Object labels and object association labels are labels related to objects in the training images. Each object has a corresponding object label and object association label, and an object can have at least one object association label.

[0089] Object labels are used to characterize the object category corresponding to an object in an image. Different object labels are used to distinguish different objects. For example, if the training image contains game characters, the object label can be the character name, specifically game character A, game character B, game character C, etc.; if the training image contains animals, the object label can be the animal species, specifically duck, mandarin duck, etc. Object labels can be binary or multi-class labels.

[0090] Object association labels are used to characterize the content displayed in the associated regions of an image. Different object association labels are used to distinguish different regions. An object association region refers to a local area in an image that is related to an object. An object association region can be a specific part of the object; for example, if the training image is a bird image, the object association region could be the area containing feathers, the beak, or the tail. The object association label could be a feather label representing the color of the feathers, a beak label representing the shape of the beak, and a tail label representing the shape of the tail. An object association region can also be a specific area surrounding the object. For example, if the training image is a game character image, the object association region could be the area containing the game character's health bar, and the object association label could be a health bar label representing the color of the health bar; if the training image is a basketball court image, the object association region could be the area containing the court's advertising boards, and the object association label could be a brand label representing the product brand displayed on the court's advertising boards. It can be understood that object association labels can be set and selected according to specific practical needs. If an object has different object association labels, then different object association labels correspond to object association regions at different locations. For example, in an animal image, an animal has a mouth label and a tail label. The mouth label is used to characterize the shape of the animal's mouth, and the tail label is used to characterize the shape of the animal's tail. Object association labels can be binary or multi-class labels.

[0091] An object representation model is a machine learning model used to extract features that represent objects in an image. The input data for an object representation model is an image containing objects, and the output data is at least one feature corresponding to the objects in the image. The initial object representation model refers to the object representation model to be trained. The initial object representation model can be an initialized object representation model or an object representation model obtained through pre-training. The first training set is the training set used to train the initial object representation model.

[0092] Specifically, the computer device can acquire the first training image set locally or from other devices, and then input the first training image set into the initial object representation model, and extract various features corresponding to each training image in the first training image set through the initial object representation model.

[0093] Step S204: Extract object association features from the training images to obtain object association features corresponding to each training image; extract object features from the training images to obtain object features corresponding to each training image; fuse the object features and object association features corresponding to the same training image to obtain object fusion features corresponding to each training image.

[0094] Object association feature extraction refers to extracting object association features corresponding to an image. Object association features are used to characterize the feature information corresponding to the object association region in an image. Object feature extraction refers to extracting object features corresponding to an image. Object features are used to characterize the feature information corresponding to the region where an object is located in an image. Object fusion features are features obtained by fusing object association features and object features belonging to the same image. Object fusion features contain the feature information corresponding to the object association region and the region where the object is located in the image, and can more accurately characterize the objects in the image.

[0095] Specifically, after the computer device inputs the first training image set into the initial object representation model, it extracts object association features from the training images through the initial object representation model to obtain object association features corresponding to each training image in the first training image set. Then, it extracts object features from the training images through the initial object representation model to obtain object features corresponding to each training image in the first training image set. Finally, it fuses the object features and object association features corresponding to the same training image through the initial object representation model to obtain object fusion features corresponding to each training image.

[0096] Step S206: Based on the object association features and object association labels corresponding to the same training image, obtain the object association classification loss; based on the object features and object labels corresponding to the same training image, obtain the object classification loss; based on the fusion features of each object corresponding to the same group of training images, obtain the fusion loss.

[0097] The object association classification loss characterizes the difference between the information represented by object association features and the information represented by object association labels. A smaller object association classification loss indicates more accurate extraction of object association features. The object classification loss characterizes the difference between the information represented by object features and the information represented by object labels. A smaller object classification loss indicates more accurate extraction of object features.

[0098] The fusion loss characterizes the difference between the object fusion feature distance between positive sample images and the object fusion feature distance between positive and negative sample images in the training image set. A smaller fusion loss indicates that the object fusion features between positive sample images are closer, while the object fusion features between positive and negative sample images are further apart.

[0099] Specifically, the computer device can calculate the object association classification sub-loss based on the object association features and object association labels corresponding to the same training image, and obtain the object association classification loss based on the object association classification sub-loss corresponding to each training image. It can also calculate the object classification sub-loss based on the object features and object labels corresponding to the same training image, and obtain the object classification loss based on the object classification sub-loss corresponding to each training image. Finally, it can calculate the fusion sub-loss based on the fusion features of each object corresponding to the same group of training images, and obtain the fusion loss based on the fusion sub-loss corresponding to each group of training images.

[0100] In one embodiment, a first predicted label is obtained based on the object association features corresponding to the training image. Then, an object association classification loss is obtained based on the first predicted label and the object association label corresponding to the same training image. It can be understood that the object association label is the correct label, the label that the model is expected to output, while the first predicted label is the predicted label, the label that the model actually outputs. During model training, it is hoped that the model can output a predicted label as close as possible to the object association label. If the model can output a predicted label that is approximately or identical to the object association label, then it can be considered that the model has extracted very accurate object association features. Similarly, a second predicted label can be obtained based on the object features corresponding to the training image, and an object classification loss can be obtained based on the second predicted label and the object label corresponding to the same training image.

[0101] Step S208: Based on object association classification loss, object classification loss and fusion loss, adjust the model parameters of the initial object representation model to obtain an intermediate object representation model. Use the intermediate object representation model as the initial object representation model, return to obtain the first training image set, and input the first training image set into the initial object representation model until the first convergence condition is met to obtain the target object representation model. The target object representation model is used to extract the object fusion features corresponding to the input image.

[0102] Here, the target object representation model refers to the currently trained object representation model. Convergence criteria refer to the conditions for determining whether the model has reached convergence. Convergence criteria include, but are not limited to, at least one of the following: loss is less than a preset loss value, the number of model iterations is greater than a preset number of iterations, the number of model iteration epochs is greater than a preset number of iterations, or the rate of change of loss is less than a preset rate of change. The first convergence criterion includes, but is not limited to, at least one of the following: all losses are less than their respective first preset loss values, the number of model iterations is greater than a first preset number of iterations, the number of model iteration epochs is greater than a first preset number of iterations, or the rate of change of all losses is less than their respective first preset rate of change.

[0103] A single model iteration refers to the data processing procedure that continues until an intermediate object representation model is obtained from steps S202 to S208. A single model iteration can also be called a model iteration batch, or simply a batch. A round of model iteration consists of a predetermined number of model iteration batches. A round of model iteration can also be called a model iteration cycle.

[0104] Specifically, the computer device performs backpropagation based on object association classification loss, object classification loss, and fusion loss to adjust the model parameters of the initial object representation model, obtaining an intermediate object representation model. Using the intermediate object representation model as the initial object representation model, the process returns to obtain the first training image set and inputs it into the initial object representation model for iterative training until the first convergence condition is met, ultimately training the target object representation model.

[0105] For example, in the first model iteration, an initialized object representation model is obtained as the initial object representation model. A first training image set is acquired and input into the initial object representation model to calculate the object association classification loss, object classification loss, and fusion loss. Based on these losses, the model parameters of the initial object representation model are adjusted to obtain an intermediate object representation model. In the second model iteration, the intermediate object representation model obtained in the previous iteration is used as the initial object representation model. A new first training image set is acquired and input into a new initial object representation model to calculate new object association classification losses, object classification losses, and fusion losses. Based on these new losses, the model parameters of the new initial object representation model are adjusted to obtain a new intermediate object representation model. In the third model iteration, the intermediate object representation model obtained in the previous iteration is used as the initial object representation model, and the data processing process of a single model iteration is repeated to obtain a new intermediate object representation model. This process continues until the first convergence condition is met, resulting in the target object representation model. If the first preset number of iterations is two hundred, then the intermediate object representation model obtained from the two hundred and first model iterations will be used as the target object representation model.

[0106] After training a target object representation model, a computer device can extract object fusion features corresponding to the input image based on the target object representation model. An image containing objects is input into the target object representation model, which extracts object association features and object features corresponding to the input image. These two features are then fused to obtain the object fusion features corresponding to the input image, which are finally output as the object fusion features. Object fusion features can be applied to various image-related tasks, such as image retrieval, video retrieval, and video deduplication.

[0107] In one embodiment, model parameters can be adjusted based on the loss using a gradient descent algorithm. For example, the gradient descent algorithm could be SGD (Stochastic Gradient Descent).

[0108] In the above-described object representation model training method, the model extracts object features and object association features from the image, and fuses these features from the training image to obtain object fusion features. These object fusion features contain feature information of different dimensions, enabling better representation of objects in the image. Furthermore, it eliminates the need to train a separate model for each feature or multiple models. During model training, object features, object association features, and object fusion features corresponding to the same training image are extracted from a single model. Object association classification loss is calculated based on the object association features and object association labels corresponding to the same training image. Object classification loss is calculated based on the object features and object labels corresponding to the same training image. Fusion loss is calculated based on the fusion features of each object corresponding to the same group of training images. Joint training is then performed based on the object association classification loss, object classification loss, and fusion loss, ultimately resulting in a model that is accurate and reliable in feature extraction for all types of features. The target object representation model obtained after model training can extract accurate and reliable object fusion features corresponding to the input image. These object fusion features better represent objects in the image and also help improve the accuracy of subsequent data processing.

[0109] In one embodiment, obtaining the first training image set includes:

[0110] Obtain target object sub-graph sequences corresponding to multiple training object trajectories; the training object trajectories are mined from training videos containing objects, and the object sub-graphs in the target object sub-graph sequences are obtained by object detection on the corresponding video frames in the training videos to which the training object trajectories belong; sample each target object sub-graph sequence to obtain multiple positive sample image pairs; obtain the negative sample image corresponding to the current positive sample image pair from other positive sample image pairs; obtain the object label and object association label corresponding to the object in the training image in the positive sample image pair, and form a training image group based on the positive sample image pair and the corresponding negative sample image; obtain the first training image set based on each training image group.

[0111] Here, object trajectory refers to the motion trajectory of an object in the video. It can be understood that a video can include at least one object, and the same object can correspond to at least one object trajectory.

[0112] An object trajectory exists as a corresponding sequence of target object subgraphs. This sequence includes multiple object subgraphs corresponding to the same object. Each object subgraph represents the region where the object appears within a video frame. Different object subgraphs within the same target object subgraph sequence correspond to different video frames within the same video. Object subgraphs are obtained by performing object detection on video frames. Object detection determines whether an object exists within a video frame and the region where the object is located within that frame. The order of object subgraphs in the target object subgraph sequence can be determined by arranging them according to the video frame order to which they belong.

[0113] The training object trajectory is the object trajectory mined from the training video containing the object. For the target object subgraph sequence corresponding to the training object trajectory, each object subgraph in the target object subgraph sequence is obtained by performing object detection on multiple video frames containing the same object in the training video to which the training object trajectory belongs.

[0114] The current positive sample image pair refers to the positive sample image pair currently being processed. The current positive sample image pair can be any positive sample image pair from among multiple sampled positive sample image pairs.

[0115] Specifically, when building a training image set, the computer device can first acquire positive sample image pairs, then acquire the negative sample images corresponding to the positive sample image pairs, combine the positive sample image pairs and the corresponding negative sample images into training image groups, and combine multiple training image groups into a training image set.

[0116] When acquiring positive sample image pairs, the computer device acquires multiple target object sub-image sequences corresponding to the trajectories of multiple training objects, resulting in multiple target object sub-image sequences. Each target object sub-image sequence is then sampled to obtain multiple positive sample image pairs. For example, adjacent object sub-images can be obtained as positive sample image pairs from the target object sub-image sequences. At least one positive sample image pair can be obtained from a single target object sub-image sequence; by sampling each target object sub-image sequence separately, multiple positive sample image pairs can ultimately be obtained.

[0117] When acquiring the negative sample image corresponding to a positive sample image pair, any one of the multiple sampled positive sample image pairs is taken as the current positive sample image pair. From the other positive sample image pairs (excluding the current positive sample image pair), at least one positive sample image is acquired as the corresponding negative sample image. For example, a positive sample image can be randomly acquired from other positive sample image pairs, or a positive sample image that meets certain conditions can be acquired from other positive sample image pairs. In the same way, a corresponding negative sample image can be acquired for each positive sample image pair.

[0118] It can be understood that the object labels and associated labels corresponding to objects within a training image can be obtained by annotating the image content of the training image. One positive sample image pair corresponds to one training image group.

[0119] In the above embodiments, sampling positive sample image pairs from the training object trajectory ensures that the positive sample image pairs are composed of similar images. Obtaining the negative sample image corresponding to the current positive sample image pair from other positive sample image pairs allows for rapid determination of the negative sample image corresponding to the positive sample image pair without acquiring new data.

[0120] In one embodiment, obtaining the target object subgraph sequence corresponding to multiple training object trajectories includes:

[0121] Multiple training video frame sequences are acquired; video frames in the same training video frame sequence belong to the same training video; object detection is performed on the video frames in the training video frame sequence to obtain object subgraphs in the video frames, and the object subgraphs are input into the initial object representation model to obtain the object fusion features corresponding to the object subgraphs; based on each object subgraph and the corresponding object fusion features in adjacent video frames, object matching is performed on adjacent video frames in the same training video frame sequence to obtain the object matching results corresponding to each group of adjacent video frames; trajectory aggregation is performed based on the object matching results of each group of adjacent video frames in the same training video frame sequence to obtain a sequence of target object subgraphs corresponding to multiple training object trajectories.

[0122] A video frame sequence comprises multiple video frames corresponding to the same video. The video frames in a video frame sequence are arranged in order according to their position within the video. A training video frame sequence refers to the sequence of video frames corresponding to the training video. A training video is the video used to generate training data for the model.

[0123] Object matching is used to determine the object subgraphs corresponding to the same object in a set of adjacent video frames. The object matching results for a set of adjacent video frames include each successfully matched object subgraph in the starting video frame and its corresponding matched object subgraph in the subsequent video frames. The starting video frame is the first video frame in the set of adjacent videos, and the subsequent video frame is the second video frame in the set of adjacent videos. A successfully matched object subgraph is the object subgraph that finds a matching object subgraph in the subsequent video frame. A matched object subgraph is the object subgraph that finds a matching object subgraph in the subsequent video frame.

[0124] Trajectory aggregation is used to determine the object subgraphs belonging to the same object from the object matching results of each group of adjacent video frames in the same training video frame sequence. It is equivalent to aggregating the trajectory points belonging to the same object together to generate object trajectories.

[0125] Specifically, the target object subgraph sequence can be obtained through manual annotation, but to reduce labor costs, it can also be intelligently extracted from the video. When acquiring the target object subgraph sequence corresponding to the object trajectory, the computer device performs object detection on the video frames in the corresponding video frame sequence to obtain object subgraphs. Object fusion features corresponding to the object subgraphs are extracted using an object representation model. Matching object subgraphs determined in adjacent video frames based on these object fusion features are considered as object subgraphs belonging to the same object. The target object subgraph sequence is then obtained by combining the object subgraphs belonging to the same object in each group of adjacent video frames.

[0126] When obtaining target object sub-image sequences corresponding to multiple training object trajectories, multiple training video frame sequences are first acquired. Then, object detection is performed on the video frames in the training video frame sequences to obtain object sub-images within the video frames. During object detection, the video frames from the training video frame sequence can be input into an initial object detection model. The initial object detection model outputs object prediction boxes in the video frames, and the image regions containing these object prediction boxes are used as object sub-images. After obtaining the object sub-images, they are input into an initial object representation model to obtain the object fusion features corresponding to the object sub-images. To generate object trajectories, it is necessary to first determine the trajectory points of the object. Therefore, based on the object sub-images and corresponding object fusion features in adjacent video frames, object matching is performed on adjacent video frames in the same training video frame sequence to determine the object sub-images that match within a set of adjacent video frames, thus obtaining the object matching results for each set of adjacent video frames. The object sub-images that match within adjacent video frames can be considered the representation of the same object in adjacent video frames and can be used to determine two adjacent trajectory points of the same object. Finally, based on the object matching results of each group of adjacent video frames in the same training video frame sequence, trajectory aggregation is performed to generate the target object subgraph sequence corresponding to the training object trajectory. From the object matching results corresponding to each group of adjacent video frames in the same training video frame sequence, the object matching results belonging to the same object are identified. Based on the object matching results belonging to the same object, the target object subgraph sequence corresponding to the training object trajectory is obtained.

[0127] The target object subgraph sequence can be used to generate object trajectories. The target object subgraphs in the target object subgraph sequence can be used as trajectory points, and the object trajectory can be obtained based on each trajectory point.

[0128] It is understandable that during the model training process, a new initial object representation model can be obtained after each model iteration, and a new training image set can be generated based on the new initial object representation model.

[0129] In one embodiment, the training video frame sequence can be obtained by sampling the training video, which can be random sampling or uniform sampling. For example, for a training video with a frame rate of 25fps, 5 frames are extracted per second to form the training video frame sequence.

[0130] In one embodiment, video frames from a training video frame sequence are input into an object detection model. The object detection model outputs object prediction boxes and their corresponding prediction probabilities. The prediction probabilities characterize the probability that an object exists within the object prediction box. Image regions containing object prediction boxes with prediction probabilities greater than a preset probability can be designated as object sub-images.

[0131] In the above embodiments, object detection, model processing, object matching, and trajectory aggregation can intelligently extract object trajectories and target object subgraph sequences corresponding to the object trajectories from the video frame sequences corresponding to the video, reducing manual costs and improving data processing efficiency.

[0132] In one embodiment, based on each object subgraph in adjacent video frames and the corresponding object fusion features, object matching is performed on adjacent video frames in the same training video frame sequence to obtain object matching results for each group of adjacent video frames, including:

[0133] The process begins by determining the target video frame from the current video frame sequence; then, each object subgraph corresponding to the target video frame is sequentially used as the target object subgraph; based on the feature distance between the object fusion features corresponding to the target object subgraph and the object fusion features corresponding to each object subgraph in the target backward video frame, candidate object subgraphs corresponding to the target object subgraph are determined from each object subgraph corresponding to the target backward video frame; candidate object subgraphs whose positional distance to the target object subgraph is less than a distance threshold are used as the backward matching object subgraphs corresponding to the target object subgraph; the target backward video frame is the adjacent backward video frame of the target video frame; based on each target object subgraph and its corresponding backward matching object subgraph, the object matching result between the target video frame and the target backward video frame is obtained; the next video frame is obtained from the current video frame sequence as the target video frame, and the process of sequentially using each object subgraph corresponding to the target video frame as the target object subgraph is repeated until the object matching results corresponding to each group of adjacent video frames in the current video frame sequence are obtained.

[0134] The current video frame sequence refers to the currently processed training video frame sequence. The current video frame sequence can be any one of multiple training video frame sequences.

[0135] The target backward video frame is the adjacent backward video frame of the target video frame. An adjacent backward video frame of the target video frame refers to a video frame that follows and is adjacent to the target video frame. For example, if the current video frame sequence is video frame A-video frame B-video frame C-video frame D, and the target video frame is video frame A, then the target backward video frame is video frame B; if the target video frame is video frame B, then the target backward video frame is video frame C; and if the target video frame is video frame C, then the target backward video frame is video frame D.

[0136] The candidate object subgraph corresponding to the target object subgraph is the object subgraph with the smallest feature distance to the target object subgraph in the backward video frame of the target. The candidate object subgraph corresponding to the target object subgraph can be considered as the object subgraph most similar to the target object subgraph in the backward video frame of the target. The backward matching object subgraph corresponding to the target object subgraph is the object subgraph that matches the target object subgraph in the backward video frame of the target. The backward matching object subgraph corresponding to the target object subgraph can be considered as the object subgraph in the backward video frame of the target object subgraph that is not only similar to the target object subgraph but also belongs to the same object.

[0137] Feature distance refers to the distance between different features, used to characterize the differences between them. The feature similarity between fused features of different objects can be calculated as the feature distance between fused features of different objects; for example, the Euclidean distance or the cosine distance between fused features of different objects can be used as the feature distance. Positional distance refers to the distance between the positions of different object sub-images within a video frame. For example, the coordinate distance between the center coordinates of different object sub-images can be calculated as the positional distance.

[0138] Specifically, when performing object matching, the computer device searches for an object subgraph in the subsequent video frames that matches a certain object subgraph in the starting video frame, for a set of adjacent video frames. Based on the search results of all object subgraphs in the starting video frame, the object matching results for a set of adjacent video frames are obtained. The object matching method is the same for each set of adjacent video frames in the same training video frame sequence, thus ultimately obtaining the object matching results for each set of adjacent video frames in the same training video frame sequence.

[0139] During object matching, an arbitrary sequence of training video frames is selected as the current video frame sequence. The first video frame in the current sequence is selected as the target video frame. Each object subgraph corresponding to the target video frame is then used as a target object subgraph. The backward matching object subgraphs corresponding to each target object subgraph are then searched sequentially in the target backward video frames. Based on the target object subgraphs with backward matching object subgraphs and their corresponding backward matching object subgraphs, the object matching result between the target video frame and the target backward video frame is obtained. The next video frame in the current video frame sequence is selected as the new target video frame, and the same method is used to obtain the object matching result between the new target video frame and the target backward video frame. This process is repeated until the object matching results for each group of adjacent video frames in the current video frame sequence are obtained.

[0140] When searching for the backward matching object subgraph corresponding to the target object subgraph, the feature distances between the object fusion features corresponding to the target object subgraph and the object fusion features corresponding to each object subgraph in the target backward video frame are calculated. Based on these feature distances, candidate object subgraphs corresponding to the target object subgraph are determined from each object subgraph in the target backward video frame. Specifically, the object subgraph corresponding to the smallest feature distance in the target backward video frame is selected as the candidate object subgraph corresponding to the target object subgraph. Considering that similar object subgraphs do not necessarily belong to the same object, the positional distance between object subgraphs is further used to determine whether the candidate object subgraph belongs to the same object as the target object subgraph. The positional distance between the target object subgraph and its corresponding candidate object subgraph is calculated, and the candidate object subgraph whose positional distance from the target object subgraph is less than a distance threshold is selected as the backward matching object subgraph corresponding to the target object subgraph. It can be understood that the movement distance of the same object between adjacent video frames is usually small. Therefore, if the positional distance between the target object subgraph and its corresponding candidate object subgraph is too large, it can be indicated that the two object subgraphs do not belong to the same object.

[0141] The distance threshold can be set according to actual needs. The distance threshold can be a fixed value or it can change dynamically.

[0142] In the above embodiments, during object matching, candidate object subgraphs corresponding to the target object subgraph are first determined from each object subgraph corresponding to the target backward video frame based on feature distance. Then, backward matching object subgraphs corresponding to the target object subgraph are determined from the candidate object subgraphs based on positional distance. This improves the accuracy of the target object subgraph and its corresponding backward matching object subgraph belonging to the same object. This object matching method ensures the accuracy of the final object matching results for each group of adjacent video frames.

[0143] In one embodiment, the object matching result includes an object subgraph pair consisting of a target object subgraph and a corresponding backward matching object subgraph, and a trajectory identifier corresponding to the object subgraph pair. Different object subgraph pairs with overlapping object subgraphs have the same trajectory identifier.

[0144] Trajectory aggregation is performed based on the object matching results of each group of adjacent video frames in the same training video frame sequence to obtain multiple target object sub-graph sequences corresponding to the training object trajectories, including:

[0145] From the object matching results corresponding to each group of adjacent video frames in the current video frame sequence, obtain the object subgraph pairs corresponding to the current trajectory identifier to form the current object subgraph pair set corresponding to the current trajectory identifier; sort each object subgraph in the current object subgraph pair set according to the video frame order of the corresponding video frames to obtain the current initial object subgraph sequence corresponding to the current trajectory identifier; statistically analyze the fusion features of each object corresponding to the current initial object subgraph sequence to obtain the current object center feature; based on the feature distance between each object fusion feature corresponding to the current initial object subgraph sequence and the current object center feature, delete a first preset number of object subgraphs from the current initial object subgraph sequence according to the feature distance from largest to smallest to obtain the target object subgraph sequence corresponding to the current trajectory identifier.

[0146] The trajectory identifier is used to identify the trajectory. Each object subgraph pair belonging to the same object trajectory corresponds to the same trajectory identifier. Different object subgraph pairs with overlapping object subgraphs also correspond to the same trajectory identifier. For example, object subgraph A1 in video frame A and object subgraph B4 in video frame B form one object subgraph pair; object subgraph B4 in video frame B and object subgraph C4 in video frame C form another object subgraph pair. Since both object subgraph pairs contain object subgraph B4 from video frame B, these two object subgraph pairs are different object subgraph pairs with overlapping object subgraphs. These two object subgraph pairs correspond to the same trajectory identifier. Object subgraphs A1, B4, and C4 can reflect the appearance of the same object in the three video frames, equivalent to the trajectory points corresponding to the same object in the three video frames respectively.

[0147] The current trajectory identifier refers to the trajectory identifier currently being processed, which can be any trajectory identifier from the matching results of various objects corresponding to the current video frame sequence. The first preset number can be set according to actual needs.

[0148] Specifically, when performing trajectory aggregation, the trajectory identifiers corresponding to each object subgraph pair in the object matching results of each group of adjacent video frames in the same video frame sequence can be used to aggregate the object subgraph pairs corresponding to the same trajectory identifier to obtain the initial object subgraph sequence corresponding to the trajectory identifier. The initial object subgraph sequence can be directly used as the target object subgraph sequence. However, in order to improve accuracy, the initial object subgraph sequence can be further cleaned to obtain the target object subgraph sequence.

[0149] When performing trajectory aggregation on the object matching results of each group of adjacent video frames in the current video frame sequence, each object sub-graph pair corresponding to the current trajectory identifier is obtained from the object matching results of each group of adjacent video frames in the current video frame sequence. These object sub-graph pairs are then grouped into a set of current object sub-graph pairs corresponding to the current trajectory identifier. The object sub-graphs in this set are then arranged according to the video frame order of their respective video frames to obtain the current initial object sub-graph sequence. Further, the fusion features of each object corresponding to the current initial object sub-graph sequence are statistically analyzed to obtain the current object center feature. For example, the mean of the fusion features of each object corresponding to the current initial object sub-graph sequence can be calculated as the current object center feature, or the median value among the fusion features of each object corresponding to the current initial object sub-graph sequence can be obtained as the current object center feature. Then, using the current object center feature as a benchmark, the feature distance between each fusion feature of the current initial object sub-graph sequence and the current object center feature is calculated. The smaller the feature distance corresponding to an object subgraph, the more similar the image content reflected by that object subgraph and other object subgraphs in the sequence, and it can be considered that the object subgraph and other object subgraphs in the sequence belong to the same object. To avoid the existence of object subgraphs corresponding to other objects in the current initial object subgraph sequence, a first preset number of object subgraphs are deleted from the current initial object subgraph sequence according to the feature distance from largest to smallest. The current initial object subgraph sequence after deletion is used as the target object subgraph sequence corresponding to the current trajectory identifier.

[0150] The trajectory aggregation method is the same for each trajectory identifier, and finally, multiple target object subgraph sequences can be obtained for each trajectory identifier.

[0151] In the above embodiments, when performing trajectory aggregation, an initial object subgraph sequence corresponding to an object can be quickly obtained from the object matching results based on the trajectory identifier. The initial object subgraph sequence is then cleaned based on the object center feature to obtain the target object subgraph sequence. Cleaning can reduce the probability of different target object subgraphs corresponding to different objects appearing in the target object subgraph sequence, thereby improving the accuracy of the target object subgraph sequence.

[0152] In one embodiment, the distance threshold is dynamically changed. The distance threshold is obtained by statistically analyzing the positional distance between the updated target object sub-graph and the corresponding backward matching object sub-graph after completing object matching of a second preset number of adjacent video frames.

[0153] Specifically, the distance threshold used to determine the backward matching object subgraph is dynamically changed. After completing object matching of a second preset number of adjacent video frames, the positional distance between the newly added target object subgraph and the corresponding backward matching object subgraph is statistically analyzed, and a new distance threshold is obtained based on the statistical results. For example, the mean of each newly added positional distance is calculated as the distance threshold, and the median of each newly added positional distance is calculated as the distance threshold. In subsequent object matching of the second preset number of adjacent video frames, the backward matching object subgraph is determined based on the new distance threshold.

[0154] The second preset quantity can be set according to actual needs.

[0155] In the above embodiments, the distance threshold is dynamically changed, which can improve the accuracy of object matching.

[0156] In a specific embodiment, the trajectory mining process is illustrated using a game character as an example. Video frames from a game video are input into a hero detection model for hero detection, and the model outputs hero detection boxes. Each hero detection box has a corresponding detection confidence level, which represents the probability that the object within the detection box is a hero. Considering that the detection results of the hero detection model cannot be guaranteed to be 100% accurate, some heroes may have low detection confidence levels due to sideways movement or changes in action, and thus be incorrectly filtered out by the detection threshold. These filtered-out detection results greatly help to enhance the diversity of hero characters and improve the model's classification branch recognition performance. Therefore, for those true heroes missed due to low detection confidence levels in the hero detection model, they need to be added to the model training data.

[0157] First, frames are extracted from the game video used as training material, and a hero detection model is used to detect the hero in each frame. The detection threshold of this model is lower than the threshold used in business applications; for example, if the business application uses a threshold of 0.3, then 0.1 can be used here. Then, the current stage model (i.e., the current initial object representation model) is used to extract the hero fusion features (i.e., object fusion features) from each hero image, and object trajectories are generated according to a trajectory strategy. The trajectory strategy is as follows:

[0158] The first step is to determine the starting point of the hero's trajectory based on the detection results of the first frame in the video frame sequence.

[0159] For the detection result of the first frame in the video frame sequence, which contains multiple detection boxes with confidence levels, the detection boxes with confidence levels greater than the first detection threshold are selected as the starting point of the hero trajectory. Record Information 1, which may include the hero trajectory sequence number, the order in the trajectory occurrence time, the hero coordinates, the hero fusion feature, the frame sequence number, and the video ID of each detection box with a confidence level greater than the first detection threshold. Initialize the positional distance threshold thr0 between heroes in the preceding and following frames on the hero trajectory; for example, set the positional distance threshold thr0 to 1 / 20 of the shortest side of the video. Set the distance update queue D to empty. The following steps will then mine detection boxes belonging to a certain hero from the detection boxes with confidence levels between the second and first detection thresholds. Since the first detection threshold is used to determine the starting point of the hero trajectory, it is necessary to ensure that the starting point of the hero trajectory is correct; therefore, the first detection threshold is greater than the second detection threshold. For example, the first detection threshold is 0.4, and the second detection threshold is 0.1.

[0160] The second step is to aggregate the hero trajectories for all detection results S in the backward first frame (which is the prediction frame).

[0161] For a hero x (e.g., the first hero, with a trajectory number of 1) in Information 1, calculate the Euclidean distance between the hero fusion feature of hero x and the hero fusion features of all heroes S in the prediction frame. Select the hero y with the closest distance. If the Euclidean distance between the detection box of hero y and the detection box of hero x in the previous frame is less than the hero distance threshold thr0, it means that hero y is the performance of hero x in the next frame. In this case, x and y are two photos of the same hero trajectory. Record Information 2, which includes the hero trajectory number corresponding to each detection box between the second and first detection thresholds, the ranking in the trajectory occurrence time, the hero coordinates, the hero fusion feature, the frame number, and the video ID. In this case, the hero trajectory number of y is the same as that of x, but the ranking number of y is 2 and the ranking number of x is 1. At the same time, save the detection box distance between x and y to D.

[0162] For the remaining heroes in the first frame of the video frame sequence, find their trajectory hero map in the next frame using the same method, and update D accordingly. If a hero does not have a matching hero in the next frame, it does not need to be recorded.

[0163] For all heroes in the second frame of the video frame sequence, if there is a hero that is not matched in the previous frame and has a confidence level greater than the first detection threshold, then this hero is a newly added hero and is recorded in information 1. If it is matched in the previous frame, then information 2 is recorded.

[0164] In addition, after calculating hero matchmaking for every 25 frames, the thr0 threshold is updated: the average of all distances in D is used as the new thr0, and then the data in D is cleared.

[0165] By doing so, the initial hero trajectories of all heroes in the video frame sequence are eventually obtained.

[0166] The third step is to manually label the hero corresponding to each trajectory and then clean it.

[0167] For a single initial hero trajectory, calculate the central fusion feature for all hero fusion features corresponding to the trajectory. Calculate the feature distance between each hero fusion feature and the central fusion feature. Sort these feature distances in ascending order. Remove the 10% of samples farthest from the central fusion feature, and the remaining 90% of samples form the target hero trajectory. Label the target hero trajectory, indicating which hero it corresponds to, and record the samples that do not belong to the same hero.

[0168] The final result is hero trajectories labeled with hero names and with noise samples removed. The noise samples are the target hero trajectories that were found to include multiple heroes after labeling. Compared to labeling each image in a video frame sequence with which hero is present, this method only requires labeling each trajectory once. This method infers the hero trajectory from the video trajectory and then cleans and labels the hero's name. This method requires less labeling, is faster, and requires less labeling cost.

[0169] In one embodiment, obtaining the negative sample image corresponding to the current positive sample image pair from other positive sample image pairs besides the current positive sample image pair includes:

[0170] The training image is obtained from the current positive sample image pair as the target positive sample image, and the training image is obtained from each other positive sample image pair as the candidate image. Based on the image distance between the target positive sample image and each candidate image, the candidate images are sorted in ascending order of image distance to obtain a candidate image sequence. A third preset number of candidate images are deleted from the candidate image sequence in ascending order of image distance to obtain an updated image sequence. A fourth preset number of candidate images are obtained from the updated image sequence in ascending order of image distance to serve as the negative sample images corresponding to the current positive sample image pair.

[0171] Image distance is used to characterize the differences between different images. The similarity between different images can be calculated as the image distance between them. For example, the feature distance between two images can be based on the feature distance between the fused object features corresponding to each image. The third preset number and the fourth preset number can be set according to actual needs.

[0172] Specifically, the computer device selects any training image from the current positive sample image pair as the target positive sample image, and selects any training image from each of the other positive sample image pairs as candidate images. The image distance between the target positive sample image and each candidate image is calculated. The computer device sorts the candidate images according to their image distance from smallest to largest, obtaining a candidate image sequence. A third preset number of candidate images are then removed from the candidate image sequence according to their image distance from smallest to largest, resulting in an updated image sequence. The smaller the image distance, the more similar the target positive sample image and the candidate images are; candidate images that are very similar to the target positive sample image cannot be used as negative sample images corresponding to the target positive sample image. Furthermore, to ensure that the training image set contains training image groups containing difficult negative samples, the computer device selects a fourth preset number of candidate images from the updated image sequence according to their image distance from smallest to largest, using them as negative sample images corresponding to the current positive sample image pair, thus obtaining each negative sample image corresponding to the current positive sample image pair. Among the negative sample images, the negative sample images with relatively small image distances are considered difficult negative samples, which helps improve the model training quality. When generating training image sets, a training image set is obtained based on the current positive sample image pair and a corresponding negative sample image. If there are multiple negative sample images for the current positive sample image pair, multiple training image sets can be obtained in the end.

[0173] In the above embodiments, a third preset number of candidate images are deleted from the candidate image sequence in ascending order of image distance to obtain an updated image sequence. A fourth preset number of candidate images are obtained from the updated image sequence in ascending order of image distance and used as the negative sample images corresponding to the current positive sample image pairs. This ensures that the negative sample images are difficult negative samples, which helps to improve the training quality of the model.

[0174] In one specific embodiment, the process of mining image triples is described. Two images in adjacent frames of the same object trajectory are very similar, and these two images in adjacent frames of the same object trajectory are combined to form a positive sample pair.

[0175] Before the start of each epoch (representing the model iteration round or training round), 10 pairs of positive samples are randomly selected from each object trajectory to generate a similar sample pair dataset. The similar sample pair dataset is then mined using negative samples to obtain triple samples (i.e., image triples).

[0176] Considering the limited memory of computer GPUs, the entire set of positive sample pairs needs to be fed into the GPU in batches during the overall training process. Therefore, the mining of negative samples within each positive sample pair will be performed within a single batch. The mining within each batch is as follows:

[0177] For all bs positive sample pairs in a batch, perform the following operations: For the x-anchor image of a sample pair x (randomly select one image from the sample pair as the anchor image), calculate the distance between the x-anchor and the remaining bs-1 sample pairs (randomly select one image from each pair). Sort the images by distance from smallest to largest, remove the top 5 images, and take the top 20 samples as hard negative samples. Each hard negative sample is paired with sample pair x to form a triplet. Therefore, each sample pair generates 20 triplets, and the entire batch yields 20 * bs triplets. To ensure effective negative sample pair mining, bs needs to be set to a relatively large value, such as 128.

[0178] Since deduplication tasks require learning the features of extremely similar samples, assuming that smaller distances indicate greater similarity, and given the low probability that two images are extremely similar in massive datasets, directly removing the top 5 similar samples effectively avoids situations where two images are extremely similar. The remaining samples can form the hard-negative samples in the triplet. When adjusting model parameters based on the fusion loss, hard-negative samples are more valuable than negative samples. Calculating the fusion loss based on image triples and adjusting model parameters based on the fusion loss allows the model to learn discriminative representations.

[0179] In one embodiment, when the current feature is an object association feature, the current label is an object association label, and the current classification loss is an object association classification loss; when the current feature is an object feature, the current label is an object label, and the current classification loss is an object classification loss.

[0180] The current process for calculating classification loss includes the following steps:

[0181] Obtain the prototype object feature sets corresponding to various current labels. Each prototype object feature set includes multiple prototype object features corresponding to the same current label. The prototype object features are obtained from the current feature library by selecting the current feature that is closest to the cluster center feature corresponding to the current label. The cluster center feature is obtained by clustering the current features corresponding to the same current label in the current feature library. From each prototype object feature set, select the prototype object feature with the smallest feature distance to the current feature corresponding to the current training image as the reference object feature corresponding to the current training image. Based on the difference between the current feature and the reference object feature corresponding to the same training image, obtain the current classification loss.

[0182] The prototype object feature set includes multiple prototype object features corresponding to the same current label. For example, if the current feature is an object feature and the current label is an object label, and the object labels include three types: role A, role B, and role C, role A has a corresponding prototype object feature set a, role B has a corresponding prototype object feature set b, and role C has a corresponding prototype object feature set c.

[0183] The current feature library refers to the feature set corresponding to the current feature, which includes multiple current features. When the current feature is an object feature, the current feature library is an object feature library; when the current feature is an object association feature, the current feature library is an object association feature library. Clustering the current features corresponding to the same current label in the current feature library yields multiple cluster center features for each current label. Cluster center features are more representative. The current feature corresponding to the current label refers to the current feature corresponding to the training image belonging to the current label. In the current feature library, the current feature closest to each cluster center feature corresponding to a current label is obtained from the current features corresponding to that current label. These closest current features are then used as the prototype object features corresponding to that current label. Compared to the current features corresponding to the current label, prototype object features are fewer in number and more representative, better representing the commonalities of current features corresponding to training images belonging to the same label. Other current labels can use the same method to obtain their corresponding prototype object features. A prototype object feature set is formed by the features of each prototype object corresponding to a current label.

[0184] For example, taking object features as an example, if role A corresponds to 10,000 training images, we can obtain the object features corresponding to each of the 10,000 training images. Then, we perform K-means clustering (K-means is a clustering algorithm) on these 10,000 object features, grouping them into 10 classes, resulting in 10 cluster center features. For any given cluster center feature, we select the object feature closest to that cluster center feature from the 10,000 object features as the prototype object feature. Since there are 10 cluster center features, we can ultimately obtain 10 prototype object features. These 10 prototype object features form the prototype object feature set corresponding to role A.

[0185] Specifically, object association classification loss and object classification loss can be calculated using a loss calculation method based on prototype object features. The computer device acquires prototype object feature sets corresponding to various current labels. From each prototype object feature set, reference object features corresponding to each training image are acquired. The current classifier sub-loss is calculated based on the difference between the current features and reference object features corresponding to the same training image. The current classification loss is obtained based on the current classifier sub-loss corresponding to each training image.

[0186] When obtaining the reference object features corresponding to the training images, any training image is selected as the current training image. The feature distances between each prototype object feature in the prototype object feature set and the current feature corresponding to the current training image are calculated. From each prototype object feature set, the prototype object feature corresponding to the minimum feature distance is selected as the reference object feature corresponding to the current training image. Other training images can obtain their corresponding reference object features in the same way.

[0187] In the above embodiments, the object association classification loss or object classification loss is calculated based on the loss calculation method of the prototype object features. When adjusting the model parameters, it can avoid model overfitting and improve the model training quality.

[0188] In one embodiment, the process of determining the feature set of the prototype object includes the following steps:

[0189] Within the initial model iteration rounds, every fifth preset number of batches trained, the prototype object feature sets corresponding to various current labels are determined based on the current feature library; within the backward model iteration rounds within the initial model iteration rounds, every sixth preset number of batches trained, the prototype object feature sets corresponding to various current labels are determined based on the current feature library; the sixth preset number is greater than the fifth preset number; after each batch is trained, the obtained current features are stored in the current feature library, and after each prototype object feature set is determined, the current feature library is cleared.

[0190] The initial model iteration range refers to the range of model iterations consisting of at least one initial model iteration across all model iterations. The backward model iteration range refers to the range of model iterations consisting of at least one iteration performed after the initial model iteration range across all model iterations.

[0191] The fifth preset quantity can be set according to actual needs, and the sixth preset quantity can also be set according to actual needs. The sixth preset quantity is greater than the fifth preset quantity.

[0192] Specifically, the prototype object feature set can be dynamically updated. In the initial training phase of the model, because the model parameters are not very accurate, the model's feature representation ability is not very accurate, so it is updated at a higher frequency. In the subsequent training phase of the model, because the model's feature representation ability gradually becomes stronger, it is updated at a lower frequency.

[0193] For each training batch, the current features obtained in that batch are stored in the current feature library. Within the initial model iteration epochs, every fifth preset number of training batches, the prototype object feature sets corresponding to each current label are determined based on the current feature library. Each time a prototype object feature set is determined, the current feature library is cleared. Within the subsequent model iteration epochs within the initial model iteration epochs, every sixth preset number of training batches, the prototype object feature sets corresponding to each current label are determined based on the current feature library. Each time a prototype object feature set is determined, the current feature library is cleared.

[0194] For example, in the first epoch of training, the prototype object feature set is updated after every 10 batches, and in subsequent epochs of training, the prototype object feature set is updated every 50 batches.

[0195] In the above embodiments, within the initial model iteration rounds, every fifth preset number of batches trained, the prototype object feature sets corresponding to various current labels are determined based on the current feature library. Within the subsequent model iteration rounds within the initial model iteration rounds, every sixth preset number of batches trained (greater than the fifth preset number), the prototype object feature sets corresponding to various current labels are determined based on the current feature library. This approach can improve model training speed while ensuring model training quality.

[0196] In one specific embodiment, the initial object representation model includes an object classification network used to extract object features. (See reference...) Figure 3 This section uses an object classification network as an example to illustrate the loss calculation method based on prototype object features. The object classification network includes pooling layers, fully connected layers, and matching layers. The fully connected layers output object features, and the matching layers determine reference object features. When calculating the object classification loss, prototype object feature sets corresponding to various object labels are obtained. From each prototype object feature set, reference object features corresponding to the training image are obtained based on the object features corresponding to the training image. The object classification loss is then calculated based on the object features corresponding to the same training image and the reference object features.

[0197] The prototype object feature set is updated asynchronously, meaning the update is not synchronized with model training. This means the prototype object feature set does not need to be updated with each batch when model parameters are updated. Within each batch, object features are saved to an object feature library. Preset conditions control when the prototype object feature set is calculated based on this library. When the preset conditions are met, the prototype object feature set is updated using the object feature library. After the update, the object feature library is cleared, allowing the updated model to extract new object features that can then be stored in the library, supporting future updates to the prototype object feature set.

[0198] During the initialization phase, for each object, the object features output by the object classification network of the model are first obtained from the training images. Assuming there are 10,000 samples for a certain object, 10,000 object features are obtained. These 10,000 object features are then clustered into 10 classes, resulting in 10 cluster center features. From these 10,000 object features, the object features closest to the 10 cluster center features are selected, resulting in 10 object features. These 10 object features are stored as the prototype object features of the object in the corresponding prototype object feature set. After the initialization phase, the prototype object feature sets corresponding to all heroes are obtained.

[0199] After the prototype object feature set is updated for the first time or at some point, the update status is recorded as successful, the number of trained batches is recorded as 0, and the object feature library is cleared. For each subsequent batch, the number of trained batches is incremented by 1, and the object features obtained during training are stored in the object feature library. When the number of trained batches reaches the preset number of updates, but the number of object features corresponding to a certain object in the object feature library is less than the preset number (e.g., 10), the update status is recorded as failed, the update stops, and training continues for the next batch. When the number of trained batches reaches the preset number of updates, and the number of object features corresponding to each type of object in the object feature library is greater than or equal to the preset number, the update status is recorded as failed, and the prototype object feature set is updated based on the object feature library.

[0200] In one embodiment, the current feature and the reference object feature are feature vectors of a preset length.

[0201] Based on the difference between the current features and the reference object features corresponding to the same training image, the current classification loss is obtained, including:

[0202] Obtain a random feature vector of a preset length, and retain the vector positions corresponding to data whose values ​​are greater than or equal to a preset threshold in the random feature vector. Based on the data differences between the current feature and the reference object feature at the same retained position in the same training image, obtain the current classification loss.

[0203] Here, the current feature and the reference object feature are feature vectors of a preset length. For example, the current feature and the reference object feature are 128-bit feature vectors. Random feature vectors refer to randomly generated feature vectors.

[0204] Specifically, when calculating the current classification loss based on the current features and the reference object features, in addition to directly calculating the current classification loss based on the complete current features and the reference object features, one can also randomly discard some data points from the current features and the reference object features, and calculate the current classification loss based on the remaining data points from the current features and the reference object features. The current classification loss calculated by randomly discarding data helps to avoid model overfitting.

[0205] The computer device randomly generates a random feature vector of a preset length. Vector positions corresponding to data with values ​​less than a preset threshold are discarded, while vector positions corresponding to data with values ​​greater than or equal to the preset threshold are retained. The computer device calculates the current classifier loss based on the data differences between the current feature and the reference object feature at the same retained position for the same training image. The current classification loss is then calculated based on the current classifier loss for each training image.

[0206] It is understandable that a new random feature vector can be generated for each batch of training, or a new random feature vector can be generated for each preset number of batches of training.

[0207] In the above embodiments, the object association classification loss or object classification loss is a random dropout loss based on the features of the prototype object. When adjusting the model parameters, it can avoid model overfitting and improve the model training quality.

[0208] In one embodiment, when the current feature is an object association feature, the current label is an object association label, and the current classification loss is an object association classification loss; when the current feature is an object feature, the current label is an object label, and the current classification loss is an object classification loss.

[0209] The current process for calculating classification loss includes the following steps:

[0210] The predicted probability of the training image is obtained based on the current features corresponding to the training image; the current classification loss is obtained based on the predicted probability of the same training image and the current label.

[0211] The prediction probability refers to the probability that the training image predicted by the model belongs to each of the various current labels.

[0212] Specifically, object association classification loss and object classification loss can be calculated using a loss method based on predicted probabilities. The computer device obtains the predicted probability corresponding to the training image based on the current features corresponding to the training image, obtains the current sub-loss based on the difference between the predicted probability corresponding to the same training image and the current label, and obtains the current classification loss based on the current sub-loss corresponding to each training image.

[0213] In one embodiment, the cross-entropy loss function is used to calculate the current classification loss, and the calculation formula is as follows:

[0214]

[0215] Where, p ic y represents the predicted probability that image i belongs to category c. ic This indicates whether the current label of image i is c. When it is c, y ic =1, otherwise y ic =0. N is the number of images in the batch.

[0216] In the above embodiments, the object association classification loss or object classification loss is calculated based on the loss calculation method of prediction probability, which can quickly calculate the object association classification loss or object classification loss.

[0217] In a specific embodiment, the loss calculation method based on prototype object features is illustrated using object classification loss as an example. The model extracts object features from the training images to obtain the object features corresponding to the training images. The object features are 1x128 dimensional feature vectors. If the cross-entropy loss function mentioned above is directly used, the object features can easily change from high-dimensional features with rich information to one-hot outputs (i.e., the predicted probability is 1 only for the corresponding object label and 0 for all other object labels). Such rapid changes will directly generate a large loss value. This loss value, after backpropagation to adjust the model parameters, will cause the overall model to fit the object classification task more, that is, overfit on the object classification task, thus making the model perform poorly on other tasks.

[0218] To prevent this problem, this method replaces the classification loss with a regression loss, specifically using MSE (Mean Squared Error) loss to match object features with prototype object features (also known as category prototypes) using Euclidean distance. This includes the following two matching methods:

[0219] Matching-based loss: For each training image, the model outputs the object features corresponding to the training image, calculates the Euclidean distance between the object features and all classification prototypes (i.e., all prototype object features), and takes the category of the prototype with the closest Euclidean distance (denoted as the reference prototype, i.e., the reference object feature) as the prediction result. For example, if the object feature of a training image has the closest Euclidean distance to one of the prototype object features of character A, then the model considers the training image to correspond to character A. Furthermore, if the object label of the training image is also character A, it indicates that the training image has been correctly classified; if the object label of the training image is not character A, it indicates that the model has made a wrong prediction.

[0220] The loss between the object features and the reference prototype is calculated using MSE, and the formula is as follows:

[0221]

[0222] Among them, y i The reference object features corresponding to the i-th image, Let be the object features corresponding to the i-th image, and n be the number of images in the training image set.

[0223] Loss based on random dropout: When calculating regression loss, instead of using the full 1x128 feature set, 128 random numbers between 0 and 1 are generated using a normal distribution during training for a given batch. The positions of these random numbers less than 0.3 are designated as dropout positions, and the remaining positions are retained. For the object classification task in this batch, only the loss of the retained positions is calculated. For example, in a batch, if only the 1st, 2nd, and 4th random numbers are less than 0.3, then the 1st, 2nd, and 4th positions in the object feature vector and the reference prototype feature vector are discarded. The loss for this batch is calculated based on the MSE loss of the 3rd, 5th, 6th, 7th...128th positions. At the start of the next batch, 128 new random numbers are generated, and the retained positions for loss calculation are redefined.

[0224] In one embodiment, a fusion loss is obtained based on the fusion features of each object corresponding to the same training image group, including:

[0225] A first difference is obtained based on the difference between the fusion features of each object corresponding to the positive sample image pair in the same training image group; a reference positive sample image is determined from the positive sample image pair; a second difference is obtained based on the difference between the fusion features of the objects corresponding to the reference positive sample image and the negative sample image in the same training image group; and a fusion loss is obtained based on the distance between the first difference and the second difference corresponding to the same training image group and a preset distance.

[0226] Specifically, when calculating the fusion loss, the computer device obtains a first difference based on the difference between the object fusion features corresponding to positive sample image pairs in the same training image group, and obtains a second difference based on the difference between the object fusion features corresponding to any positive sample image pair and negative sample image in the same training image group. The fusion loss is obtained based on the first and second differences corresponding to the same training image group.

[0227] When calculating the second difference, the computer device obtains an arbitrary positive sample image from the positive sample image pair as a reference positive sample image, and obtains the first difference based on the difference between the object fusion features corresponding to the reference positive sample image and the negative sample image in the same training image group.

[0228] When obtaining the fusion loss based on the first and second differences corresponding to the same training image group, the fusion loss can be specifically based on the distance between the first and second differences corresponding to the same training image group and a preset distance. For example, the distance between the first and second differences and the preset distance are fused to obtain the fusion distance, and the maximum value between the fusion distance and the preset value is taken as the target distance. The fusion loss is then obtained based on the target distance.

[0229] In one embodiment, the fusion loss is calculated using the following formula:

[0230] L tri =max(||x a -x p ||-||x a -x n ||+α,0)

[0231] Where α is the preset distance, set to 0.6. a This represents the object fusion feature corresponding to the reference positive sample image, that is, the object fusion feature corresponding to the anchor image in the triplet. p This represents the object fusion feature corresponding to the images other than the reference positive image in a positive sample image pair, i.e., the object fusion feature corresponding to the positive image in the triplet. The anchor image and the positive image constitute the positive sample pair in the triplet. n This represents the object fusion feature corresponding to the negative sample image, that is, the object fusion feature corresponding to the negative image in the triplet. ||x a -x p || represents x a and x p L2 distance, ||x a -x n || represents x a and x n The L2 distance. triThe goal is to make the feature distance between the anchor and the negative feature greater than 0.6 compared to the feature distance between the anchor and the positive feature.

[0232] In the above embodiments, the fusion loss is calculated based on the distance between the first difference and the second difference corresponding to the same training image group and the preset distance. Adjusting the model parameters based on such fusion loss can improve the feature discrimination ability of the model, so that the object fusion feature discrimination of the positive sample image and the negative sample image respectively is large.

[0233] In one embodiment, the initial object representation model includes an object association classification network, an object classification network, and a feature fusion network. The object association classification network is used to obtain object association features, the object classification network is used to obtain object features, and the feature fusion network is used to obtain object fusion features.

[0234] Based on object association classification loss, object classification loss, and fusion loss, the model parameters of the initial object representation model are adjusted to obtain an intermediate object representation model, including:

[0235] The network parameters of the object association classification network are adjusted according to the first learning rate based on the object association classification loss, the network parameters of the object classification network are adjusted according to the second learning rate based on the object classification loss, and the network parameters of the object association classification network, object classification network, and feature fusion network are adjusted according to the third learning rate based on the fusion loss to obtain an intermediate object representation model; wherein the first learning rate and the second learning rate are both greater than the third learning rate.

[0236] The initial object representation model includes an object association classification network, an object classification network, and a feature fusion network. The object association classification network extracts object association features from the image to obtain the corresponding object association features. The object classification network extracts object features from the image to obtain the corresponding object features. The feature fusion network fuses the object association features and object features corresponding to the same image to obtain the corresponding object fusion features.

[0237] Specifically, when adjusting the model parameters of the initial object representation model based on object association classification loss, object classification loss, and fusion loss, the network parameters of the object association classification network are adjusted based on the object association classification loss, the network parameters of the object classification network are adjusted based on the object classification loss, and the network parameters of the object association classification network, object classification network, and feature fusion network are adjusted based on the fusion loss, so as to achieve targeted adjustments.

[0238] Furthermore, when adjusting network parameters, the network parameters of the object association classification network are adjusted according to a first learning rate based on the object association classification loss, the network parameters of the object classification network are adjusted according to a second learning rate based on the object classification loss, and the network parameters of the object association classification network, object classification network, and feature fusion network are adjusted according to a third learning rate based on the fusion loss. Since the object fusion features used to calculate the fusion loss are obtained based on the object association features and object features output by the object association classification network and the object classification network, the learning focus of the model can be placed on the learning of the object association classification network and the object classification network. Therefore, the first learning rate can be set to be greater than the third learning rate, and the second learning rate can also be greater than the third learning rate, thereby ensuring the model's fitting effect on the object association classification network and the object classification network.

[0239] In the above embodiments, during model training, the network parameters of the object association classification network and the object classification network are adjusted according to the first learning rate, the network parameters of the object classification network are adjusted according to the second learning rate, and the network parameters of the object association classification network, the object classification network, and the feature fusion network are adjusted according to the third learning rate, which is less than the first learning rate and the second learning rate, based on the fusion loss. This can balance the convergence speed of different networks and improve the model training quality.

[0240] In one embodiment, reference Figure 4 The initial object representation model consists of an object association classification network, an object classification network, and a feature fusion network. The input data for the initial object representation model is an image. The object association classification network consists of a base layer, a pooling layer 1, a fully connected layer 1 (also known as the FC1 layer), and a fully connected layer 2 (also known as the object association classification layer or FC2 layer). The fully connected layer 1 outputs object association features (also known as embedding1), and the fully connected layer 2 outputs the predicted probability.

[0241] The object association classification network and the object classification network share a base layer. The object classification network includes a base layer, pooling layer 2, fully connected layer 3 (also known as the FC3 layer), and an object matching layer (also known as the match layer). The fully connected layer 3 is used to output object features (also known as embedding 2).

[0242] The object association features and object features corresponding to the same image are concatenated and then input into the feature fusion network. The feature fusion network includes a fully connected layer 4 (also known as the FC4 layer) and a fully connected layer 5 (also known as the FC5 layer). The fully connected layer 5 is used to output the object fusion features (also known as the total embedding).

[0243] Table 1 is the model structure table for the basic layer.

[0244]

[0245] Table 2 shows the model structure of the object association classification network excluding the base layer.

[0246]

[0247]

[0248] Table 3. Model structure of the object classification network except for the base layer.

[0249] Layer name Output size Layer Pool 1x2048 Pooling layer Embedding2(FC3) 1x128 Fully connected layer Match - none

[0250] Table 4 shows the model structure of the feature fusion network.

[0251] Layer name Output size Layer Embedding (FC4) 1x256 Fully connected layer Embedding (FC5) 1x128 Fully connected layer

[0252] The object association classification network outputs the predicted probabilities of training images on the object association classification task. The object association classification loss, Loss_ce1, is calculated based on the predicted probabilities of the same training image and the object association label. The object classification network outputs the object features and reference object features corresponding to the training image. The object classification loss, Loss_selected, is calculated based on the object features and reference object features corresponding to the same training image. The feature fusion network outputs the object fusion features corresponding to the training image. The fusion loss, Loss_triplet, is calculated based on the fusion features of each object corresponding to the same group of training images. Loss_ce1 is used to update the network parameters of the object association classification network, i.e., to update the model parameters in Tables 2 and 1. Loss_selected is used to update the network parameters of the object classification network, i.e., the model parameters in Tables 3 and 1. Loss_triplet is used to update the network parameters of the object association classification network, the object classification network, and the feature fusion network; specifically, it can update the model parameters of all structures in the model except FC2 in Table 2.

[0253] During model training, Conv1-Conv5 can use the parameters of ResNet101 pre-trained on the ImageNet dataset, and newly added layers can be initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0254] In one embodiment, reference Figure 5 The initial object representation model consists of an object association classification network, an object classification network, and a feature fusion network. The input data for the initial object representation model is an image. The object association classification network consists of a shared base layer, base layer 1, fully connected layer 1, and fully connected layer 2. Fully connected layer 1 outputs object association features, and fully connected layer 2 outputs predicted probabilities.

[0255] The object classification network consists of a shared base layer, a base layer 2, a fully connected layer 3, and a fully connected layer 4 (also known as the object classification layer). The fully connected layer 3 is used to output object features, and the fully connected layer 4 is used to output predicted probabilities.

[0256] The object association features and object features corresponding to the same image are concatenated and then input into the feature fusion network. The feature fusion network includes a fully connected layer 5, which is used to output the fused object features.

[0257] The object association classification network outputs the predicted probabilities of training images for the object association classification task. The object association classification loss (Loss_ce1) is calculated based on the predicted probabilities of the same training image and the object association label. The object classification network outputs the predicted probabilities of training images for the object classification task. The object classification loss (Loss_ce2) is calculated based on the predicted probabilities of the same training image and the object label. The feature fusion network outputs the object fusion features corresponding to the training images. The fusion loss (Loss_triplet) is calculated based on the fusion features of each object in the same group of training images.

[0258] It's understandable that the object classification network could output the object features corresponding to the training image and the corresponding reference object features for the object association classification task, and then calculate the object classification loss Loss_selected1 based on the object features corresponding to the same training image and the reference object features. Alternatively, the object classification network could output the object features corresponding to the training image and the corresponding reference object features for the object classification task, and then calculate the object classification loss Loss_selected2 based on the object features corresponding to the same training image and the reference object features.

[0259] In one embodiment, the base layer is a CNN (Convolutional Neural Network).

[0260] In one embodiment, such as Figure 6 As shown, before obtaining the first training image set, the object representation model training method further includes:

[0261] Step S602: Obtain the second training image set, input the second training image set into the basic object representation model to be trained, and obtain the initial object features corresponding to each training image in the second training image set; the basic object representation model includes a feature extraction layer and a fully connected layer.

[0262] Step S604: Based on the features of each initial object corresponding to the same training image group in the second training image set, obtain the initial loss.

[0263] Step S606: Adjust the model parameters of the basic object representation model based on the initial loss until the second convergence condition is met, and obtain the trained basic object representation model.

[0264] Step S608: Based on the feature extraction layer in the trained basic object representation model, initialize the feature extraction layer in the candidate object representation model; based on the fully connected layer in the trained basic object representation model, initialize the fully connected layer in the candidate object representation model to obtain the initial object representation model.

[0265] The candidate object representation model has the same model structure as the initial object representation model. The second training set is used to train the basic object representation model. Both the basic object representation model and the candidate object representation model include a feature extraction layer and a fully connected layer. The feature extraction layer is used to extract features, and the fully connected layer is used for feature interaction. Through the feature extraction layer and the fully connected layer, features with better representational capabilities can be extracted.

[0266] The basic object representation model takes an image as input and outputs the initial object features corresponding to that image. The model consists of a feature extraction layer and a fully connected layer connected sequentially; the fully connected layer outputs the initial object features.

[0267] The input data of the candidate object representation model is an image, and the output data includes the object features corresponding to the image, object association features, and fused object features. The candidate object representation model includes at least one feature extraction layer and multiple fully connected layers. The candidate object representation model includes fully connected layers for outputting object features, fully connected layers for outputting object association features, and fully connected layers for outputting fused object features.

[0268] Similar to the first convergence condition, the second convergence condition includes, but is not limited to, at least one of the following: the initial loss is less than the corresponding second preset loss value, the number of model iterations is greater than the second preset number of iterations, the number of model iteration rounds is greater than the second preset number of iteration rounds, or the rate of change of the initial loss is less than the corresponding second preset rate of change.

[0269] Specifically, in order to improve the accuracy of the initialization parameters of the initial object representation model, the basic object representation model can be trained first, and then the parameters of the candidate object representation model can be initialized based on the trained basic object representation model, thereby obtaining the initial object representation model.

[0270] The computer device can acquire a second training image set locally or from other devices, and train the basic object representation model based on the second training image set. The computer device inputs the second training image set into the basic object representation model to be trained, obtaining the initial object features corresponding to each training image in the second training image set. Similar to the method for calculating the fusion loss, an initial loss is calculated based on the initial object features corresponding to each group of training images in the second training image set. Then, backpropagation is performed based on the initial loss to adjust the model parameters of the basic object representation model. Through multiple iterations of model training, until the second convergence condition is met, the trained basic object representation model is obtained.

[0271] Since both the basic object representation model and the candidate object representation model include feature extraction layers and fully connected layers, the computer device can initialize the feature extraction layers in the candidate object representation model based on the feature extraction layers in the trained basic object representation model, and initialize the fully connected layers in the candidate object representation model based on the fully connected layers in the trained basic object representation model, and use the initialized candidate object representation model as the initial object representation model.

[0272] In the above embodiments, the feature extraction layer in the candidate object representation model is initialized based on the feature extraction layer in the trained basic object representation model, and the fully connected layer in the candidate object representation model is initialized based on the fully connected layer in the trained basic object representation model, thus obtaining an initial object representation model. This ensures that the initial object representation model has a certain degree of accuracy and improves the convergence speed of the initial object representation model.

[0273] In one embodiment, based on the feature extraction layer in the trained base object representation model, the feature extraction layer in the candidate object representation model is initialized; based on the fully connected layer in the trained base object representation model, the fully connected layer in the candidate object representation model is initialized, resulting in an initial object representation model, including:

[0274] Based on the feature extraction layer in the trained basic object representation model, the feature extraction layer shared between the object association classification network and the object classification network in the candidate object representation model is initialized. Based on the fully connected layer in the trained basic object representation model, the fully connected layer of the object association classification network and the fully connected layer of the object classification network in the candidate object representation model are initialized to obtain the initial object representation model.

[0275] The candidate object representation model includes an object association classification network and an object classification network. Similar to the basic object representation model, the object association classification network consists of sequentially connected feature extraction layers and fully connected layers, while the object classification network also consists of sequentially connected feature extraction layers and fully connected layers. In the candidate object representation model, the object association classification network and the object classification network share the feature extraction layer. The fully connected layers of the object association classification network output object association features, while the fully connected layers of the object classification network output object features.

[0276] The candidate object representation model also includes a feature fusion network, which is used to perform feature fusion. It can be understood that the role of the feature fusion network is to perform feature fusion, and it may not include a feature extraction layer. The feature fusion network also includes fully connected layers. Considering that the network structure of the feature fusion network differs from that of the object association classification network and the object classification network, the fully connected layers of the feature fusion network in the candidate object representation model may not need to be initialized based on the fully connected layers in the trained base object representation model.

[0277] Specifically, when initializing the model parameters of the candidate object representation model based on the trained basic object representation model, the feature extraction layer shared between the object association classification network and the object classification network in the candidate object representation model can be initialized based on the feature extraction layer in the trained basic object representation model, so that the feature extraction layer in the candidate object representation model has good feature extraction capability. Based on the fully connected layer in the trained basic object representation model, the fully connected layer of the object association classification network and the fully connected layer of the object classification network in the candidate object representation model can be initialized, so that the fully connected layer of the object association classification network and the fully connected layer of the object classification network in the candidate object representation model has good feature interaction capability.

[0278] In the above embodiments, based on the feature extraction layer in the trained basic object representation model, the feature extraction layer shared between the object association classification network and the object classification network in the candidate object representation model is initialized. Based on the fully connected layer in the trained basic object representation model, the fully connected layer of the object association classification network and the fully connected layer of the object classification network in the candidate object representation model are initialized. This ensures that the initialization parameters of the object association classification network and the object classification network, which have similar structures to the basic object representation model, have a certain degree of accuracy.

[0279] In one embodiment, reference Figure 7The basic object representation model includes a feature extraction layer (also called the base layer), a pooling layer, and a fully connected layer 0 (also called the FC0 layer). The input data for the basic object representation model is images, and the output is initial object features. The initial loss, Loss_triplet, is calculated based on the initial object features corresponding to each image group from the same training image set. After training, the fully connected layer 0 (FC0) in the basic object representation model is directly used for initialization. Figure 4 Fully connected layer 1 (FC1) and fully connected layer 3 (FC3) in the basic object representation model are used for initialization. Figure 4 The basic layer in the middle.

[0280] In one embodiment, such as Figure 8 As shown, object representation model training methods also include:

[0281] Step S802: Obtain the third training image set, input the third training image set into the target object representation model, and obtain the object fusion features corresponding to each training image in the third training image set.

[0282] Step S804: Based on the fusion features of each object corresponding to the same training image group in the third training image set, the updated loss is obtained.

[0283] Step S806: Adjust the model parameters of the target object representation model based on the update loss until the third convergence condition is met, and obtain the updated object representation model.

[0284] The third training set is used to train the target object representation model. Similar to the first convergence condition, the third convergence condition includes, but is not limited to, at least one of the following: the update loss is less than the corresponding third preset loss value, the number of model iterations is greater than the third preset number of iterations, the number of model iteration rounds is greater than the third preset number of iteration rounds, or the rate of change of the update loss is less than the corresponding third preset rate of change. The updated object representation model is the finally trained object representation model.

[0285] Specifically, in order to further improve the accuracy of the target object representation model, the target object representation model can be further trained to obtain an updated object representation model with stronger feature representation capabilities.

[0286] The computer device can acquire a third training image set locally or from other devices, and train the target object representation model based on this third training image set. The computer device inputs the third training image set into the target object representation model, obtaining the object fusion features corresponding to each training image in the third training image set. Similar to the method used to calculate the fusion loss, an update loss is calculated based on the object fusion features corresponding to each group of training images in the third training image set. Then, backpropagation is performed based on the update loss to adjust the model parameters of the target object representation model. Through multiple iterations of model training, until the third convergence condition is met, an updated object representation model is obtained. The updated object representation model is used to extract the object fusion features corresponding to the input images.

[0287] In the above embodiments, further training of the target object representation model can yield a more accurate updated object representation model, and the object fusion features output by the updated object representation model are more accurate.

[0288] In one embodiment, reference Figure 9 When training the target object representation model, the object association classification network and the classification module of the object classification network are no longer updated; that is, the classification layer or matching layer of the object association classification network and the object classification network are no longer updated. The object association classification network outputs object association features, and the object classification network outputs object features. The object association features and object features corresponding to the same image are concatenated and input into the feature fusion network, which outputs object fusion features. The update loss Loss_triplet is calculated based on the object fusion features corresponding to each image in the same training image group.

[0289] In one embodiment, such as Figure 10 As shown, an object representation method is provided. Taking the application of this method to a computer device as an example, the computer device can be a terminal or a server. The method can be executed independently by the terminal or server, or it can be implemented through interaction between the terminal and the server. (Reference) Figure 10 The object representation method includes the following steps:

[0290] Step S1002: Obtain the image of the object to be processed.

[0291] Step S1004: Input the image of the object to be processed into the target object representation model to obtain the object fusion features corresponding to the image of the object to be processed; the object fusion features corresponding to the image of the object to be processed are used to determine the image that matches the image of the object to be processed.

[0292] Here, an object image refers to an image containing an object. The object image to be processed refers to the object image from which features are to be extracted.

[0293] The training process of the target object representation model includes the following steps: First, a first training image set is acquired and input into the initial object representation model. The first training image set includes multiple training image groups, each containing multiple comparative training images. Objects in each training image have corresponding object labels and object association labels. Object association features are extracted from the training images to obtain object association features corresponding to each training image. Object features are extracted from the training images to obtain object features corresponding to each training image. Object features and object association features corresponding to the same training image are fused to obtain object fusion features corresponding to each training image. Based on the object association features and object association labels corresponding to the same training image, object association classification loss is obtained. Based on the object features and object labels corresponding to the same training image, object classification loss is obtained. Based on the object fusion features corresponding to each object in the same training image group, fusion loss is obtained. Based on the object association classification loss, object classification loss, and fusion loss, the model parameters of the initial object representation model are adjusted to obtain an intermediate object representation model. The intermediate object representation model is used as the initial object representation model. The process of acquiring the first training image set and inputting it into the initial object representation model is repeated until the first convergence condition is met, resulting in the target object representation model.

[0294] It is understood that the specific training process for obtaining the target object representation model can be referred to the contents of the various embodiments of the aforementioned object representation model training method, and will not be repeated here.

[0295] Specifically, a target object representation model can be pre-trained. The computer device can acquire the image of the object to be processed locally or from other devices, and then input the image of the object to be processed into the target object representation model to obtain the object fusion features corresponding to the image of the object to be processed.

[0296] Object fusion features corresponding to the image of the object to be processed are used to determine images that match the image of the object to be processed. Images that match the image of the object to be processed are equivalent to similar or duplicate images of the image of the object to be processed. Object fusion features can be applied to various image-related tasks, such as image retrieval tasks, video retrieval tasks, and video deduplication tasks.

[0297] In image retrieval tasks, an image retrieval library can be built based on images and their corresponding object fusion features. When building the library, candidate images are used as object images to be processed. Object fusion features corresponding to these object images are extracted using a target object representation model. The library includes object fusion features corresponding to multiple candidate images. During image retrieval, the image to be retrieved is used as the object image to be processed. Based on the object fusion features of the image to be retrieved, images matching the image to be retrieved are obtained from the library. The feature distances between the object fusion features of the image to be retrieved and the object fusion features of each candidate image are calculated. At least one candidate image with the smallest feature distance is selected as the image matching the image to be retrieved.

[0298] Similar to image retrieval tasks, video retrieval tasks treat video frames as images of objects to be processed, and extract object fusion features corresponding to the video frames using a target object representation model. These object fusion features can be used to build video retrieval libraries or for video retrieval itself.

[0299] In the video deduplication task, video frames in the video are treated as images of objects to be processed. The object fusion features corresponding to the video frames are extracted through the target object representation model. Based on the object fusion features corresponding to the video frames, the video duplication degree between different videos is calculated, and the duplicate videos are determined based on the video duplication degree.

[0300] The object representation method described above extracts object features and object association features from images within the model. It then fuses these features from training images to obtain object fusion features. These fusion features contain feature information across different dimensions, enabling better representation of objects in the image. Furthermore, it eliminates the need to train a separate model for each feature or multiple models. During model training, object features, object association features, and object fusion features corresponding to the same training image are extracted from a single model. Object association classification loss is calculated based on the object association features and object association labels corresponding to the same training image. Object classification loss is calculated based on the object features and object labels corresponding to the same training image. Fusion loss is calculated based on the fusion features of each object corresponding to the same group of training images. Joint training is then performed using the object association classification loss, object classification loss, and fusion loss, ultimately resulting in a model that is accurate and reliable in feature extraction across various features. The target object representation model obtained after model training can extract accurate and reliable object fusion features corresponding to the input image. These object fusion features better represent objects in the image and also help improve the accuracy of subsequent data processing, enhancing image matching accuracy when determining images that match the images of the objects to be processed.

[0301] In one embodiment, the object representation method further includes:

[0302] Obtain the video frame sequence corresponding to the video to be processed; perform object detection on the video frames in the video frame sequence to obtain object subgraphs in the video frames, input the object subgraphs into the target object representation model to obtain the object fusion features corresponding to the object subgraphs; based on each object subgraph and the corresponding object fusion features in the adjacent video frames of the video frame sequence to be processed, obtain the reference object trajectory corresponding to the video to be processed; the reference object trajectory includes the object fusion features of the corresponding object subgraphs; based on each reference object trajectory corresponding to the video to be processed and each candidate object trajectory corresponding to the candidate videos in the video library, query the video library for candidate videos similar to the video to be processed to obtain the query results.

[0303] The video to be processed refers to the video that corresponds to a similar or duplicate video in the query. The video frame sequence to be processed includes multiple video frames from the video to be processed. The video frame sequence to be processed can be obtained by sampling the video to be processed.

[0304] The video library includes multiple candidate videos and corresponding candidate object trajectories for each video. A candidate object trajectory is the trajectory of an object within a candidate video. A reference object trajectory is the trajectory of an object within the video to be processed. It can be understood that a video can have at least one corresponding object trajectory.

[0305] Specifically, when querying similar videos to the video to be processed, the video to be processed can be matched with candidate videos in the video library based on the object trajectory corresponding to the video, and the successfully matched candidate videos can be used as candidate videos similar to the video to be processed.

[0306] Similar to the method of establishing training object trajectories, the computer device performs object detection on video frames in the video frame sequence to be processed, obtaining object subgraphs in the video frames. These object subgraphs are then input into the target object representation model to obtain object fusion features corresponding to the object subgraphs. Based on each object subgraph and its corresponding object fusion feature in adjacent video frames of the video frame sequence to be processed, a reference object trajectory corresponding to the video frame to be processed is established. Specifically, based on each object subgraph and its corresponding object fusion feature in adjacent video frames of the video frame sequence to be processed, object matching is performed on adjacent video frames to obtain object matching results for each group of adjacent video frames in the video frame sequence to be processed. Trajectory aggregation is then performed based on the object matching results of each group of adjacent video frames in the video frame sequence to be processed, resulting in the reference object trajectory corresponding to the video frame sequence to be processed. The reference object trajectory includes the object fusion features of each object subgraph in the corresponding target object subgraph sequence.

[0307] During the query, the computer device matches the video to be processed with the trajectory of each reference object corresponding to the video to be processed and the trajectory of each candidate object corresponding to each candidate video in the video library to find candidate videos similar to the video to be processed. Based on the trajectory of each reference object corresponding to the video to be processed and the trajectory of each candidate object corresponding to each candidate video in the video library, the video repetition degree between the video to be processed and the candidate videos is calculated. Based on the video repetition degree, it is determined whether the video to be processed and the candidate videos match. The candidate videos that match successfully are considered as candidate videos similar to the video to be processed. If there are candidate videos similar to the video to be processed in the video library, the query result is a successful query; if there are no candidate videos similar to the video to be processed in the video library, the query result is a failed query.

[0308] In the above embodiments, based on the trajectories of each reference object corresponding to the video to be processed and the trajectories of each candidate object corresponding to the candidate videos in the video library, a more accurate query result can be obtained by querying the candidate videos similar to the video to be processed from the video library.

[0309] In one embodiment, based on the trajectories of each reference object corresponding to the video to be processed and the trajectories of each candidate object corresponding to candidate videos in the video library, a query is performed from the video library to obtain candidate videos similar to the video to be processed, including:

[0310] Based on the feature distance between the fusion features of each object corresponding to the current reference object trajectory and the fusion features of each object corresponding to the current candidate object trajectory of the current candidate video, the number of feature similarities between the current reference object trajectory and the current candidate object trajectory is determined. Based on the number of feature similarities, the trajectory similarity between the current reference object trajectory and the current candidate object trajectory is determined, and the current candidate object trajectory and the current reference object trajectory with a trajectory similarity greater than a preset similarity are considered similar trajectories. Based on the number of similar trajectories between the video to be processed and the current candidate video, the video duplication between the video to be processed and the current candidate video is determined. When the video duplication is greater than a preset duplication, the query result is determined to be a successful query, and the current candidate video is considered a similar video to the video to be processed. When the video duplication is less than or equal to the preset duplication, the query result is determined to be a failed query.

[0311] Here, "current reference object trajectory" refers to the trajectory of the reference object currently being processed, and can be any object trajectory among various reference object trajectories. "Current candidate video" refers to the candidate video currently being processed, and can be any video among various candidate videos.

[0312] Specifically, during the query, the number of trajectory similarities between the video to be processed and the candidate video is first determined based on the trajectory of the reference object and the trajectory of the candidate object. Then, the video duplication rate between the video to be processed and the candidate video is determined based on the number of trajectory similarities. Finally, the video duplication rate is used to determine whether the video to be processed and the candidate video match.

[0313] When determining the number of trajectory similarities, the computer device selects an arbitrary video from each candidate video as the current candidate video and an arbitrary reference object trajectory from each reference object trajectory as the current reference object trajectory. The computer device calculates the feature distances between the fusion features of each object corresponding to the current reference object trajectory and the fusion features of each object corresponding to the current candidate object trajectory in the current candidate video, obtaining multiple feature distances. Based on these feature distances, the number of feature similarities between the current reference object trajectory and the current candidate object trajectory is determined. For example, two object fusion features whose feature distance is less than a preset distance and are located in the current reference object trajectory and the current candidate object trajectory respectively can be considered as similar features. The number of similar features between the current reference object trajectory and the current candidate object trajectory is counted to obtain the number of feature similarities. Furthermore, the computer device determines the trajectory similarity between the current reference object trajectory and the current candidate object trajectory based on the number of feature similarities. Current candidate object trajectories and current reference trajectories with trajectory similarity greater than a preset similarity are considered as similar trajectories, and the number of similar features between the current reference object trajectory and the current candidate object trajectory is counted to obtain the number of trajectory similarities.

[0314] When determining video repetition, the repetition rate is based on the number of similar trajectories between the video to be processed and the current candidate video. For example, the ratio of the number of similar trajectories to the number of object trajectories in either the video to be processed or the current candidate video can be used as the video repetition rate. The higher the video repetition rate between two videos, the more similar the two videos are.

[0315] When determining the query results, if the video repetition rate is greater than the preset repetition rate, the query result is considered successful, and the current candidate video is considered a similar video to the video to be processed. If the video repetition rate is less than or equal to the preset repetition rate, the query result is considered unsuccessful, and there are no candidate videos similar to the video to be processed in the video library.

[0316] In one embodiment, if the number of similar object fusion features in two trajectories is greater than or equal to a preset proportion of the number of the smallest trajectory features in the two trajectories, then the two trajectories are determined to be similar or identical trajectories. For example, if the number of similar object fusion features in two trajectories is greater than or equal to 1 / 3 of the number of the smallest trajectory features in the two trajectories, then the two trajectories are determined to be similar or identical trajectories. The number of trajectory features refers to the total number of object fusion features corresponding to the trajectory. If the feature distance between two object fusion features is less than a preset distance, then the two object fusion features are determined to be similar object fusion features. For example, if the Euclidean distance between two object fusion features is less than 0.2, then the two object fusion features are determined to be similar object fusion features.

[0317] In one embodiment, video repetition = number of similar trajectories in two videos / minimum number of trajectories in two videos.

[0318] In the above embodiments, the number of similar features between two videos is first determined, then similar trajectories are determined based on the number of similar features, then the number of similar trajectories between the two videos is determined, and finally the video repetition rate between the two videos is determined based on the number of similar trajectories. This ensures the accuracy of the video repetition rate. Determining whether two videos are similar based on their video repetition rate, and then determining the query results, can improve the accuracy of the query results.

[0319] In one embodiment, the object representation method further includes:

[0320] If the video to be processed is a video to be uploaded, and the query result is successful, the upload suggestion for the video to be processed is to prohibit upload; if the query result is unsuccessful, the upload suggestion for the video to be processed is to allow upload.

[0321] If the video to be processed is an already uploaded video, and the query result is a successful query, the target video is determined from the video to be processed and the candidate videos similar to the video to be processed. The processing suggestion for the target video is to notify it to be taken down. If the query result is a failed query, the processing suggestion for the video to be processed and the candidate videos is to not process them.

[0322] Specifically, the video library includes multiple uploaded videos. In a deduplication scenario, the video to be uploaded is treated as a video to be processed, and an upload suggestion is determined based on the query results corresponding to the video to be uploaded. If the query result is successful, it indicates that the video to be uploaded is a duplicate of an uploaded video in the video library. To avoid duplication, the processing suggestion for the video to be processed can be determined as prohibiting upload. If the query result is unsuccessful, it indicates that the video to be uploaded is not a duplicate of an uploaded video in the video library. In this case, the processing suggestion for the video to be processed can be determined as allowing upload.

[0323] In detection scenarios, the system can periodically or irregularly check for duplicate videos among uploaded videos from different users, and determine the recommended processing for each video based on the query results. If the query is successful, it indicates the existence of duplicate uploaded videos. If these duplicate videos belong to different users, one or more videos can be selected as target videos, and the recommended processing for each target video is to notify it to be removed from the platform. Target videos among the duplicate uploaded videos can be videos uploaded later or videos that were not registered. If the query fails, it indicates the absence of duplicate uploaded videos, and in this case, the recommended processing for both the video to be processed and the candidate videos is to leave them alone.

[0324] It is understandable that even if duplicate uploaded videos belong to the same user, any video can be selected from the duplicate uploaded videos as the target video, and the corresponding processing suggestion for the target video is to notify it to be taken down.

[0325] In the above embodiments, different processing suggestions are set for different videos to achieve targeted processing. If the video to be processed is a video yet to be uploaded, and a similar video is found, uploading is prohibited. If the video to be processed is an already uploaded video, and a similar video is found, the target video is further identified, and the target video is notified to be taken down.

[0326] In one embodiment, the object representation method further includes:

[0327] Obtain the object to be retrieved; retrieve the target object trajectory corresponding to the object to be retrieved from the object trajectory library; the object trajectory library includes candidate object trajectories corresponding to multiple candidate objects, which are mined from videos containing candidate objects based on the target object representation model; obtain the video segment corresponding to the target object trajectory in the video to which it belongs; sort the video segments corresponding to each target object trajectory based on the video popularity of the video to which the target object trajectory belongs, and obtain the retrieval results corresponding to the object to be retrieved.

[0328] The object to be searched is used as a search keyword to retrieve relevant videos containing that object. The object trajectory library includes candidate object trajectories for multiple candidate objects, which are mined from videos containing the candidate objects based on the target object representation model. It can be understood that the method for establishing candidate object trajectories is the same as the method for establishing reference object trajectories.

[0329] Specifically, in video retrieval based on objects as search keywords, multiple target object trajectories corresponding to the object to be searched can be obtained from the object trajectory library, the video segments corresponding to the target object trajectories in the videos to which they belong can be obtained, and the search results corresponding to the object to be searched can be obtained based on the video segments corresponding to each target object trajectory.

[0330] Furthermore, when generating search results, the video segments corresponding to each target object's trajectory can be sorted from highest to lowest popularity based on the video popularity of the corresponding video. The ordered video segments are then used as the search results for the target object. The search results are then returned to the terminal of the retrieval party corresponding to the target object, where the retrieval party can display the results.

[0331] In the above embodiments, the target object trajectory corresponding to the object to be searched is obtained from the object trajectory library. Based on the video popularity of the video to which the target object trajectory belongs, the video segments corresponding to each target object trajectory in the video are sorted to obtain the search results corresponding to the object to be searched, which can ensure the effectiveness and reliability of the search results.

[0332] In one specific embodiment, the object representation model training method and object representation method of this application can be applied to game video deduplication scenarios.

[0333] For MOBA (Multiplayer Online Battle Arena) game videos, deduplication is challenging due to the rapid hero changes between frames and the difficulty in representing heroes relative to the map. Since the maps for the same game are relatively fixed, the main difference lies in the performance of the heroes on both sides. Therefore, in deduplication, effectively representing the rapidly changing heroes who occupy a small area on the screen will lead to higher accuracy.

[0334] This application's method leverages rich training data from rapidly changing video character trajectories and enhances hero feature representation through image-specific information during model training. Specifically, it effectively designs auxiliary tasks on the metric learning network to focus on health bar color and hero character information, thereby establishing key feature information for hero deduplication. Simultaneously, through random dropout loss based on classification prototype matching and multi-stage learning, it improves the effectiveness of metric learning in multi-auxiliary task learning from aspects such as network structure, loss function calculation, and model learning, thus obtaining more accurate hero joint embeddings (i.e., hero fusion features, hero joint features). This application's method can be used in game video deduplication based on hero representations, thereby optimizing the inaccuracy of conventional video deduplication for game videos.

[0335] refer to Figure 11 The hero representation model includes a health bar classification branch (i.e., an object association classification network), a hero classification branch (i.e., an object classification network), and a feature fusion branch (i.e., a feature fusion network). The health bar classification branch outputs the health bar features corresponding to the input image and is used for health bar classification. The hero classification branch outputs the hero features corresponding to the input image and is used for hero classification. The feature fusion branch outputs the joint hero features corresponding to the input image.

[0336] The method described in this application includes the following parts:

[0337] I. Exploring the Trajectory of Heroes

[0338] Acquire game video and extract frames from it to obtain a video frame sequence. (Reference) Figure 12 The system employs a hero detection model to identify heroes in video frames within a video frame sequence, generating hero sub-images. Then, it uses the current-stage model to extract joint hero features corresponding to each hero sub-image, and finally generates hero trajectories based on a trajectory strategy.

[0339] Since the health bar of a game hero is generally larger than the hero's body part, other interfering characters, such as minions or other nearby heroes, can easily appear below the health bar. This application uses hero combination features to represent heroes, which can improve the accuracy of the representation.

[0340] II. Establishing Triples

[0341] Two images in adjacent frames of the same hero trajectory are formed into positive sample pairs. Ten pairs of positive samples are randomly selected from each hero trajectory to generate a similar sample pair dataset. The hard negative samples corresponding to each positive sample are mined from the similar sample pair dataset. Image triples are formed based on the positive sample pairs and the corresponding hard negative samples.

[0342] III. Training Model

[0343] Considering that the convergence efficiency of different branches and the degree of fit required by the model for these branches are different, it is necessary to adjust the different tasks in stages and train the model in stages.

[0344] During model training, for all trajectories (N in total), 10 pairs of images are randomly selected from each trajectory to form 10 positive sample pairs, resulting in a total of 10N positive sample pairs. These sample pairs are grouped into batches of 10N / bs. The model updates its parameters for each batch, and each iteration (one epoch) is completed after training 10N / bs of batches.

[0345] 1. First stage of model training

[0346] For example Figure 7 The basic representation model shown is trained. After training, the FCO in the basic representation model is directly used for initialization. Figure 11 Fully connected layer 1 (FC1) and fully connected layer 3 (FC3) in the basic representation model are used for initialization. Figure 11 The base layer is defined as follows. Other layers can be initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0347] 2. Second stage training of the model

[0348] This stage primarily trains the two auxiliary branches of the hero representation model. However, since the objective task is metric learning (i.e., outputting joint hero features), and metric learning converges more slowly than classification tasks, metric learning also participates in the learning process at this stage. The learning rates for the health bar classification branch and the hero classification branch are set to 0.01, while the metric learning rate is 0.001, thus ensuring the model effectively fits the classification auxiliary task.

[0349] The health bar classification branch uses the cross-entropy loss function to calculate the health bar classification loss. The hero classification branch uses random dropout loss based on classification prototype matching to calculate the hero classification loss, and the feature fusion branch uses triplet loss to calculate the fusion loss. Random dropout loss based on classification prototype matching is used to avoid overfitting in classification tasks, which can lead to poor metric learning performance. For example, different sides of the same hero should have different feature representations, but traditional classification loss learning can result in heroes from different sides having very similar features. Deduplication tasks require distinguishing different sides of a hero; if video A shows the hero's left side and video B shows the hero's front, then videos A and B are not duplicates, and this is reflected in the different hero features of the heroes in videos A and B. However, under the influence of classification tasks, since heroes from different sides belong to the same hero category during classification (i.e., learning with the same category's supervised information), the learned features are sufficiently similar. Therefore, without anti-overfitting design, it is impossible to distinguish different sides of a hero.

[0350] 3. Model training in the third stage

[0351] In this phase, only the metric learning branch is trained; the classification module of the auxiliary branch is no longer updated. The learning rate is set to 0.001. The training method can be found in [reference needed]. Figure 9 .

[0352] IV. Application Model

[0353] 1. Establish a hero trajectory database

[0354] First, frames are extracted from the game video, and a hero detection model is used to detect characters in each frame. After obtaining the hero bounding box for each frame, the corresponding hero joint features are extracted using a trained hero representation model. The hero bounding box information is recorded, specifically including the frame number, character coordinates x1, y1, x2, y2, and hero joint features.

[0355] Hero trajectories are generated by matching hero bounding boxes between adjacent frames. For each hero trajectory, the average joint feature of the hero joint features in the trajectory is calculated. The top 20% of samples in the trajectory that are farthest from the average joint feature are removed, and only 80% of the hero joint features in the trajectory are retained as the record of that trajectory. This step is to remove poorly performing character images in the trajectory, such as some characters that move too fast and have only artifacts, which are not conducive to retrieval and need to be removed.

[0356] Save 80% of the hero joint features for each trajectory to the library. Record the total number of hero trajectories for each video.

[0357] 2. Retrieval and Recall: Retrieve and sort videos from the inventory.

[0358] When performing a video query based on a specific game video, the hero trajectories of the video are first obtained, and then clustered and cleaned to obtain the final hero trajectories. Based on the hero trajectories corresponding to the game video, videos with the same trajectories and high video repetition are retrieved from the inventory as query results. The same trajectories must meet the following condition: the number of similar hero joint features in the two trajectories must be at least 1 / 3 of the number of hero joint features in the smallest trajectory between the two trajectories. The number of identical hero trajectories in two videos / the number of smallest trajectories in the two videos = video repetition rate.

[0359] In the deduplication application, if a user wants to upload a video, and the method in this application finds that the video is duplicated in the inventory, a warning message is returned, and the upload is not allowed.

[0360] If a video in an app is found to be duplicated in the registered inventory, an alert is issued to the app, and it is ordered to be removed from the app store.

[0361] In the recommendation of game hero gameplay strategy videos, during the database entry phase, the hero's name and the start and end frame numbers of the hero's trajectory in the video need to be recorded simultaneously. For example, the hero's name is recorded based on the output of the hero classification branch of the model based on the joint features of each hero in the trajectory. During the retrieval phase, the user inputs the name of the hero they want to display. The database finds multiple hero trajectories for the corresponding hero based on the hero name records from the database entry phase. Then, based on these trajectories, the corresponding videos are located, and the video popularity is obtained. The videos are sorted according to popularity (the more views, the higher the ranking). The video segment in which the trajectory appears in the video is extracted (the video segment is located based on the start and end frame numbers of the trajectory), and the video segment is returned to the user for viewing.

[0362] In the above embodiments, the method of this application has the following beneficial effects:

[0363] 1. Improve detection performance by leveraging video hero trajectories and obtain richer and more accurate hero images through clustering trajectory cleaning, thereby providing diverse game video hero performance images for model training.

[0364] 2. Add an auxiliary health bar classification branch before the hero representation model outputs the representation results. This allows the health bar features to be embedded into the hero features, so that the final hero joint features carry information that can distinguish between the two sides of the game, avoiding the problem of deduplication and mismatch when both sides of the game use the same hero.

[0365] 3. A multi-task, multi-stage model learning method was designed for the hero representation model to improve the model training quality. A hero classification loss based on prototype matching and random dropout was specifically designed to reduce the problem of the model's excessive fitting of the classification loss to the metric learning representation effect.

[0366] It is understandable that, besides being applied to deduplication of game videos, the method of this application can also be applied to deduplication of other types of videos. This method can also be applied to tasks involving fine-grained recognition of animal images. For example, the animal representation model includes an animal part classification branch, an animal classification branch, and a feature fusion branch. The animal classification branch mainly considers the effect of the overall image information of the animal image on animal recognition. The animal part classification branch can further consider the effect of image information of certain parts of the animal on animal recognition. The feature fusion branch can ultimately obtain animal fusion features that accurately represent the animal, and these animal fusion features help improve the effect of fine-grained animal recognition.

[0367] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0368] Based on the same inventive concept, embodiments of this application also provide an object representation model training apparatus for implementing the object representation model training method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more object representation model training apparatus embodiments provided below can be found in the limitations of the object representation model training method described above, and will not be repeated here. Embodiments of this application also provide an object representation apparatus for implementing the object representation method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more object representation apparatus embodiments provided below can be found in the limitations of the object representation method described above, and will not be repeated here.

[0369] In one embodiment, such as Figure 13 As shown, an object representation model training device is provided, including: a training image set acquisition module 1302, a feature determination module 1304, a loss determination module 1306, and a model adjustment module 1308, wherein:

[0370] The training image set acquisition module 1302 is used to acquire a first training image set and input the first training image set into the initial object representation model. The first training image set includes multiple training image groups, and each training image group includes positive sample image pairs, negative sample images corresponding to the positive sample image pairs, and object labels and object association labels corresponding to objects in each training image.

[0371] The feature determination module 1304 is used to extract object association features from training images to obtain object association features corresponding to each training image, extract object features from training images to obtain object features corresponding to each training image, and fuse object features and object association features corresponding to the same training image to obtain object fusion features corresponding to each training image.

[0372] The loss determination module 1306 is used to obtain object association classification loss based on object association features and object association labels corresponding to the same training image, object classification loss based on object features and object labels corresponding to the same training image, and fusion loss based on the fusion features of each object corresponding to the same group of training images.

[0373] The model adjustment module 1308 is used to adjust the model parameters of the initial object representation model based on object association classification loss, object classification loss and fusion loss to obtain an intermediate object representation model. The intermediate object representation model is used as the initial object representation model. The steps of obtaining the first training image set and inputting the first training image set into the initial object representation model are executed until the first convergence condition is met to obtain the target object representation model. The target object representation model is used to extract the object fusion features corresponding to the input image.

[0374] The aforementioned object representation model training device, by fusing different features to obtain object fusion features containing feature information of different dimensions, can better represent objects in images. Furthermore, it eliminates the need to train a separate model for each feature, or multiple models. Object features, object association features, and object fusion features corresponding to the same training image are extracted from a single model. Joint training is then performed based on object association classification loss, object classification loss, and fusion loss, ultimately resulting in a model that is accurate and reliable in feature extraction for all types of features. The target object representation model obtained after model training can extract accurate and reliable object fusion features corresponding to the input image. These object fusion features can better represent objects in images and also help improve the accuracy of subsequent data processing.

[0375] In one embodiment, the training image set acquisition module 1302 is further configured to:

[0376] Obtain target object sub-graph sequences corresponding to multiple training object trajectories; the training object trajectories are mined from training videos containing objects, and the object sub-graphs in the target object sub-graph sequences are obtained by object detection on the corresponding video frames in the training videos to which the training object trajectories belong; sample each target object sub-graph sequence to obtain multiple positive sample image pairs; obtain the negative sample image corresponding to the current positive sample image pair from other positive sample image pairs; obtain the object label and object association label corresponding to the object in the training image in the positive sample image pair, and form a training image group based on the positive sample image pair and the corresponding negative sample image; obtain the first training image set based on each training image group.

[0377] In one embodiment, the training image set acquisition module 1302 is further configured to:

[0378] Multiple training video frame sequences are acquired; video frames in the same training video frame sequence belong to the same training video; object detection is performed on the video frames in the training video frame sequence to obtain object subgraphs in the video frames, and the object subgraphs are input into the initial object representation model to obtain the object fusion features corresponding to the object subgraphs; based on each object subgraph and the corresponding object fusion features in adjacent video frames, object matching is performed on adjacent video frames in the same training video frame sequence to obtain the object matching results corresponding to each group of adjacent video frames; trajectory aggregation is performed based on the object matching results of each group of adjacent video frames in the same training video frame sequence to obtain a sequence of target object subgraphs corresponding to multiple training object trajectories.

[0379] In one embodiment, the training image set acquisition module 1302 is further configured to:

[0380] The process begins by determining the target video frame from the current video frame sequence; then, each object subgraph corresponding to the target video frame is sequentially used as the target object subgraph; based on the feature distance between the object fusion features corresponding to the target object subgraph and the object fusion features corresponding to each object subgraph in the target backward video frame, candidate object subgraphs corresponding to the target object subgraph are determined from each object subgraph corresponding to the target backward video frame; candidate object subgraphs whose positional distance to the target object subgraph is less than a distance threshold are used as the backward matching object subgraphs corresponding to the target object subgraph; the target backward video frame is the adjacent backward video frame of the target video frame; based on each target object subgraph and its corresponding backward matching object subgraph, the object matching result between the target video frame and the target backward video frame is obtained; the next video frame is obtained from the current video frame sequence as the target video frame, and the process of sequentially using each object subgraph corresponding to the target video frame as the target object subgraph is repeated until the object matching results corresponding to each group of adjacent video frames in the current video frame sequence are obtained.

[0381] In one embodiment, the object matching result includes a pair of object subgraphs consisting of a target object subgraph and a corresponding backward matching object subgraph, and a trajectory identifier corresponding to the object subgraph pair. Different object subgraph pairs with overlapping object subgraphs may have the same trajectory identifier. The training image set acquisition module 1302 is further configured to:

[0382] From the object matching results corresponding to each group of adjacent video frames in the current video frame sequence, obtain the object subgraph pairs corresponding to the current trajectory identifier to form the current object subgraph pair set corresponding to the current trajectory identifier; sort each object subgraph in the current object subgraph pair set according to the video frame order of the corresponding video frames to obtain the current initial object subgraph sequence corresponding to the current trajectory identifier; statistically analyze the fusion features of each object corresponding to the current initial object subgraph sequence to obtain the current object center feature; based on the feature distance between each object fusion feature corresponding to the current initial object subgraph sequence and the current object center feature, delete a first preset number of object subgraphs from the current initial object subgraph sequence according to the feature distance from largest to smallest to obtain the target object subgraph sequence corresponding to the current trajectory identifier.

[0383] In one embodiment, the distance threshold is dynamically changed. The distance threshold is obtained by statistically analyzing the positional distance between the updated target object sub-graph and the corresponding backward matching object sub-graph after completing object matching of a second preset number of adjacent video frames.

[0384] In one embodiment, the training image set acquisition module 1302 is further configured to:

[0385] The training image is obtained from the current positive sample image pair as the target positive sample image, and the training image is obtained from each other positive sample image pair as the candidate image. Based on the image distance between the target positive sample image and each candidate image, the candidate images are sorted in ascending order of image distance to obtain a candidate image sequence. A third preset number of candidate images are deleted from the candidate image sequence in ascending order of image distance to obtain an updated image sequence. A fourth preset number of candidate images are obtained from the updated image sequence in ascending order of image distance to serve as the negative sample images corresponding to the current positive sample image pair.

[0386] In one embodiment, when the current feature is an object association feature, the current label is an object association label, and the current classification loss is an object association classification loss; when the current feature is an object feature, the current label is an object label, and the current classification loss is an object classification loss. The loss determination module 1306 is further configured to:

[0387] Obtain the prototype object feature sets corresponding to various current labels. Each prototype object feature set includes multiple prototype object features corresponding to the same current label. The prototype object features are obtained from the current feature library by selecting the current feature that is closest to the cluster center feature corresponding to the current label. The cluster center feature is obtained by clustering the current features corresponding to the same current label in the current feature library. From each prototype object feature set, select the prototype object feature with the smallest feature distance to the current feature corresponding to the current training image as the reference object feature corresponding to the current training image. Based on the difference between the current feature and the reference object feature corresponding to the same training image, obtain the current classification loss.

[0388] In one embodiment, the loss determination module 1306 is further configured to:

[0389] Within the initial model iteration rounds, every fifth preset number of batches trained, the prototype object feature sets corresponding to various current labels are determined based on the current feature library; within the backward model iteration rounds within the initial model iteration rounds, every sixth preset number of batches trained, the prototype object feature sets corresponding to various current labels are determined based on the current feature library; the sixth preset number is greater than the fifth preset number; after each batch is trained, the obtained current features are stored in the current feature library, and after each prototype object feature set is determined, the current feature library is cleared.

[0390] In one embodiment, the current feature and the reference object feature are feature vectors of a preset length. The loss determination module 1306 is further configured to:

[0391] Obtain a random feature vector of a preset length, and retain the vector positions corresponding to data whose values ​​are greater than or equal to a preset threshold in the random feature vector. Based on the data differences between the current feature and the reference object feature at the same retained position in the same training image, obtain the current classification loss.

[0392] In one embodiment, when the current feature is an object association feature, the current label is an object association label, and the current classification loss is an object association classification loss; when the current feature is an object feature, the current label is an object label, and the current classification loss is an object classification loss. The loss determination module 1306 is further configured to:

[0393] The predicted probability of the training image is obtained based on the current features corresponding to the training image; the current classification loss is obtained based on the predicted probability of the same training image and the current label.

[0394] In one embodiment, the loss determination module 1306 is further configured to:

[0395] A first difference is obtained based on the difference between the fusion features of each object corresponding to the positive sample image pair in the same training image group; a reference positive sample image is determined from the positive sample image pair; a second difference is obtained based on the difference between the fusion features of the objects corresponding to the reference positive sample image and the negative sample image in the same training image group; and a fusion loss is obtained based on the distance between the first difference and the second difference corresponding to the same training image group and a preset distance.

[0396] In one embodiment, the initial object representation model includes an object association classification network, an object classification network, and a feature fusion network. The object association classification network is used to obtain object association features, the object classification network is used to obtain object features, and the feature fusion network is used to obtain object fusion features. The model adjustment module 1308 is also used for:

[0397] The network parameters of the object association classification network are adjusted according to the first learning rate based on the object association classification loss, the network parameters of the object classification network are adjusted according to the second learning rate based on the object classification loss, and the network parameters of the object association classification network, object classification network, and feature fusion network are adjusted according to the third learning rate based on the fusion loss to obtain an intermediate object representation model; wherein the first learning rate and the second learning rate are both greater than the third learning rate.

[0398] In one embodiment, the object representation model training apparatus further includes:

[0399] The forward training module is used to acquire a second training image set and input it into the basic object representation model to be trained, thereby obtaining the initial object features corresponding to each training image in the second training image set. The basic object representation model includes a feature extraction layer and a fully connected layer. Based on the initial object features corresponding to the same group of training images in the second training image set, an initial loss is obtained. The model parameters of the basic object representation model are adjusted based on the initial loss until the second convergence condition is met, resulting in a trained basic object representation model. Based on the feature extraction layer in the trained basic object representation model, the feature extraction layer in the candidate object representation model is initialized. Based on the fully connected layer in the trained basic object representation model, the fully connected layer in the candidate object representation model is initialized, resulting in an initial object representation model.

[0400] In one embodiment, the forward training module is further configured to initialize the shared feature extraction layer between the object association classification network and the object classification network in the candidate object representation model based on the feature extraction layer in the trained basic object representation model, and initialize the fully connected layer of the object association classification network and the fully connected layer of the object classification network in the candidate object representation model based on the fully connected layer in the trained basic object representation model, thereby obtaining an initial object representation model.

[0401] In one embodiment, the object representation model training apparatus further includes:

[0402] The backward training module is used to acquire the third training image set, input the third training image set into the target object representation model, and obtain the object fusion features corresponding to each training image in the third training image set; based on the object fusion features corresponding to each group of training images in the third training image set, the update loss is obtained; based on the update loss, the model parameters of the target object representation model are adjusted until the third convergence condition is met, and the updated object representation model is obtained.

[0403] In one embodiment, such as Figure 14 As shown, an object representation device is provided, including: an image acquisition module 1402 and an image processing module 1404, wherein:

[0404] Image acquisition module 1402 is used to acquire the image of the object to be processed.

[0405] The image processing module 1404 is used to input the image of the object to be processed into the target object representation model to obtain the object fusion features corresponding to the image of the object to be processed; the object fusion features corresponding to the image of the object to be processed are used to determine the image that matches the image of the object to be processed.

[0406] The training process for the target object representation model can be found in the description of the object representation model training method mentioned above.

[0407] The object representation device described above, through model training, can extract accurate and reliable object fusion features corresponding to the input image. The object fusion features can better represent the objects in the image and also help improve the accuracy of subsequent data processing, thereby improving the accuracy of image matching when determining the image that matches the image of the object to be processed.

[0408] In one embodiment, the object representation device further includes:

[0409] The video processing module is used to acquire the video frame sequence corresponding to the video to be processed; perform object detection on the video frames in the video frame sequence to obtain object subgraphs in the video frames, input the object subgraphs into the target object representation model, and obtain the object fusion features corresponding to the object subgraphs; based on each object subgraph and the corresponding object fusion features in the adjacent video frames of the video frame sequence to be processed, obtain the reference object trajectory corresponding to the video to be processed; the reference object trajectory includes the object fusion features of the corresponding object subgraphs; based on each reference object trajectory corresponding to the video to be processed and each candidate object trajectory corresponding to candidate videos in the video library, query the video library for candidate videos similar to the video to be processed, and obtain the query results.

[0410] In one embodiment, the video processing module is further configured to:

[0411] Based on the feature distance between the fusion features of each object corresponding to the current reference object trajectory and the fusion features of each object corresponding to the current candidate object trajectory of the current candidate video, the number of feature similarities between the current reference object trajectory and the current candidate object trajectory is determined. Based on the number of feature similarities, the trajectory similarity between the current reference object trajectory and the current candidate object trajectory is determined, and the current candidate object trajectory and the current reference object trajectory with a trajectory similarity greater than a preset similarity are considered similar trajectories. Based on the number of similar trajectories between the video to be processed and the current candidate video, the video duplication between the video to be processed and the current candidate video is determined. When the video duplication is greater than a preset duplication, the query result is determined to be a successful query, and the current candidate video is considered a similar video to the video to be processed. When the video duplication is less than or equal to the preset duplication, the query result is determined to be a failed query.

[0412] In one embodiment, the video processing module is further configured to:

[0413] If the video to be processed is a video to be uploaded, and the query result is successful, the upload suggestion for the video to be processed is to prohibit upload; if the query result is unsuccessful, the upload suggestion for the video to be processed is to allow upload.

[0414] If the video to be processed is an already uploaded video, and the query result is a successful query, the target video is determined from the video to be processed and the candidate videos similar to the video to be processed. The processing suggestion for the target video is to notify it to be taken down. If the query result is a failed query, the processing suggestion for the video to be processed and the candidate videos is to not process them.

[0415] In one embodiment, the object representation device further includes:

[0416] The object retrieval module is used to obtain the object to be retrieved; retrieve the target object trajectory corresponding to the object to be retrieved from the object trajectory library; the object trajectory library includes candidate object trajectories corresponding to multiple candidate objects, which are mined from videos containing candidate objects based on the target object representation model; obtain the video segment corresponding to the target object trajectory in the video to which it belongs; and sort the video segments corresponding to each target object trajectory according to the video popularity of the video to which the target object trajectory belongs, to obtain the retrieval results corresponding to the object to be retrieved.

[0417] Each module in the aforementioned object representation model training device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0418] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 15 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores training image sets, target object representation models, and data for updating object representation models. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an object representation model training method or an object representation method.

[0419] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 16As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an object representation model training method or an object representation method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0420] Those skilled in the art will understand that Figure 15 , 16 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0421] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0422] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0423] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0424] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0425] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0426] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0427] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for training an object representation model, characterized in that, The method includes: A first training image set is obtained and input into an initial object representation model. The first training image set includes multiple training image groups, each training image group including multiple mutually contrasting training images. Each training image contains an object label and an object association label. The object association label is used to characterize the content of the region displayed by the object association region in the image. The object association region refers to the local region in the image that is related to the object. Object association features are extracted from the training images to obtain object association features corresponding to each training image. Object features are extracted from the training images to obtain object features corresponding to each training image. The object features and object association features corresponding to the same training image are fused to obtain object fusion features corresponding to each training image. The object association features are used to characterize the feature information corresponding to the object association region in the image. Based on the object association features and object association labels corresponding to the same training image, we obtain the object association classification loss; based on the object features and object labels corresponding to the same training image, we obtain the object classification loss; based on the fusion features of each object corresponding to the same group of training images, we obtain the fusion loss. Based on the object association classification loss, the object classification loss, and the fusion loss, the model parameters of the initial object representation model are adjusted to obtain an intermediate object representation model. The intermediate object representation model is then used as the initial object representation model. The steps of obtaining the first training image set and inputting the first training image set into the initial object representation model are repeated until the first convergence condition is met to obtain the target object representation model. The target object representation model is used to extract the object fusion features corresponding to the input image.

2. The method according to claim 1, characterized in that, The acquisition of the first training image set includes: Obtain target object subgraph sequences corresponding to multiple training object trajectories; the training object trajectories are mined from training videos containing objects, and the object subgraphs in the target object subgraph sequences are obtained by performing object detection on the corresponding video frames in the training videos to which the training object trajectories belong. Each target object sub-image sequence is sampled separately to obtain multiple positive sample image pairs; Obtain the negative sample image corresponding to the current positive sample image pair from other positive sample image pairs besides the current positive sample image pair; Training image groups are composed of positive sample image pairs and corresponding negative sample images; The first training image set is obtained based on each training image group.

3. The method according to claim 2, characterized in that, The step of obtaining the target object subgraph sequence corresponding to the trajectories of multiple training objects includes: Obtain multiple training video frame sequences; video frames in the same training video frame sequence belong to the same training video. Object detection is performed on video frames in the training video frame sequence to obtain object subgraphs in the video frames. The object subgraphs are then input into the initial object representation model to obtain the object fusion features corresponding to the object subgraphs. Based on the object subgraphs and corresponding object fusion features in adjacent video frames, object matching is performed on adjacent video frames in the same training video frame sequence to obtain the object matching results corresponding to each group of adjacent video frames. Trajectory aggregation is performed based on the object matching results of each group of adjacent video frames in the same training video frame sequence to obtain target object sub-graph sequences corresponding to multiple training object trajectories.

4. The method according to claim 3, characterized in that, The method involves performing object matching on adjacent video frames within the same training video frame sequence based on each object subgraph and its corresponding object fusion features in adjacent video frames, resulting in object matching results for each group of adjacent video frames, including: Determine the target video frame from the current video frame sequence; Each object subgraph corresponding to the target video frame is used as the target object subgraph in turn. Based on the feature distance between the object fusion features corresponding to the target object subgraph and the object fusion features corresponding to each object subgraph in the target backward video frame, candidate object subgraphs corresponding to the target object subgraph are determined from each object subgraph corresponding to the target backward video frame. Candidate object subgraphs whose positional distance to the target object subgraph is less than a distance threshold are taken as the backward matching object subgraphs corresponding to the target object subgraph. The target backward video frame is the adjacent backward video frame of the target video frame. Based on each target object subgraph and the corresponding backward matching object subgraph, the object matching results between the target video frame and the target backward video frame are obtained; The next video frame is obtained from the current video frame sequence as the target video frame. The step of sequentially using each object subgraph corresponding to the target video frame as the target object subgraph is then performed until the object matching results corresponding to each group of adjacent video frames in the current video frame sequence are obtained.

5. The method according to claim 4, characterized in that, The object matching result includes an object subgraph pair consisting of a target object subgraph and a corresponding backward matching object subgraph, and a trajectory identifier corresponding to the object subgraph pair. Different object subgraph pairs with overlapping object subgraphs have the same trajectory identifier. The trajectory aggregation based on the object matching results of each group of adjacent video frames in the same training video frame sequence yields multiple target object sub-graph sequences corresponding to the training object trajectories, including: From the object matching results corresponding to each group of adjacent video frames in the current video frame sequence, obtain the object subgraph pairs corresponding to the current trajectory identifier to form the current object subgraph pair set corresponding to the current trajectory identifier; The current object subgraphs in the set are sorted according to the video frame order of the corresponding video frames to obtain the current initial object subgraph sequence corresponding to the current trajectory identifier; The fusion features of each object corresponding to the current initial object subgraph sequence are statistically analyzed to obtain the current object center features; Based on the feature distance between the fusion features of each object corresponding to the current initial object subgraph sequence and the current object center feature, a first preset number of object subgraphs are deleted from the current initial object subgraph sequence according to the feature distance from largest to smallest, to obtain the target object subgraph sequence corresponding to the current trajectory identifier.

6. The method according to claim 1, characterized in that, When the current feature is an object association feature, the current label is the object association label, and the current classification loss is the object association classification loss; when the current feature is an object feature, the current label is the object label, and the current classification loss is the object classification loss. The calculation process for the current classification loss includes the following steps: Obtain the prototype object feature set corresponding to each current label; the prototype object feature set includes multiple prototype object features corresponding to the same current label, and the prototype object feature is obtained from the current feature library by obtaining the current feature that is closest to the cluster center feature corresponding to the current label. The cluster center feature is obtained by clustering the current features corresponding to the same current label in the current feature library. From each set of prototype object features, the prototype object feature with the smallest feature distance to the current feature corresponding to the current training image is selected as the reference object feature corresponding to the current training image. The current classification loss is obtained based on the difference between the current features and the reference object features corresponding to the same training image.

7. The method according to claim 6, characterized in that, The current feature and the reference object feature are feature vectors of a preset length; The current classification loss is obtained based on the difference between the current features and the reference object features corresponding to the same training image, including: Obtain a random feature vector of the preset length, and retain the vector positions corresponding to data whose values ​​are greater than or equal to a preset threshold in the random feature vector as reserved positions; The current classification loss is obtained based on the data differences between the current features and the reference object features corresponding to the same training image at the same retention position.

8. The method according to claim 1, characterized in that, The initial object representation model includes an object association classification network, an object classification network, and a feature fusion network. The object association classification network is used to obtain object association features, the object classification network is used to obtain object features, and the feature fusion network is used to obtain object fusion features. The step of adjusting the model parameters of the initial object representation model based on the object association classification loss, the object classification loss, and the fusion loss to obtain the intermediate object representation model includes: Based on the object association classification loss, the network parameters of the object association classification network are adjusted according to the first learning rate; based on the object classification loss, the network parameters of the object classification network are adjusted according to the second learning rate; based on the fusion loss, the network parameters of the object association classification network, the object classification network, and the feature fusion network are adjusted according to the third learning rate to obtain an intermediate object representation model. Wherein, both the first learning rate and the second learning rate are greater than the third learning rate.

9. The method according to claim 1, characterized in that, Before acquiring the first training image set, the method further includes: A second training image set is obtained and input into the basic object representation model to be trained to obtain the initial object features corresponding to each training image in the second training image set; the basic object representation model includes a feature extraction layer and a fully connected layer. The initial loss is obtained based on the features of each initial object corresponding to the same group of training images in the second training image set; The model parameters of the basic object representation model are adjusted based on the initial loss until the second convergence condition is met, thus obtaining the trained basic object representation model. Based on the feature extraction layer in the trained basic object representation model, the feature extraction layer in the candidate object representation model is initialized. Based on the fully connected layer in the trained basic object representation model, the fully connected layer in the candidate object representation model is initialized to obtain the initial object representation model.

10. The method according to claim 9, characterized in that, The initial object representation model is obtained by initializing the feature extraction layer in the candidate object representation model based on the feature extraction layer in the trained basic object representation model, and by initializing the fully connected layer in the candidate object representation model based on the fully connected layer in the trained basic object representation model, including: Based on the feature extraction layer in the trained basic object representation model, the feature extraction layer shared between the object association classification network and the object classification network in the candidate object representation model is initialized. Based on the fully connected layer in the trained basic object representation model, the fully connected layer of the object association classification network and the fully connected layer of the object classification network in the candidate object representation model are initialized to obtain the initial object representation model.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: Obtain a third training image set, input the third training image set into the target object representation model, and obtain the object fusion features corresponding to each training image in the third training image set; Based on the fusion features of each object corresponding to the same group of training images in the third training image set, the updated loss is obtained; The model parameters of the target object representation model are adjusted based on the update loss until the third convergence condition is met, thus obtaining the updated object representation model.

12. An object representation method, characterized in that, The method includes: Obtain the image of the object to be processed; The image of the object to be processed is input into the target object representation model to obtain the object fusion feature corresponding to the image of the object to be processed; the object fusion feature corresponding to the image of the object to be processed is used to determine the image that matches the image of the object to be processed. The training process of the target object representation model includes the following steps: A first training image set is obtained and input into an initial object representation model. The first training image set includes multiple training image groups, each training image group including multiple mutually contrasting training images. Each training image contains an object label and an object association label. The object association label is used to characterize the content of the region displayed by the object association region in the image. The object association region refers to the local region in the image that is related to the object. Object association features are extracted from the training images to obtain object association features corresponding to each training image. Object features are extracted from the training images to obtain object features corresponding to each training image. The object features and object association features corresponding to the same training image are fused to obtain object fusion features corresponding to each training image. The object association features are used to characterize the feature information corresponding to the object association region in the image. Based on the object association features and object association labels corresponding to the same training image, we obtain the object association classification loss; based on the object features and object labels corresponding to the same training image, we obtain the object classification loss; based on the fusion features of each object corresponding to the same group of training images, we obtain the fusion loss. Based on the object association classification loss, the object classification loss, and the fusion loss, the model parameters of the initial object representation model are adjusted to obtain an intermediate object representation model. The intermediate object representation model is then used as the initial object representation model. The steps of obtaining the first training image set and inputting the first training image set into the initial object representation model are repeated until the first convergence condition is met, thereby obtaining the target object representation model.

13. The method according to claim 12, characterized in that, The method further includes: Obtain the video frame sequence corresponding to the video to be processed; Object detection is performed on the video frames in the video frame sequence to be processed to obtain object sub-graphs in the video frames. The object sub-graphs are then input into the target object representation model to obtain the object fusion features corresponding to the object sub-graphs. Based on each object subgraph and corresponding object fusion feature in adjacent video frames of the video frame sequence to be processed, a reference object trajectory corresponding to the video to be processed is obtained; the reference object trajectory includes the object fusion feature of the corresponding object subgraph. Based on the trajectories of each reference object corresponding to the video to be processed and the trajectories of each candidate object corresponding to the candidate videos in the video library, a query is performed from the video library to obtain the query results.

14. The method according to claim 13, characterized in that, The step of querying candidate videos similar to the video to be processed from the video library based on the trajectories of each reference object corresponding to the video to be processed and the trajectories of each candidate video corresponding to the candidate videos in the video library, and obtaining the query results, includes: Based on the feature distance between the fusion features of each object corresponding to the current reference object trajectory and the fusion features of each object corresponding to the current candidate object trajectory in the current candidate video, the number of feature similarities between the current reference object trajectory and the current candidate object trajectory is determined. The trajectory similarity between the current reference object trajectory and the current candidate object trajectory is determined based on the number of feature similarities. The current candidate object trajectory and the current reference object trajectory with a trajectory similarity greater than the preset similarity are regarded as similar trajectories. The video duplication rate between the video to be processed and the current candidate video is determined based on the number of similar trajectories between them. When the video repetition rate is greater than the preset repetition rate, the query result is determined to be a successful query, and the current candidate video is taken as a similar video to the video to be processed. When the video repetition is less than or equal to the preset repetition, the query result is determined to be a query failure.

15. The method according to claim 12, characterized in that, The method further includes: Retrieve the object to be searched; The target object trajectory corresponding to the object to be retrieved is obtained from the object trajectory library; the object trajectory library includes candidate object trajectories corresponding to multiple candidate objects, and the candidate object trajectories are mined from videos containing candidate objects based on the target object representation model; Obtain the video segment corresponding to the trajectory of the target object in its respective video; Based on the video popularity of the video to which the target object's trajectory belongs, the video segments corresponding to each target object's trajectory are sorted to obtain the search results for the object to be searched.

16. A training device for an object representation model, characterized in that, The device includes: The training image set acquisition module is used to acquire a first training image set and input the first training image set into the initial object representation model. The first training image set includes multiple training image groups, and each training image group includes multiple training images that are compared with each other. The objects in each training image have corresponding object labels and object association labels. The object association labels are used to characterize the content of the region displayed by the object association region in the image. The object association region refers to the local region in the image that is related to the object. The feature determination module is used to extract object association features from training images to obtain object association features corresponding to each training image, extract object features from training images to obtain object features corresponding to each training image, and fuse object features and object association features corresponding to the same training image to obtain object fusion features corresponding to each training image; the object association features are used to characterize the feature information corresponding to the object association region in the image. The loss determination module is used to obtain object association classification loss based on object association features and object association labels corresponding to the same training image, object classification loss based on object features and object labels corresponding to the same training image, and fusion loss based on the fusion features of each object corresponding to the same group of training images. The model adjustment module is used to adjust the model parameters of the initial object representation model based on the object association classification loss, the object classification loss, and the fusion loss to obtain an intermediate object representation model. The intermediate object representation model is then used as the initial object representation model. The steps of obtaining the first training image set and inputting the first training image set into the initial object representation model are repeated until the first convergence condition is met to obtain the target object representation model. The target object representation model is used to extract the object fusion features corresponding to the input image.

17. The object representation model training device according to claim 16, characterized in that, The training image set acquisition module is also used for: Obtain target object subgraph sequences corresponding to multiple training object trajectories; the training object trajectories are mined from training videos containing objects, and the object subgraphs in the target object subgraph sequences are obtained by performing object detection on the corresponding video frames in the training videos to which the training object trajectories belong. Each target object sub-image sequence is sampled separately to obtain multiple positive sample image pairs; Obtain the negative sample image corresponding to the current positive sample image pair from other positive sample image pairs besides the current positive sample image pair; Training image groups are composed of positive sample image pairs and corresponding negative sample images; The first training image set is obtained based on each training image group.

18. The object representation model training device according to claim 17, characterized in that, The training image set acquisition module is also used for: Obtain multiple training video frame sequences; video frames in the same training video frame sequence belong to the same training video. Object detection is performed on video frames in the training video frame sequence to obtain object subgraphs in the video frames. The object subgraphs are then input into the initial object representation model to obtain the object fusion features corresponding to the object subgraphs. Based on the object subgraphs and corresponding object fusion features in adjacent video frames, object matching is performed on adjacent video frames in the same training video frame sequence to obtain the object matching results corresponding to each group of adjacent video frames. Trajectory aggregation is performed based on the object matching results of each group of adjacent video frames in the same training video frame sequence to obtain target object sub-graph sequences corresponding to multiple training object trajectories.

19. The object representation model training device according to claim 18, characterized in that, The training image set acquisition module is also used for: Determine the target video frame from the current video frame sequence; Each object subgraph corresponding to the target video frame is used as the target object subgraph in turn. Based on the feature distance between the object fusion features corresponding to the target object subgraph and the object fusion features corresponding to each object subgraph in the target backward video frame, candidate object subgraphs corresponding to the target object subgraph are determined from each object subgraph corresponding to the target backward video frame. Candidate object subgraphs whose positional distance to the target object subgraph is less than a distance threshold are taken as the backward matching object subgraphs corresponding to the target object subgraph. The target backward video frame is the adjacent backward video frame of the target video frame. Based on each target object subgraph and the corresponding backward matching object subgraph, the object matching results between the target video frame and the target backward video frame are obtained; The next video frame is obtained from the current video frame sequence as the target video frame. The step of sequentially using each object subgraph corresponding to the target video frame as the target object subgraph is then performed until the object matching results corresponding to each group of adjacent video frames in the current video frame sequence are obtained.

20. The object representation model training device according to claim 19, characterized in that, The object matching result includes object subgraph pairs consisting of a target object subgraph and a corresponding backward matching object subgraph, and trajectory identifiers corresponding to the object subgraph pairs. Different object subgraph pairs with overlapping object subgraphs have the same trajectory identifier. The training image set acquisition module is further used for: From the object matching results corresponding to each group of adjacent video frames in the current video frame sequence, obtain the object subgraph pairs corresponding to the current trajectory identifier to form the current object subgraph pair set corresponding to the current trajectory identifier; The current object subgraphs in the set are sorted according to the video frame order of the corresponding video frames to obtain the current initial object subgraph sequence corresponding to the current trajectory identifier; The fusion features of each object corresponding to the current initial object subgraph sequence are statistically analyzed to obtain the current object center features; Based on the feature distance between the fusion features of each object corresponding to the current initial object subgraph sequence and the current object center feature, a first preset number of object subgraphs are deleted from the current initial object subgraph sequence according to the feature distance from largest to smallest, to obtain the target object subgraph sequence corresponding to the current trajectory identifier.

21. The object representation model training device according to claim 16, characterized in that, When the current feature is an object association feature, the current label is the object association label, and the current classification loss is the object association classification loss; when the current feature is an object feature, the current label is the object label, and the current classification loss is the object classification loss; the loss determination module is further used for: Obtain the prototype object feature set corresponding to each current label; the prototype object feature set includes multiple prototype object features corresponding to the same current label, and the prototype object feature is obtained from the current feature library by obtaining the current feature that is closest to the cluster center feature corresponding to the current label. The cluster center feature is obtained by clustering the current features corresponding to the same current label in the current feature library. From each set of prototype object features, the prototype object feature with the smallest feature distance to the current feature corresponding to the current training image is selected as the reference object feature corresponding to the current training image. The current classification loss is obtained based on the difference between the current features and the reference object features corresponding to the same training image.

22. The object representation model training device according to claim 21, characterized in that, The current feature and the reference object feature are feature vectors of a preset length; the loss determination module is further configured to: Obtain a random feature vector of the preset length, and retain the vector positions corresponding to data whose values ​​are greater than or equal to a preset threshold in the random feature vector as reserved positions; The current classification loss is obtained based on the data differences between the current features and the reference object features corresponding to the same training image at the same retention position.

23. The object representation model training device according to claim 16, characterized in that, The initial object representation model includes an object association classification network, an object classification network, and a feature fusion network. The object association classification network is used to obtain object association features, the object classification network is used to obtain object features, and the feature fusion network is used to obtain object fusion features. The model adjustment module is further used for: Based on the object association classification loss, the network parameters of the object association classification network are adjusted according to the first learning rate; based on the object classification loss, the network parameters of the object classification network are adjusted according to the second learning rate; based on the fusion loss, the network parameters of the object association classification network, the object classification network, and the feature fusion network are adjusted according to the third learning rate to obtain an intermediate object representation model. Wherein, both the first learning rate and the second learning rate are greater than the third learning rate.

24. The object representation model training device according to claim 16, characterized in that, The device further includes a forward training module, which is used for: Obtain the second training image set, input the second training image set into the basic object representation model to be trained, and obtain the initial object features corresponding to each training image in the second training image set; The basic object representation model includes a feature extraction layer and a fully connected layer; The initial loss is obtained based on the features of each initial object corresponding to the same group of training images in the second training image set; The model parameters of the basic object representation model are adjusted based on the initial loss until the second convergence condition is met, thus obtaining the trained basic object representation model. Based on the feature extraction layer in the trained basic object representation model, the feature extraction layer in the candidate object representation model is initialized. Based on the fully connected layer in the trained basic object representation model, the fully connected layer in the candidate object representation model is initialized to obtain the initial object representation model.

25. The object representation model training device according to claim 24, characterized in that, The forward training module is also used for: Based on the feature extraction layer in the trained basic object representation model, the feature extraction layer shared between the object association classification network and the object classification network in the candidate object representation model is initialized. Based on the fully connected layer in the trained basic object representation model, the fully connected layer of the object association classification network and the fully connected layer of the object classification network in the candidate object representation model are initialized to obtain the initial object representation model.

26. The object representation model training apparatus according to any one of claims 16 to 25, characterized in that, The device further includes a backward training module, which is used for: Obtain a third training image set, input the third training image set into the target object representation model, and obtain the object fusion features corresponding to each training image in the third training image set; Based on the fusion features of each object corresponding to the same group of training images in the third training image set, the updated loss is obtained; The model parameters of the target object representation model are adjusted based on the update loss until the third convergence condition is met, thus obtaining the updated object representation model.

27. An object representation device, characterized in that, The device includes: The image acquisition module is used to acquire images of the object to be processed. The image processing module is used to input the image of the object to be processed into the target object representation model to obtain the object fusion features corresponding to the image of the object to be processed; the object fusion features corresponding to the image of the object to be processed are used to determine the image that matches the image of the object to be processed. The training process of the target object representation model includes the following steps: A first training image set is obtained and input into an initial object representation model. The first training image set includes multiple training image groups, each training image group including multiple mutually contrasting training images. Each training image contains an object label and an object association label. The object association label is used to characterize the content of the region displayed by the object association region in the image. The object association region refers to the local region in the image that is related to the object. Object association features are extracted from the training images to obtain object association features corresponding to each training image. Object features are extracted from the training images to obtain object features corresponding to each training image. The object features and object association features corresponding to the same training image are fused to obtain object fusion features corresponding to each training image. The object association features are used to characterize the feature information corresponding to the object association region in the image. Based on the object association features and object association labels corresponding to the same training image, we obtain the object association classification loss; based on the object features and object labels corresponding to the same training image, we obtain the object classification loss; based on the fusion features of each object corresponding to the same group of training images, we obtain the fusion loss. Based on the object association classification loss, the object classification loss, and the fusion loss, the model parameters of the initial object representation model are adjusted to obtain an intermediate object representation model. The intermediate object representation model is then used as the initial object representation model. The steps of obtaining the first training image set and inputting the first training image set into the initial object representation model are repeated until the first convergence condition is met, thereby obtaining the target object representation model.

28. The object characterization device according to claim 27, characterized in that, The device further includes a video processing module, the video processing module being used for: Obtain the video frame sequence corresponding to the video to be processed; Object detection is performed on the video frames in the video frame sequence to be processed to obtain object sub-graphs in the video frames. The object sub-graphs are then input into the target object representation model to obtain the object fusion features corresponding to the object sub-graphs. Based on each object subgraph and corresponding object fusion feature in adjacent video frames of the video frame sequence to be processed, a reference object trajectory corresponding to the video to be processed is obtained; the reference object trajectory includes the object fusion feature of the corresponding object subgraph. Based on the trajectories of each reference object corresponding to the video to be processed and the trajectories of each candidate object corresponding to the candidate videos in the video library, a query is performed from the video library to obtain the query results.

29. The object characterization device according to claim 28, characterized in that, The video processing module is also used for: Based on the feature distance between the fusion features of each object corresponding to the current reference object trajectory and the fusion features of each object corresponding to the current candidate object trajectory in the current candidate video, the number of feature similarities between the current reference object trajectory and the current candidate object trajectory is determined. The trajectory similarity between the current reference object trajectory and the current candidate object trajectory is determined based on the number of feature similarities. The current candidate object trajectory and the current reference object trajectory with a trajectory similarity greater than the preset similarity are regarded as similar trajectories. The video duplication rate between the video to be processed and the current candidate video is determined based on the number of similar trajectories between them. When the video repetition rate is greater than the preset repetition rate, the query result is determined to be a successful query, and the current candidate video is taken as a similar video to the video to be processed. When the video repetition is less than or equal to the preset repetition, the query result is determined to be a query failure.

30. The object characterization device according to claim 28, characterized in that, The video processing module is also used for: If the video to be processed is a video to be uploaded, and the query result is successful, the upload suggestion for the video to be processed is to prohibit upload; if the query result is unsuccessful, the upload suggestion for the video to be processed is to allow upload. If the video to be processed is an already uploaded video, and the query result is a successful query, the target video is determined from the video to be processed and the candidate videos similar to the video to be processed. The processing suggestion for the target video is to notify it to be taken down. If the query result is a failed query, the processing suggestion for the video to be processed and the candidate videos is to not process them.

31. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 15.

32. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 15.

33. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 15.

Images