Processing method, device and storage medium for multi-target tracking of video

Abstract

The present invention discloses a processing method, device and storage medium for multi-target tracking of a video. The method comprises: obtaining a played video, wherein there are multiple target objects to be tracked in the video; extracting feature information of each target object in the video; performing feature similarity composition and spatial similarity composition based on the feature information of each target object to generate a global feature map; performing aggregation and classification of nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and nodes in the same cluster represent the same target object. The present invention solves the technical problem of missing tracking results in the multi-target tracking process.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to a processing method, device and storage medium for performing multi-target tracking on a video. Background Art

[0002] Video multi-target tracking has a wide range of applications in the field of computer vision, such as pedestrian and vehicle tracking in unmanned driving, object tracking in image and video search, and pedestrian tracking in intelligent monitoring. Offline video multi-target tracking refers to the tracking task after all detected targets in the video have been obtained. Offline tracking is mainly divided into two steps: feature extraction and data association. The purpose of feature extraction is to vectorize the target detection frame, and data association is to associate all detection frames of the same target together. At present, the mainstream offline multi-target tracking algorithm usually follows the framework of track-by-detect, mainly focusing on using convolutional neural network (CNN) to learn better features of detection targets, and then calculate the matching similarity between targets, and measure whether different detection results belong to the same tracking target by similarity. However, this solution can usually only compare the similarity between two targets, which will lead to the technical problem of missing tracking results in the multi-target tracking process.

[0003] With regard to the problem of missing tracking results in the above-mentioned multi-target tracking process, no effective solution has been proposed so far. Summary of the invention

[0004] The embodiments of the present invention provide a processing method, device and storage medium for performing multi-target tracking on a video, so as to at least solve the technical problem of missing tracking results in the multi-target tracking process.

[0005] According to one aspect of an embodiment of the present invention, a method for processing multi-target tracking in a video is provided, comprising: obtaining a played video, wherein there are multiple target objects to be tracked in the video; extracting feature information of each target object in the video; performing feature similarity composition and spatial similarity composition based on the feature information of each target object to generate a global feature map; aggregating and classifying nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and nodes in the same cluster represent the same target object.

[0006] According to another aspect of an embodiment of the present invention, another processing method for multi-target tracking in a video is provided, comprising: recording a played video in an input interface of an operation interface, wherein there are multiple target objects to be tracked in the video; sensing a feature extraction instruction in the operation interface, and extracting feature information of each target object in the video; and displaying classification results on the operation interface, wherein the classification results include multiple clusters, nodes in the same cluster represent the same target object, and are obtained by aggregating and classifying nodes in a global feature map, and the global feature map is generated by performing feature similarity composition and spatial similarity composition based on the feature information of each target object.

[0007] According to another aspect of an embodiment of the present invention, another processing method for multi-target tracking in a video is provided, including: displaying a played video in an operation interface, wherein there are multiple target objects to be tracked in the video; sensing a feature extraction instruction in the operation interface; responding to the feature extraction instruction, extracting feature information of each target object in the video; displaying classification results on the operation interface, wherein the classification results include multiple clusters, nodes in the same cluster represent the same target object, and are obtained by aggregating and classifying nodes in a global feature map, and the global feature map is generated by performing feature similarity composition and spatial similarity composition based on the feature information of each target object.

[0008] According to another aspect of an embodiment of the present invention, another processing method for multi-target tracking in a video is provided, including: a front-end client uploads a played video, wherein there are multiple target objects to be tracked in the video; the front-end client transmits the video to a back-end server; the front-end client receives a global feature map returned by the back-end server, wherein the global feature map is generated by the back-end server based on feature information of each target object in the video by performing feature similarity composition and spatial similarity composition; the front-end client aggregates and classifies the nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object.

[0009] According to another aspect of an embodiment of the present invention, another processing method for multi-target tracking in a video is provided, comprising: obtaining a played video by calling a first interface, wherein the first interface comprises: a first parameter, a parameter value of the first parameter is a video, and there are multiple target objects to be tracked in the video; extracting feature information of each target object in the video; based on the feature information of each target object, performing feature similarity composition and spatial similarity composition to generate a global feature map; aggregating and classifying nodes in the global feature map to generate classification results, wherein the classification results include multiple clusters, and nodes in the same cluster represent the same target object; outputting the classification results by calling a second interface, wherein the second interface comprises: a second parameter, a parameter value of the second parameter is the classification result.

[0010] According to another aspect of an embodiment of the present invention, a processing device for performing multi-target tracking on a video is also provided, including: an acquisition unit, used to acquire a played video, wherein there are multiple target objects to be tracked in the video; an extraction unit, used to extract feature information of each target object in the video; a composition unit, used to perform feature similarity composition and spatial similarity composition based on the feature information of each target object to generate a global feature map; a classification unit, used to aggregate and classify the nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object.

[0011] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium may include a stored program, wherein when the program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the processing method for multi-target tracking in a video according to an embodiment of the present invention.

[0012] According to another aspect of an embodiment of the present invention, a processor is provided, which is used to run a program, wherein when the program is run, the method for processing multi-target tracking on a video according to an embodiment of the present invention is executed.

[0013] According to another aspect of an embodiment of the present invention, a processing system for tracking multiple targets in a video is also provided. The processing system for tracking multiple targets in a video may include: a processor; a memory connected to the processor, and used to provide the processor with instructions for processing the following processing steps: obtaining a played video, wherein there are multiple target objects to be tracked in the video; extracting feature information of each target object in the video; performing feature similarity composition and spatial similarity composition based on the feature information of each target object to generate a global feature map; performing aggregation and classification of nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and nodes in the same cluster represent the same target object.

[0014] In an embodiment of the present invention, a played video is obtained, wherein there are multiple target objects to be tracked in the video; feature information of each target object in the video is extracted; based on the feature information of each target object, feature similarity composition and spatial similarity composition are performed to generate a global feature map; nodes in the global feature map are aggregated and classified to generate a classification result, wherein the classification result includes multiple clusters, and nodes in the same cluster represent the same target object. In the related art, when an offline multi-target tracking algorithm is used to perform multi-target tracking processing on a video, it is usually based on a track-by-detect framework, mainly focusing on using a convolutional neural network (CNN) to learn better features of the detection target, and then calculating the matching similarity between the targets, and measuring whether different detection results belong to the same tracking target through the similarity, but this scheme only obtains the similarity between two targets, resulting in the problem of missing tracking results in the multi-target tracking process. In the present application, by extracting the feature information of each target object in the video, the extracted features can be subjected to feature similarity composition and spatial similarity composition, thereby constructing a global feature map, and using the graph convolutional neural network architecture to aggregate and classify the nodes in the global feature map, the neighborhood information of the nodes can be fully mined, thereby improving the use of global and spatial information, avoiding the problem of missing tracking results in the multi-target tracking process due to only being able to obtain the similarity between two targets, and thus solving the technical problem of missing tracking results in the multi-target tracking process, and achieving the technical effect of not missing tracking results in the multi-target tracking process. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0016] Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a processing method for multi-target tracking in a video according to an embodiment of the present invention;

[0017] Figure 2 is a flowchart of a method for processing multi-target tracking in a video according to an embodiment of the present invention;

[0018] Figure 3 is a schematic diagram of a scenario for performing multi-target tracking on a video according to an embodiment of the present invention;

[0019] Figure 4 is a schematic diagram of a processing device for performing multi-target tracking on a video according to an embodiment of the present invention;

[0020] Figure 5 is a structural block diagram of a mobile terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0024] Convolutional Neural Network (CNN): A feed-forward neural network whose artificial neurons can respond to surrounding units within a certain coverage area. It is a deep learning method for image recognition.

[0025] Graph Neural Network (GNN): A deep learning method for irregular data domains;

[0026] Intersection over Union (IOU) is expressed as a score to calculate the degree of overlap between two areas. The higher the IOU score, the higher the overlap between the two areas.

[0027] Example 1

[0028] According to an embodiment of the present invention, an embodiment of a processing method for multi-target tracking in a video is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a processing method for multi-target tracking in a video according to an embodiment of the present invention. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0030] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0031] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the processing method for multi-target tracking of video in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the processing method for multi-target tracking of video is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0034] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the above-described computer device (or mobile device).

[0035] exist Figure 1 In the operating environment shown, this application provides Figure 2 It should be noted that the processing method for performing multi-target tracking on a video in this embodiment is based on Figure 1 The illustrated embodiment is executed by a mobile terminal.

[0036] Figure 2FIG. 1 is a flowchart of a method for performing multi-target tracking on a video according to an embodiment of the present invention. Figure 2 As shown, the method may include the following steps:

[0037] Step S202, obtaining the video to be played.

[0038] In the technical solution provided in the above step S202 of the present invention, a live video played in real time can be obtained through live broadcast, or a short video that can be played at any time can be obtained by shooting a short video. In this embodiment, there can be multiple target objects to be tracked in the video, and the target object can be a commodity or product.

[0039] Step S204: extract feature information of each target object in the video.

[0040] In the technical solution provided in the above step S204 of the present invention, after the played video is acquired, feature information of each target object in the video may be extracted.

[0041] In this embodiment, a convolutional neural network model can be used, combined with a detection bounding box and a video frame file, to extract feature information of each target object in the video. Optionally, the image in the video frame can be cut out and after the image size is adjusted, the image can be input into the convolutional neural network model to extract feature information of the target object.

[0042] Step S206: Based on the feature information of each target object, feature similarity mapping and spatial similarity mapping are performed to generate a global feature map.

[0043] In the technical solution provided in the above step S206 of the present invention, after extracting the feature information of each target object in the video, feature similarity mapping and spatial similarity mapping may be performed on the extracted features to generate a global feature map.

[0044] In this embodiment, when performing feature similarity mapping on the extracted features, each object (target) in the video can be regarded as a node, and then a graph convolutional network architecture is used to obtain the neighborhood information of each node, and a feature map is constructed using the node and the neighborhood information of the node, so that a feature map can be obtained for each node, and a global feature map is generated based on this feature map.

[0045] In this embodiment, when performing spatial similarity mapping on the extracted features, the feature map of each node may be obtained first, and then the overlap between the feature maps of each node may be determined, and a global feature map may be generated based on the overlap.

[0046] Step S208 , clustering and classifying the nodes in the global feature graph to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object.

[0047] In the technical solution provided in the above step S208 of the present invention, after the global feature map is generated, the nodes in the global feature map can be aggregated and classified. Optionally, the nodes in the global feature map that represent the same target object can be first aggregated to form a cluster. Then, after all the nodes in the global feature map are aggregated, multiple clusters can be formed, and each cluster represents a type of target object. In this way, all the nodes in the global feature map can be classified, thereby generating multiple types of target objects.

[0048] In the related art, when using an offline multi-target tracking algorithm to track multiple targets in a video, it is usually based on a track-by-detect framework, focusing on using a convolutional neural network to learn better features of the detection target, and then calculating the matching similarity between the targets. The similarity is used to measure whether different detection results belong to the same tracking target. However, this solution only obtains the similarity between two targets, resulting in the problem of missing tracking results in the multi-target tracking process.

[0049] Through the above steps S202 to S208 of the present application, a played video is obtained, wherein there are multiple target objects to be tracked in the video; feature information of each target object in the video is extracted; based on the feature information of each target object, feature similarity composition and spatial similarity composition are performed to generate a global feature map; nodes in the global feature map are aggregated and classified to generate classification results, wherein the classification results include multiple clusters, and nodes in the same cluster represent the same target object. In other words, by extracting the feature information of each target object in the video, feature similarity composition and spatial similarity composition can be performed on the extracted features to construct a global feature map, and the nodes in the global feature map are aggregated and classified using the graph convolutional neural network architecture, which can fully mine the neighborhood information of the nodes, thereby improving the use of global information and spatial information, avoiding the problem of missing tracking results in the multi-target tracking process due to only being able to obtain the similarity between two targets, thereby solving the technical problem of missing tracking results in the multi-target tracking process, and achieving the technical effect of not missing tracking results in the multi-target tracking process.

[0050] The above method of this embodiment is further described below.

[0051] As an optional implementation, a pre-trained CNN model is used to detect features in the video, extract feature information of each target object, and output a feature vector of each target object.

[0052] In this embodiment, a convolutional neural network (CNN) model pre-trained on an image network (ImageNet) using a residual network (Resnet50) architecture can be used, and the detection bounding box and video frame file can be combined to cut out and resize the target object in the video, and then the processed image is input into the convolutional neural network model to extract the features of the target object, thereby obtaining the feature information of the target object and outputting the feature vector of each target object.

[0053] As an optional implementation, step S206, based on the feature information of each target object, performs feature similarity composition and spatial similarity composition, including: using a proximity algorithm to classify and divide the feature information of each target object to construct a proximity classification map; using each node in the proximity classification map as a central node for composition to generate a feature similarity composition and a spatial similarity composition.

[0054] In this embodiment, after acquiring the feature information of the target object, a proximity algorithm (KNN algorithm) can be used to construct a proximity classification graph according to the feature vector of each target object, and then each node in the proximity classification graph is used as the central node to perform composition again, thereby obtaining a feature similarity composition and a spatial similarity composition.

[0055] In the above embodiment, each node in the neighbor classification graph can be used to represent a target object.

[0056] In the above embodiment, the generated feature similarity composition and spatial similarity composition may include multiple sub-graphs, and each sub-graph may include a central node and a neighborhood associated with the central node. The nodes in the neighborhood are the neighbor nodes of the central node, and an association relationship can be set between the neighbor nodes in the neighborhood that meet the association conditions.

[0057] As an optional implementation, each node in the proximity classification graph is used as a central node for composition to generate a feature similarity composition, including: obtaining multiple neighbor nodes associated with the central node in the proximity classification graph, and the two-hop neighbor nodes of each neighbor node, to obtain the neighborhood associated with the central node; performing a similarity comparison of feature vectors for any two neighbor nodes in the neighborhood, and if the comparison result is that they are neighbors in the proximity classification graph, it is determined that the target objects represented by the two neighbor nodes are the same object in the video.

[0058] In this embodiment, a feature similarity composition is generated by obtaining a plurality of neighbor nodes associated with a central node (which may be a first-level central node) in a neighbor classification graph, and when the neighbor node is used as a central node (which may be a second-level central node), a two-hop neighbor node associated with the central node (second-level central node). Through the neighbor nodes and the two-hop neighbor nodes, a neighborhood associated with the central node (first-level central node) can be obtained. At this time, except for the central node (first-level central node), all other nodes in the neighborhood can be called neighbor nodes. Then, a similarity comparison is performed on the feature vectors of the target objects represented by any two neighbor nodes in the neighborhood. If, in the neighbor classification graph, the above two neighbor nodes are neighbors to each other, the target objects represented by the two neighbor nodes are the same target object in the video, otherwise, the target objects represented by the two neighbor nodes are different objects in the video.

[0059] In the above embodiment, the similarity of feature vectors of any two neighboring nodes in the neighborhood is compared. Specifically, in the neighborhood, if any two nodes are neighbors of each other in the neighboring classification graph, edges are added between the two neighboring nodes to generate a global feature graph.

[0060] As an optional implementation, each node in a neighbor classification graph is used as a central node for composition to generate a spatial similarity composition, including: obtaining multiple neighbor nodes associated with the central node in the neighbor classification graph, and two-hop neighbor nodes of each neighbor node, to obtain a neighborhood associated with the central node; obtaining spatial boxes of any two neighbor nodes in the neighborhood; performing spatial similarity comparison on the spatial boxes of any two neighbor nodes, and if the comparison result is that the area overlap of the spatial boxes of the two neighbor nodes exceeds a predetermined value, it is determined that the target objects represented by the two compared neighbor nodes are the same object in the video.

[0061] In this embodiment, a spatial similarity composition is generated by obtaining a plurality of neighbor nodes associated with a central node (which may be a first-level central node) in a neighbor classification graph, and when the neighbor node is used as a central node (which may be a second-level central node), a two-hop neighbor node associated with the central node (second-level central node). Through the neighbor nodes and the two-hop neighbor nodes, a neighborhood associated with the central node (first-level central node) can be obtained. At this time, except for the central node (first-level central node), all other nodes in the neighborhood can be called neighbor nodes. Then, the spatial boxes of any two neighbor nodes in the neighborhood are obtained, and the spatial similarity comparison of the spatial boxes of any two neighbor nodes is performed. If the area overlap of the spatial boxes of the two neighbor nodes exceeds a predetermined value, the target objects represented by the two neighbor nodes are the same target object in the video, otherwise, the target objects represented by the two neighbor nodes are different objects in the video.

[0062] In the above embodiment, the spatial boxes of any two neighboring nodes are compared for spatial similarity. Specifically, in the neighborhood, the overlap (IOU) between the spatial boxes of any two nodes is calculated. If the overlap is greater than 50%, an edge is added between the two nodes to generate a global feature map.

[0063] As an optional implementation, a topK algorithm is used to obtain multiple neighbor nodes associated with the central node in the proximity classification graph, as well as the two-hop neighbor nodes of each neighbor node.

[0064] In this embodiment, the topK algorithm can be used to obtain neighbor nodes associated with the central node based on the neighbor classification graph, and then the topK algorithm can be used again to obtain two-hop neighbor nodes associated with the neighbor nodes based on the neighbor classification graph, so that the central node can obtain a neighborhood of size K*K at most.

[0065] As an optional implementation, in the process of obtaining the neighborhood associated with the central node, the node vector in each subgraph and the adjacency matrix corresponding to the subgraph are input into the GCN neural network model for graph convolution, so that the neighborhood completes information aggregation.

[0066] In this embodiment, a four-layer third-generation graph convolutional neural network model can be used to aggregate and propagate the node vectors in each input subgraph and the adjacency matrix corresponding to the subgraph, so that a new node vector can be generated, so that the neighborhood associated with the central node completes information aggregation.

[0067] As an optional implementation, after completing the graph convolution, two fully connected layers are added to integrate the feature information after the convolution, and the cross entropy is used as the loss function for node classification.

[0068] In this embodiment, after adopting four layers of graph convolution layers, two layers of fully connected layers (FC) can be added to complete the integration of feature information after graph convolution, and cross entropy can be used as the loss function for node classification.

[0069] As an optional implementation, after the nodes in the global feature map are aggregated and classified, the target objects aggregated into the same cluster are associated based on the playback time of the video to obtain the playback trajectory of the target object.

[0070] In this embodiment, since the playback time of the video has a chronological order, the target objects represented by the nodes aggregated into the same cluster in the global feature map can be associated based on the playback time of the video, and then the playback trajectory of the target object can be formed according to the chronological order.

[0071] The embodiment of the present invention also provides another processing method for multi-target tracking of a video from the perspective of human-computer interaction. The method may include: recording a played video in an input interface of an operation interface, wherein there are multiple target objects to be tracked in the video; sensing a feature extraction instruction in the operation interface, extracting feature information of each target object in the video; displaying a classification result on the operation interface, wherein the classification result includes multiple clusters, nodes in the same cluster represent the same target object, and are obtained by aggregating and classifying nodes in a global feature map, and the global feature map is generated by performing feature similarity composition and spatial similarity composition based on the feature information of each target object.

[0072] In this embodiment, the video to be played can be recorded in the recording interface on the operation interface. The video can be a live video played in real time or a short video that can be played at any time. After the operation interface receives the feature extraction instruction, the feature information of each target object to be tracked in the recorded video can be extracted according to the instructions of the instruction, and then the classification results including multiple clusters are displayed on the operation interface. The nodes in the same cluster represent the same target object.

[0073] In the above embodiment, after extracting the feature information of each target object in the video, feature similarity mapping and spatial similarity mapping can be performed on the extracted feature information to generate a global feature map, and then the nodes in the global feature map can be aggregated and classified to generate a classification result. Optionally, the nodes representing the same target object in the global feature map can be first aggregated to form a cluster. After all the nodes in the global feature map are aggregated, multiple clusters can be formed, each cluster representing a type of target object. In this way, all nodes in the global feature map can be classified to generate multiple types of target objects.

[0074] Optionally, the target object in this embodiment may be a commodity or a product.

[0075] The embodiment of the present invention also provides another processing method for multi-target tracking of a video from the perspective of human-computer interaction. The method may include: displaying a played video in an operation interface, wherein there are multiple target objects to be tracked in the video; sensing a feature extraction instruction in the operation interface; responding to the feature extraction instruction, extracting feature information of each target object in the video; displaying a classification result on the operation interface, wherein the classification result includes multiple clusters, nodes in the same cluster represent the same target object, and are obtained by aggregating and classifying nodes in a global feature map, and the global feature map is generated by performing feature similarity composition and spatial similarity composition based on the feature information of each target object.

[0076] In this embodiment, the played video can be displayed through the operation interface. The video can be a live video played in real time or a short video that can be played at any time. After the operation interface receives the feature extraction instruction, it can respond to the instruction, extract the feature information of each target object to be tracked in the video, and then display the classification results including multiple clusters on the operation interface, and the nodes in the same cluster represent the same target object.

[0077] In the above embodiment, after extracting the feature information of each target object in the video, feature similarity mapping and spatial similarity mapping can be performed on the extracted feature information to generate a global feature map, and then the nodes in the global feature map can be aggregated and classified to generate a classification result. Optionally, the nodes representing the same target object in the global feature map can be first aggregated to form a cluster. After all the nodes in the global feature map are aggregated, multiple clusters can be formed, each cluster representing a type of target object. In this way, all nodes in the global feature map can be classified to generate multiple types of target objects.

[0078] Optionally, the target object in this embodiment may be a commodity or a product.

[0079] The embodiment of the present invention also provides another processing method for multi-target tracking of a video from the user's perspective. The method may include: the front-end client uploads a played video, wherein there are multiple target objects to be tracked in the video; the front-end client transmits the video to the back-end server; the front-end client receives a global feature map returned by the back-end server, wherein the global feature map is generated by the back-end server based on the feature information of each target object in the video, performing feature similarity composition and spatial similarity composition; the front-end client aggregates and classifies the nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object.

[0080] In this embodiment, the video to be played can be uploaded by the front-end client. The video can be a live video played in real time or a short video that can be played at any time. After the front-end client uploads the played video, the video can be transmitted to the back-end server. After extracting the feature information of each target object to be tracked in the video, the back-end server can perform feature similarity composition and spatial similarity composition on the extracted feature information to generate a global feature map, and return the global feature map to the front-end client. After receiving the global feature map, the front-end client can aggregate and classify the nodes in the global feature map to generate a classification result. The classification result includes multiple clusters, and the nodes in the same cluster represent the same target object.

[0081] Optionally, this embodiment may first aggregate the nodes in the global feature map that represent the same target object to form a cluster. After all the nodes in the global feature map are aggregated, multiple clusters may be formed, each cluster representing a type of target object. In this way, all the nodes in the global feature map may be classified, thereby generating multiple types of target objects.

[0082] Optionally, the target object in this embodiment may be a commodity or a product.

[0083] The embodiment of the present invention also provides another processing method for multi-target tracking of a video. The method may include: obtaining the played video by calling a first interface, wherein the first interface includes: a first parameter, the parameter value of the first parameter is a video, and there are multiple target objects to be tracked in the video; extracting feature information of each target object in the video; based on the feature information of each target object, performing feature similarity composition and spatial similarity composition to generate a global feature map; aggregating and classifying the nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object; outputting the classification result by calling a second interface, wherein the second interface includes: a second parameter, and the parameter value of the second parameter is the classification result.

[0084] In this embodiment, in order to better track multiple targets in a video, the played video can be obtained by calling a first interface in the server, and the first interface includes a first parameter whose parameter value is the video. After the server obtains the played video, the feature information of each target object to be tracked in the video is extracted, and then the extracted feature information is subjected to feature similarity composition and spatial similarity composition to generate a global feature map, and then the nodes in the global feature map can be aggregated and classified to generate a classification result including multiple clusters, and the nodes in the same cluster represent the same target object. After the classification result is generated, the generated classification result can be output by calling a second interface in the server, and the second interface includes a second parameter whose parameter value is the classification result.

[0085] Optionally, this embodiment may first aggregate the nodes in the global feature map that represent the same target object to form a cluster. After all the nodes in the global feature map are aggregated, multiple clusters may be formed, each cluster representing a type of target object. In this way, all the nodes in the global feature map may be classified, thereby generating multiple types of target objects.

[0086] Optionally, the target object in this embodiment may be a commodity or a product.

[0087] In the related art, the scheme of using CNN to learn feature vectors and then calculating similarity can only use the information between two contrasting targets, and cannot mine global information or spatiotemporal information. In this embodiment, the global detection target is regarded as a node to construct a global feature map, and the GCN network architecture is used to aggregate the node neighborhood information. The neighborhood information of the node can be fully mined to improve the use of global information. In the composition process, in addition to using similarity composition, the overlap of the bounding boxes of the nodes is also calculated. When the overlap between the bounding boxes of two nodes is greater than 50%, the neighborhood is added to improve the use of spatial information.

[0088] In addition, in the related art, the training speed of the CNN model is slow, thus increasing the model training time. However, this embodiment directly uses the pre-trained CNN model to extract features of the detection object, eliminating the CNN model training process and reducing the overall training time, thereby solving the technical problem of missing tracking results in the multi-target tracking process and achieving the technical effect of not missing tracking results in the multi-target tracking process.

[0089] Example 2

[0090] The present application is further introduced below in combination with application scenarios.

[0091] Figure 3 FIG. 1 is a schematic diagram of a scenario for performing multi-target tracking on a video according to an embodiment of the present invention. Figure 3 As shown, a computer device obtains a video for playback, which may be a live video played in real time or a short video that can be played at any time. There may be multiple target objects to be tracked in the video, such as goods or products in the video, and then the playable video is input into the computing device.

[0092] In this embodiment, the feature information of each target object in the video can be extracted in the computer device, so that feature similarity composition and spatial similarity composition can be performed based on the feature information of each target object. When performing feature similarity composition on the extracted features, each object (target) in the video can be regarded as a node, and then the graph convolution network architecture is used to obtain the neighborhood information of each node, and the node and the neighborhood information of the node are used to form a feature map, so that a feature map can be obtained for each node, and a global feature map is generated based on this feature map; when performing spatial similarity composition on the extracted features, the feature map of each node can be obtained first, and then the overlap between the feature maps of each node can be determined, and a global feature map is generated based on this overlap.

[0093] In this embodiment, after the feature similarity composition and the space similarity composition are acquired, a global feature map may be generated based on the feature similarity composition and the space similarity composition, and then the global feature map may be output.

[0094] In this embodiment, after generating the global feature map, the nodes in the global feature map can be aggregated and classified. Optionally, the nodes in the global feature map that represent the same target object can be first aggregated to form a cluster. After aggregating all the nodes in the global feature map, multiple clusters can be formed, each cluster representing a type of target object. In this way, all the nodes in the global feature map can be classified to generate multiple types of target objects, which are then output to the display interface of the computing device for display.

[0095] In the prior art, the solution of using CNN to learn feature vectors and then calculate similarity can only use the information between two contrasting targets, and cannot mine global information or spatiotemporal information, and the training speed of CNN model is slow. However, this embodiment uses the GCN network to perform graph convolution to convert the tracking problem into a clustering problem, and uses a method of performing feature composition using feature similarity and spatial similarity respectively, which can solve the problem of missing tracking results in the multi-target tracking process, thereby achieving the purpose of not missing tracking results in the multi-target tracking process.

[0096] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0098] Example 3

[0099] According to an embodiment of the present invention, a processing device for performing multi-target tracking on a video is also provided for implementing the above-mentioned processing method for performing multi-target tracking on a video. It should be noted that the processing device for performing multi-target tracking on a video of this embodiment can be used to perform the processing method of the embodiment of the present invention. Figure 2 The method for processing multi-target tracking in video is shown.

[0100] Figure 4 FIG. 1 is a schematic diagram of a processing device for performing multi-target tracking on a video according to an embodiment of the present invention. Figure 4 As shown, the processing device 40 for performing multi-target tracking on a video may include: an acquisition unit 41 , an extraction unit 42 , a composition unit 43 and a classification unit 44 .

[0101] An acquisition unit 41 is used to acquire a played video, wherein there are multiple target objects to be tracked in the video;

[0102] An extraction unit 42, used to extract feature information of each target object in the video;

[0103] A mapping unit 43, configured to perform feature similarity mapping and spatial similarity mapping based on feature information of each target object to generate a global feature map;

[0104] The classification unit 44 is used to aggregate and classify the nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object.

[0105] It should be noted that the acquisition unit 41, extraction unit 42, composition unit 43 and classification unit 44 correspond to steps S202 to S208 in Example 1, and the four units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned units, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0106] In the processing device for multi-target tracking in a video of this embodiment, by extracting the feature information of each target object in the video, the extracted features can be subjected to feature similarity mapping and spatial similarity mapping, thereby constructing a global feature map, and the nodes in the global feature map are aggregated and classified using a graph convolutional neural network architecture, which can fully mine the neighborhood information of the nodes, thereby improving the use of global and spatial information, avoiding the problem of missing tracking results in the multi-target tracking process due to only being able to obtain the similarity between two targets, thereby solving the technical problem of missing tracking results in the multi-target tracking process, and achieving the technical effect of not missing tracking results in the multi-target tracking process.

[0107] Example 4

[0108] The embodiment of the present invention can provide a computer terminal, which can be set in the processing system for multi-target tracking of video in the embodiment of the present invention, and the computer terminal can be any computer terminal device in the computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0109] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0110] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the processing method for multi-target tracking of a video: obtaining a played video, wherein there are multiple target objects to be tracked in the video; extracting feature information of each target object in the video; based on the feature information of each target object, performing feature similarity composition and spatial similarity composition to generate a global feature map; aggregating and classifying the nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object.

[0111] Optionally, Figure 5 is a structural block diagram of a computer terminal according to an embodiment of the present invention. Figure 5 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 502 , a memory 504 and a transmission device 506 .

[0112] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the security vulnerability detection method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned processing method for multi-target tracking of video. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0113] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the played video, wherein there are multiple target objects to be tracked in the video; extract the feature information of each target object in the video; based on the feature information of each target object, perform feature similarity composition and spatial similarity composition to generate a global feature map; aggregate and classify the nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object.

[0114] Optionally, the processor may also execute the following program code: using a pre-trained CNN model to detect features in the video, extracting feature information of each target object, and outputting a feature vector of each target object.

[0115] Optionally, the processor may also execute program code for the following steps: based on the feature information of each target object, perform feature similarity composition and spatial similarity composition, including: using a proximity algorithm to classify and divide the feature information of each target object to construct a proximity classification graph; using each node in the proximity classification graph as a central node for composition to generate a feature similarity composition and a spatial similarity composition, wherein the feature similarity composition and the spatial similarity composition each have multiple subgraphs, each subgraph includes: a central node, a neighborhood associated with the central node, and an association relationship is set between neighbor nodes in the neighborhood that meet the association conditions.

[0116] Optionally, the processor may also execute the program code of the following steps: composing each node in the proximity classification graph as a central node to generate a feature similarity composition, including: obtaining multiple neighbor nodes associated with the central node in the proximity classification graph, and two-hop neighbor nodes of each neighbor node, to obtain a neighborhood associated with the central node; performing a similarity comparison of feature vectors of any two neighbor nodes in the neighborhood, and if the comparison result is that the two nodes are neighbors in the proximity classification graph, then determining that the target objects represented by the two neighbor nodes are the same object in the video.

[0117] Optionally, the processor may also execute program code for the following steps: composing each node in the proximity classification map as a central node to generate a spatial similarity composition, including: obtaining multiple neighbor nodes associated with the central node in the proximity classification map, and two-hop neighbor nodes of each neighbor node, to obtain a neighborhood associated with the central node; obtaining spatial boxes of any two neighbor nodes in the neighborhood; performing spatial similarity comparison on the spatial boxes of any two neighbor nodes, and if the comparison result is that the area overlap of the spatial boxes of the two neighbor nodes exceeds a predetermined value, then determining that the target objects represented by the two compared neighbor nodes are the same object in the video.

[0118] Optionally, the processor may also execute the program code of the following steps: using the topK algorithm to obtain multiple neighbor nodes associated with the central node in the proximity classification graph, and the two-hop neighbor nodes of each neighbor node.

[0119] Optionally, the processor may also execute the program code of the following steps: in the process of obtaining the neighborhood associated with the central node, the node vector in each subgraph and the adjacency matrix corresponding to the subgraph are input into the GCN neural network model for graph convolution, so that the neighborhood completes information aggregation.

[0120] Optionally, the processor may also execute the following program code: after completing the graph convolution, add two fully connected layers FC to integrate the feature information after the convolution, and use the cross entropy as the loss function for node classification.

[0121] Optionally, the processor may also execute program code of the following steps: after aggregating and classifying the nodes in the global feature map, associating the target objects aggregated into the same cluster based on the playback time of the video to obtain the playback trajectory of the target objects.

[0122] As another optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: record the played video in the input interface of the operation interface, wherein there are multiple target objects to be tracked in the video; sense the feature extraction instruction in the operation interface, and extract the feature information of each target object in the video; display the classification results on the operation interface, wherein the classification results include multiple clusters, the nodes in the same cluster represent the same target object, and are obtained by aggregating and classifying the nodes in the global feature map, and the global feature map is generated by performing feature similarity composition and spatial similarity composition based on the feature information of each target object.

[0123] As another optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: display the played video in the operation interface, wherein there are multiple target objects to be tracked in the video; sense the feature extraction instruction in the operation interface; respond to the feature extraction instruction to extract the feature information of each target object in the video; display the classification results on the operation interface, wherein the classification results include multiple clusters, the nodes in the same cluster represent the same target object, and are obtained by aggregating and classifying the nodes in the global feature map, and the global feature map is generated by performing feature similarity composition and spatial similarity composition based on the feature information of each target object.

[0124] As another optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: the front-end client uploads the played video, wherein there are multiple target objects to be tracked in the video; the front-end client transmits the video to the back-end server; the front-end client receives the global feature map returned by the back-end server, wherein the global feature map is generated by the back-end server based on the feature information of each target object in the video, performing feature similarity composition and spatial similarity composition; the front-end client aggregates and classifies the nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object.

[0125] As another optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the played video by calling the first interface, wherein the first interface includes: a first parameter, the parameter value of the first parameter is the video, and there are multiple target objects to be tracked in the video; extract the feature information of each target object in the video; based on the feature information of each target object, perform feature similarity composition and spatial similarity composition to generate a global feature map; aggregate and classify the nodes in the global feature map to generate classification results, wherein the classification results include multiple clusters, and the nodes in the same cluster represent the same target object; output the classification results by calling the second interface, wherein the second interface includes: a second parameter, the parameter value of the second parameter is the classification result.

[0126] By adopting the embodiment of the present invention, a scheme of a processing method for multi-target tracking of a video is provided. A video to be played is obtained, wherein there are multiple target objects to be tracked in the video; feature information of each target object in the video is extracted; based on the feature information of each target object, feature similarity composition and spatial similarity composition are performed to generate a global feature map; nodes in the global feature map are aggregated and classified to generate classification results, wherein the classification results include multiple clusters, and nodes in the same cluster represent the same target object, that is, by extracting the feature information of each target object in the video, feature similarity composition and spatial similarity composition can be performed on the extracted features to construct a global feature map, and the nodes in the global feature map are aggregated and classified using a graph convolutional neural network architecture, which can fully mine the neighborhood information of the nodes, thereby improving the use of global information and spatial information, avoiding the problem of missing tracking results in the multi-target tracking process due to only being able to obtain the similarity between two targets, thereby solving the technical problem of missing tracking results in the multi-target tracking process, and achieving the technical effect of not missing tracking results in the multi-target tracking process.

[0127] It can be understood by those skilled in the art that Figure 5The structure shown is for illustration only, and the mobile terminal A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 5 It does not limit the structure of the above electronic device. For example, the mobile terminal A may also include Figure 5 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 5 Different configurations are shown.

[0128] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0129] Example 5

[0130] The embodiment of the present invention further provides a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the processing method for performing multi-target tracking on a video provided in the first embodiment.

[0131] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0132] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: obtaining a played video, wherein there are multiple target objects to be tracked in the video; extracting feature information of each target object in the video; performing feature similarity composition and spatial similarity composition based on the feature information of each target object to generate a global feature map; aggregating and classifying the nodes in the global feature map to generate classification results, wherein the classification results include multiple clusters, and the nodes in the same cluster represent the same target object.

[0133] Optionally, the storage medium is also configured to store program codes for executing the following steps: using a pre-trained CNN model to detect features in the video, extracting feature information of each target object, and outputting a feature vector of each target object.

[0134] Optionally, the storage medium is also configured to store program code for executing the following steps: based on the feature information of each target object, performing feature similarity composition and spatial similarity composition, including: using a proximity algorithm to classify and divide the feature information of each target object to construct a proximity classification graph; composing each node in the proximity classification graph as a central node to generate a feature similarity composition and a spatial similarity composition, wherein the feature similarity composition and the spatial similarity composition each have multiple subgraphs, each subgraph includes: a central node, a neighborhood associated with the central node, and an association relationship is set between neighbor nodes in the neighborhood that meet the association conditions.

[0135] Optionally, the storage medium is also configured to store program code for executing the following steps: composing each node in the proximity classification map as a central node to generate a feature similarity composition, including: obtaining multiple neighbor nodes associated with the central node in the proximity classification map, and two-hop neighbor nodes of each neighbor node, to obtain a neighborhood associated with the central node; performing a similarity comparison of feature vectors for any two neighbor nodes in the neighborhood, and if the comparison result is that they are neighbors in the proximity classification map, then determining that the target objects represented by the two neighbor nodes are the same object in the video.

[0136] Optionally, the storage medium is also configured to store program code for executing the following steps: composing each node in the proximity classification map as a central node to generate a spatial similarity composition, including: obtaining multiple neighbor nodes associated with the central node in the proximity classification map, and two-hop neighbor nodes of each neighbor node, to obtain a neighborhood associated with the central node; obtaining spatial boxes of any two neighbor nodes in the neighborhood; performing spatial similarity comparison on the spatial boxes of any two neighbor nodes, and if the comparison result is that the area overlap of the spatial boxes of the two neighbor nodes exceeds a predetermined value, it is determined that the target objects represented by the two compared neighbor nodes are the same object in the video.

[0137] Optionally, the storage medium is further configured to store program codes for executing the following steps: using a topK algorithm to obtain multiple neighbor nodes associated with the central node in the proximity classification graph, and two-hop neighbor nodes of each neighbor node.

[0138] Optionally, the storage medium is also configured to store program code for performing the following steps: in the process of obtaining the neighborhood associated with the central node, the node vector in each subgraph and the adjacency matrix corresponding to the subgraph are input into the GCN neural network model for graph convolution, so that the neighborhood completes information aggregation.

[0139] Optionally, the storage medium is also configured to store program codes for executing the following steps: after completing the graph convolution, adding two fully connected layers FC to integrate the feature information after the convolution, and using cross entropy as the loss function for node classification.

[0140] Optionally, the storage medium is also configured to store program code for executing the following steps: after aggregating and classifying the nodes in the global feature map, associating the target objects aggregated into the same cluster based on the playback time of the video to obtain the playback trajectory of the target object.

[0141] As an optional example, the computer-readable storage medium is also configured to store program code for executing the following steps: recording a played video in an input interface of an operation interface, wherein there are multiple target objects to be tracked in the video; sensing a feature extraction instruction in the operation interface, and extracting feature information of each target object in the video; displaying classification results on the operation interface, wherein the classification results include multiple clusters, nodes in the same cluster represent the same target object, and are obtained by aggregating and classifying the nodes in a global feature map, and the global feature map is generated by performing feature similarity composition and spatial similarity composition based on the feature information of each target object.

[0142] As an optional example, the computer-readable storage medium is also configured to store program code for executing the following steps: displaying a played video in an operation interface, wherein there are multiple target objects to be tracked in the video; sensing a feature extraction instruction in the operation interface; extracting feature information of each target object in the video in response to the feature extraction instruction; and displaying classification results on the operation interface, wherein the classification results include multiple clusters, nodes in the same cluster represent the same target object, and are obtained by aggregating and classifying the nodes in a global feature map, and the global feature map is generated by performing feature similarity composition and spatial similarity composition based on the feature information of each target object.

[0143] As an optional example, the computer-readable storage medium is also configured to store program code for executing the following steps: the front-end client uploads a video for playback, wherein there are multiple target objects to be tracked in the video; the front-end client transmits the video to the back-end server; the front-end client receives a global feature map returned by the back-end server, wherein the global feature map is generated by the back-end server based on feature information of each target object in the video, performing feature similarity composition and spatial similarity composition; the front-end client aggregates and classifies the nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object.

[0144] As an optional example, the computer-readable storage medium is also configured to store program code for executing the following steps: obtaining a played video by calling a first interface, wherein the first interface includes: a first parameter, a parameter value of the first parameter is a video, and there are multiple target objects to be tracked in the video; extracting feature information of each target object in the video; based on the feature information of each target object, performing feature similarity composition and spatial similarity composition to generate a global feature map; aggregating and classifying the nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object; outputting the classification result by calling a second interface, wherein the second interface includes: a second parameter, a parameter value of the second parameter is the classification result.

[0145] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0146] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0151] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for processing multi-target tracking in a video, characterized in that: include: Acquire a played video, wherein the video contains a plurality of target objects to be tracked; Extracting feature information of each target object in the video; Based on the feature information of each target object, feature similarity composition and spatial similarity composition are performed to generate a global feature map; Aggregate and classify the nodes in the global feature graph to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object; Based on the feature information of each target object, feature similarity composition and spatial similarity composition are performed to generate a global feature map, including: performing feature similarity composition on the feature information to obtain multiple feature maps, wherein each target object corresponds to each feature map; performing spatial similarity composition on the feature information to obtain the overlap between the multiple feature maps; and generating the global feature map based on the overlap.

2. The method according to claim 1, characterized in that Use the pre-trained CNN model to detect features in the video, extract feature information of each target object, and output a feature vector of each target object.

3. The method according to claim 1, characterized in that Based on the feature information of each target object, feature similarity composition and space similarity composition are performed, including: Using a proximity algorithm to classify and divide the feature information of each target object, and construct a proximity classification graph; Each node in the proximity classification graph is used as a central node for composition to generate the feature similarity composition and the spatial similarity composition, wherein the feature similarity composition and the spatial similarity composition each have multiple subgraphs, each subgraph includes: a central node, a neighborhood associated with the central node, and an association relationship is set between neighbor nodes in the neighborhood that meet the association condition.

4. The method according to claim 3, characterized in that Each node in the neighboring classification graph is used as a central node to construct a graph to generate the feature similarity graph, including: Acquire multiple neighbor nodes associated with the central node in the proximity classification graph, and two-hop neighbor nodes of each neighbor node, to acquire a neighborhood associated with the central node; A similarity comparison of feature vectors is performed on any two neighboring nodes in the neighborhood. If the comparison result shows that the two neighboring nodes are neighbors of each other in the neighborhood classification graph, it is determined that the target objects represented by the two neighboring nodes are the same object in the video.

5. The method according to claim 3, characterized in that: Each node in the neighboring classification graph is used as a central node to compose a graph, and the spatial similarity composition is generated, including: Acquire multiple neighbor nodes associated with the central node in the proximity classification graph, and two-hop neighbor nodes of each neighbor node, to acquire a neighborhood associated with the central node; Obtaining the spatial boxes of any two neighboring nodes in the neighborhood; The spatial boxes of any two neighboring nodes are compared for spatial similarity. If the comparison result shows that the area overlap of the spatial boxes of the two neighboring nodes exceeds a predetermined value, it is determined that the target objects represented by the two compared neighboring nodes are the same object in the video.

6. The method according to claim 4 or 5, characterized in that: The topK algorithm is used to obtain multiple neighbor nodes associated with the central node in the proximity classification graph, as well as the two-hop neighbor nodes of each neighbor node.

7. The method according to any one of claims 3 to 5, characterized in that In the process of obtaining the neighborhood associated with the central node, the node vector in each subgraph and the adjacency matrix corresponding to the subgraph are input into the GCN neural network model for graph convolution, so that the neighborhood completes information aggregation.

8. The method according to claim 7, characterized in that After completing the graph convolution, two fully connected layers FC are added to integrate the feature information after convolution, and cross entropy is used as the loss function for node classification.

9. The method according to claim 1, characterized in that: After the nodes in the global feature map are aggregated and classified, the target objects aggregated into the same cluster are associated based on the playback time of the video to obtain the playback track of the target objects.

10. A method for tracking multiple targets in a video, characterized in that: include: Recording a played video in an input interface of an operation interface, wherein the video contains a plurality of target objects to be tracked; A feature extraction instruction is sensed in the operation interface to extract feature information of each target object in the video; Displaying the classification result on the operation interface, wherein the classification result includes multiple clusters, nodes in the same cluster represent the same target object, and are obtained by aggregating and classifying the nodes in the global feature map, and the global feature map is generated by performing feature similarity composition and spatial similarity composition based on feature information of each target object; The global feature map is generated based on the overlap between multiple feature maps, the multiple feature maps are obtained by performing feature similarity mapping on the feature information, and the overlap is obtained by performing spatial similarity mapping on the feature information.

11. A method for tracking multiple targets in a video, characterized in that: include: Displaying a played video in an operation interface, wherein the video contains a plurality of target objects to be tracked; sensing a feature extraction instruction in the operation interface; In response to the feature extraction instruction, extract feature information of each target object in the video; Displaying the classification result on the operation interface, wherein the classification result includes multiple clusters, nodes in the same cluster represent the same target object, and are obtained by aggregating and classifying the nodes in the global feature map, and the global feature map is generated by performing feature similarity composition and spatial similarity composition based on feature information of each target object; The global feature map is generated based on the overlap between multiple feature maps, the multiple feature maps are obtained by performing feature similarity mapping on the feature information, and the overlap is obtained by performing spatial similarity mapping on the feature information.

12. A method for tracking multiple targets in a video, characterized in that: include: The front-end client uploads a video to be played, wherein the video contains multiple target objects to be tracked; The front-end client transmits the video to the back-end server; The front-end client receives the global feature map returned by the back-end server, wherein the global feature map is generated by the back-end server by performing feature similarity composition and spatial similarity composition based on feature information of each target object in the video; The front-end client aggregates and classifies the nodes in the global feature graph to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object; Among them, the global feature map is generated by the background server based on the overlap between multiple feature maps, the multiple feature maps are obtained by performing feature similarity mapping on the feature information, and the overlap is obtained by performing spatial similarity mapping on the feature information.

13. A method for tracking multiple targets in a video, characterized in that: include: The played video is acquired by calling a first interface, wherein the first interface includes: a first parameter, a parameter value of the first parameter is the video, and there are multiple target objects to be tracked in the video; Extracting feature information of each target object in the video; Based on the feature information of each target object, feature similarity composition and spatial similarity composition are performed to generate a global feature map; Aggregate and classify the nodes in the global feature graph to generate a classification result, wherein the classification result includes multiple clusters, and the nodes in the same cluster represent the same target object; Outputting the classification result by calling a second interface, wherein the second interface includes: a second parameter, a parameter value of the second parameter being the classification result; Among them, feature similarity mapping is performed on the feature information to obtain multiple feature maps, wherein each target object corresponds to each feature map; spatial similarity mapping is performed on the feature information to obtain the overlap between the multiple feature maps; based on the overlap, the global feature map is generated.

14. A processing device for performing multi-target tracking on a video, characterized in that: include: An acquisition unit, used for acquiring a played video, wherein the video contains a plurality of target objects to be tracked; An extraction unit, used to extract feature information of each target object in the video; A composition unit, used for performing feature similarity composition and spatial similarity composition based on the feature information of each target object to generate a global feature map; A classification unit, used for performing aggregation classification on the nodes in the global feature map to generate a classification result, wherein the classification result includes a plurality of clusters, and the nodes in the same cluster represent the same target object; The composition unit is also used to: perform feature similarity composition on the feature information to obtain multiple feature maps, wherein each target object corresponds to each feature map; perform spatial similarity composition on the feature information to obtain the overlap between the multiple feature maps; and generate the global feature map based on the overlap.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 13.

16. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 13 when running.

17. A processing system for multi-target tracking in video, characterized in that: include: processor; A memory is connected to the processor and is used to provide the processor with instructions for processing the following processing steps: obtaining a played video, wherein there are multiple target objects to be tracked in the video; extracting feature information of each target object in the video; performing feature similarity composition and spatial similarity composition based on the feature information of each target object to generate a global feature map; aggregating and classifying nodes in the global feature map to generate a classification result, wherein the classification result includes multiple clusters, and nodes in the same cluster represent the same target object; The memory is also used to provide the processor with instructions for processing the following processing steps: performing feature similarity mapping on the feature information to obtain multiple feature maps, wherein each target object corresponds to each feature map; performing spatial similarity mapping on the feature information to obtain the overlap between the multiple feature maps; Based on the overlap, the global feature map is generated.

Images