Image clustering method, device and storage medium
By constructing a time interaction graph and calculating the clustering probability of nodes, and integrating feature-level and timing-level results, the problem of poor clustering effect in deep clustering networks is solved, and the accuracy and stability of image clustering are improved.
Patent Information
- Application Number
- CN202411402391.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2044-10-09
AI Technical Summary
The existing deep clustering network only considers the similarity of images in feature dimensions, resulting in the clustering effect that needs to be improved.
By extracting image features in image frame sequences, building a time interaction graph, calculating the similarity between nodes and creating edges, selecting historical interaction nodes, calculating cluster probability based on the importance and time decay of nodes, and fusion feature level and Time-series clustering results to improve clustering accuracy.
By capturing node interaction information in the time dimension, the accuracy and stability of clustering are improved and the effect of image clustering is improved.
Smart Images
Figure CN118918350B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image clustering method, device and storage medium. Background Art
[0002] In recent years, with the development of deep learning, image clustering algorithms based on deep learning have also been widely studied and applied.
[0003] The deep clustering network is an unsupervised learning method that combines deep learning and clustering, which can automatically learn feature representation and clustering results. The principle of the deep clustering network is to map the original image data to a low-dimensional space through a deep learning network, and then use a clustering algorithm to cluster the mapped data. It can effectively process nonlinear data and high-dimensional data.
[0004] However, the current deep clustering network only considers the similarity of the images to be clustered in the feature dimension, and its clustering effect needs to be further improved. Summary of the invention
[0005] In order to solve the above technical problems, the present application at least provides an image clustering method, device and storage medium.
[0006] The first aspect of the present application provides an image clustering method, which includes: extracting image features of each image frame in an image frame sequence to obtain an embedded expression corresponding to the image frame; wherein the image frames in the image frame sequence are sorted in chronological order; using the embedded expression corresponding to the image frame as a node, calculating the similarity between each node, creating an edge between nodes whose similarity is greater than a preset similarity threshold, and sorting each node in the image frame time sequence to obtain a time interaction graph; selecting any two nodes in the time interaction graph as a first node and a second node, obtaining a historical interaction node corresponding to the first node at a preset time, wherein the historical interaction node refers to a node in the time interaction graph whose image frame time is before the preset time and has an edge with the first node; based on the historical interaction node corresponding to the first node, calculating the probability that the first node and the second node belong to the same clustering cluster.
[0007] In one embodiment, based on the historical interaction nodes corresponding to the first node, the probability that the first node and the second node belong to the same cluster is calculated, including: obtaining the similarity of the embedded expression between the first node and the historical interaction node, determining the importance of any historical interaction node relative to other historical interaction nodes based on the similarity of each historical interaction node; and calculating the time attenuation between the historical interaction node at a preset moment and the second node, wherein the time interval between the image frame time of the historical interaction node and the preset moment is proportional to the time attenuation; based on the importance of the historical interaction node and the time attenuation between the historical interaction node and the second node, the probability that the first node and the second node belong to the same cluster is calculated.
[0008] In one embodiment, based on the importance of the historical interaction node and the time decay between the historical interaction node and the second node, the probability that the first node and the second node belong to the same cluster is calculated, including: based on the importance of the historical interaction node and the time decay between the historical interaction node and the second node, the influence strength of each historical interaction node on the second node is calculated; wherein the importance is directly proportional to the influence strength, and the time decay is inversely proportional to the influence strength; the influence strength of each historical interaction node on the second node is summed to calculate the influence strength of the first node on the second node; based on the influence strength of the first node on the second node, the probability that the first node and the second node belong to the same cluster is calculated.
[0009] In one embodiment, based on the importance of the historical interaction node and the time decay degree between the historical interaction node and the second node, the probability that the first node and the second node belong to the same cluster is calculated, including: based on the embedded expression of the first node and the embedded expression of the second node, the probability that there is an edge between the first node and the second node is calculated; based on the probability that there is an edge between the first node and the second node, the importance of the historical interaction node, and the time decay degree between the historical interaction node and the second node, the probability that the first node and the second node belong to the same cluster is calculated.
[0010] In one embodiment, the method also includes: obtaining the cluster center corresponding to the current existing cluster cluster; calculating the similarity between the embedded expression corresponding to the node and the cluster center, and obtaining the probability that the node belongs to the existing cluster cluster based on the similarity between the embedded expression and the cluster center, and obtaining a feature-level clustering result; and, using the clustering result of each node obtained based on the time interaction graph as a time series-level clustering result; fusing the feature-level clustering result and the time series-level clustering result to obtain the final clustering result.
[0011] In one embodiment, the probability that a node belongs to an existing cluster is obtained based on the similarity between the embedded expression and the cluster center, including: obtaining all clustered nodes contained in the existing cluster; calculating the similarity between the embedded expression corresponding to the node and each clustered node; combining the similarity between the embedded expression and each clustered node and the similarity with the cluster center, to calculate the probability that the node belongs to the existing cluster.
[0012] In one embodiment, the feature-level clustering results and the time-series-level clustering results are fused to obtain the final clustering results, including: counting the number of edges and the number of nodes in the time interaction graph; calculating the ratio between the number of edges and the number of nodes, and determining the fusion weight based on the ratio; and performing weighted fusion on the feature-level clustering results and the time-series-level clustering results according to the fusion weight to obtain the final clustering result.
[0013] In one embodiment, image features of each image frame in an image frame sequence are extracted to obtain an embedded expression corresponding to the image frame, including: performing target recognition on the image frame to determine the target object contained in the image frame; extracting image features of the target object contained in the image frame to obtain an embedded expression corresponding to the target object.
[0014] The second aspect of the present application provides an image clustering device, which includes: a feature extraction module, which is used to extract image features of each image frame in an image frame sequence to obtain an embedded expression corresponding to the image frame; wherein the image frames in the image frame sequence are sorted in chronological order; an interaction graph construction module, which is used to use the embedded expression corresponding to the image frame as a node, calculate the similarity between each node, create an edge between nodes whose similarity is greater than a preset similarity threshold, and sort each node in the image frame time sequence to obtain a time interaction graph; a historical node determination module, which is used to select any two nodes in the time interaction graph as the first node and the second node, and obtain the historical interaction node corresponding to the first node at a preset time, wherein the historical interaction node refers to a node in the time interaction graph whose image frame time is before the preset time and has an edge with the first node; a temporal clustering module, which is used to calculate the probability that the first node and the second node belong to the same clustering cluster based on the historical interaction node corresponding to the first node.
[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned image clustering method.
[0016] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned image clustering method when the program instructions are executed by a processor.
[0017] The above scheme obtains the embedded expression corresponding to the image frame by extracting the image features of each image frame in the image frame sequence; takes the embedded expression corresponding to the image frame as a node, calculates the similarity between each node, creates an edge between nodes whose similarity is greater than a preset similarity threshold, and sorts each node in the image frame time sequence to obtain a time interaction graph; selects any two nodes in the time interaction graph as the first node and the second node, and obtains the historical interaction node corresponding to the first node at a preset time, and the historical interaction node refers to the node in the time interaction graph whose image frame time is before the preset time and has an edge with the first node; based on the historical interaction node corresponding to the first node, calculates the probability that the first node and the second node belong to the same clustering cluster, records the interaction between nodes in the time dimension through the time interaction graph, and captures the dynamic change information between nodes according to the historical interaction nodes, so as to mine the timing information and improve the accuracy and stability of clustering.
[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.
[0020] Figure 1 It is a schematic diagram of a solution implementation environment provided by an embodiment of the present application;
[0021] Figure 2 is a flow chart of an image clustering method shown in an exemplary embodiment of the present application;
[0022] Figure 3 is a schematic diagram of performing image clustering on an image frame sequence according to an exemplary embodiment of the present application;
[0023] Figure 4 is a block diagram of an image clustering device shown in an exemplary embodiment of the present application;
[0024] Figure 5 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;
[0025] Figure 6 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0026] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.
[0027] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.
[0028] The term "and / or" in this article is only an association information describing the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.
[0029] The image clustering method provided in the embodiment of the present application is described below.
[0030] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 are connected to each other in communication.
[0031] The number of the terminal 110 may be one or more. The terminal 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.
[0032] Server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.
[0033] In one example, the server 120 may perform image clustering processing on the image frame sequence obtained from the terminal 110 to obtain a clustering result. Of course, the server 120 may store the clustering result locally, transmit it back to the terminal 110, or transmit it to other terminals.
[0034] In the image clustering method provided in the embodiment of the present application, the execution subject of each step can be the terminal 110, or the server 120, or the terminal 110 and the server 120 can cooperate with each other to execute, that is, part of the steps of the method are executed by the terminal 110 and the other steps are executed by the server 120.
[0035] See also Figure 2 , Figure 2 is a flowchart of an image clustering method shown in an exemplary embodiment of the present application. The image clustering method can be applied to Figure 1 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.
[0036] like Figure 2 As shown, the image clustering method at least includes steps S210 to S240, which are described in detail as follows:
[0037] Step S210: extracting image features of each image frame in the image frame sequence to obtain an embedded expression corresponding to the image frame; wherein the image frames in the image frame sequence are sorted in chronological order.
[0038] An image frame sequence is acquired, where the image frame sequence includes a plurality of image frames that are continuously ordered in a time dimension, and each image frame corresponds to an image frame time.
[0039] The image frame sequence may be extracted from a video data stream, or may be obtained by sorting a plurality of image data containing image frame time, which is not limited in the present application.
[0040] The image features of the image frames are extracted using a pre-trained image encoding network to obtain the corresponding embedded expression for each image frame.
[0041] For example, the entire image frame may be subjected to image feature extraction to obtain an embedded expression; or a key image region in the image frame may be identified, and image feature extraction may be performed on the key image region to obtain an embedded expression. The specific extraction method may be flexibly selected according to the actual application scenario, and this application does not limit this.
[0042] Step S220: Take the embedded expression corresponding to the image frame as a node, calculate the similarity between each node, create edges between nodes whose similarity is greater than a preset similarity threshold, and sort each node in the time sequence of the image frame to obtain a time interaction graph.
[0043] Each embedded expression is regarded as a node, and each node corresponds to an image frame time.
[0044] Then, the similarity between each node is calculated, and edges are created between nodes whose similarity is greater than a preset similarity threshold. For example, the Euclidean distance, cosine distance, etc. between the embedded expressions are calculated to obtain the similarity between each node, and it is determined whether the similarity between the nodes is greater than a preset similarity threshold. If so, edges are created between the nodes.
[0045] Among them, the preset similarity threshold can be pre-set or flexibly calculated. For example, the time interval between the image frame times corresponding to two nodes is calculated, and the similarity threshold is set according to the time interval. For example, the shorter the time interval, the larger the similarity threshold, and the longer the time interval, the smaller the similarity threshold. For example, the similarity threshold can also be set based on the image quality of the image frame corresponding to the node, and the present application does not limit this.
[0046] Exemplarily, the similarity between the current node and all nodes may be calculated to create an edge between the current node and a node whose similarity is greater than a preset similarity threshold.
[0047] Exemplarily, the adjacent image frames corresponding to the image frame to which the current node belongs may be determined, the nodes contained in the adjacent image frames may be obtained, the similarity between the current node and the nodes contained in the adjacent image frames may be calculated, and an edge may be created between the current node and the nodes whose similarity is greater than a preset similarity threshold. The adjacent image frames may be image frames whose time interval between image frames is less than a preset interval, or may be n frames adjacent to the image frame to which the current node belongs in the image frame sequence.
[0048] It should be noted that the edge construction method can be flexibly selected according to the amount of available resources of the execution entity, clustering accuracy, etc., and this application does not limit this.
[0049] For example, please refer to the following Table 1, which shows some nodes and the edges between nodes in the time interaction graph:
[0050]
[0051] Table 1
[0052] As shown in Table 1, nodes include 1, 2, and 3. A row in the table represents an edge. The time can be the image frame time corresponding to any of the two nodes corresponding to the edge, or the image frame time corresponding to the last created node of the two nodes corresponding to the edge.
[0053] The image features of each image frame in the image frame sequence are used as nodes, edges are constructed according to node similarity, and each node is sorted in the time order of the image frames to obtain a temporal interaction graph.
[0054] Step S230: Select any two nodes in the time interaction graph as the first node and the second node, and obtain the historical interaction node corresponding to the first node at the preset time. The historical interaction node refers to the node in the time interaction graph whose image frame time is before the preset time and has an edge with the first node.
[0055] Based on the time interaction graph, each node is clustered.
[0056] Specifically, for any two nodes, take them as the first node and the second node, and obtain the historical interaction node corresponding to the first node at the preset time. The historical interaction node refers to the node whose image frame time in the time interaction graph is before the preset time and has an edge with the first node. The preset time can be the current time or other preset time, which is not limited in this application.
[0057] Step S240: Based on the historical interaction nodes corresponding to the first node, calculate the probability that the first node and the second node belong to the same cluster.
[0058] According to the historical interaction nodes corresponding to the first node, the probability that the first node and the second node belong to the same cluster is calculated.
[0059] Exemplarily, a preset formula may be used to substitute the image frame time, embedded expression, etc. of the historical interaction node into the preset formula to obtain the probability that the first node and the second node belong to the same cluster.
[0060] Exemplarily, a graph clustering network may be pre-trained, and the historical interaction node and the second node corresponding to the first node in the time interaction graph may be input into the graph clustering network to obtain the probability that the first node and the second node output by the graph clustering network belong to the same cluster cluster.
[0061] Optionally, in addition to referring to the historical interaction nodes of the first node, the historical interaction nodes of the second node may be further obtained, and the probability that the first node and the second node belong to the same cluster may be comprehensively calculated by combining the historical interaction nodes of the first node and the historical interaction nodes of the second node.
[0062] In the time interaction graph, the interaction between nodes in the historical time period affects the interaction between nodes in the subsequent time period, and this influence will decay over time. Therefore, the present application uses a time interaction graph to record the interaction between nodes in the time dimension, captures the dynamic change information between nodes based on the historical interaction nodes, and improves the accuracy and stability of clustering.
[0063] Next, some embodiments of the present application are described in detail by way of example:
[0064] In some embodiments, step S210 extracts image features of each image frame in the image frame sequence to obtain an embedded expression corresponding to the image frame, including: performing target recognition on the image frame to determine the target object contained in the image frame; extracting image features of the target object contained in the image frame to obtain an embedded expression corresponding to the target object.
[0065] The target objects include but are not limited to vehicles, animals, commodities, etc.
[0066] The target object contained in the image frame is identified to obtain the image area corresponding to the target object, and then image features are extracted for the image area corresponding to the target object to obtain the embedded expression corresponding to the target object.
[0067] An image frame can obtain the corresponding embedded expressions of one or more target objects.
[0068] The embedded expression corresponding to each target object extracted from the image frame sequence is taken as a node, and the similarity between each node is calculated. Edges are created between nodes whose similarity is greater than the preset similarity threshold, and each node is sorted in the time sequence of the image frame to obtain a time interaction graph.
[0069] That is, the image clustering method of the present application can not only cluster images directly, but also cluster target objects in combination with the target recognition algorithm.
[0070] After obtaining the time interaction graph, time series information mining is performed on the time interaction graph to perform node clustering based on the mined time series information.
[0071] Specifically, time series information mining is performed based on the historical interaction nodes of the nodes to obtain the interaction information of the nodes in the historical time period, and then predict the interaction of the nodes in the subsequent time period.
[0072] In some embodiments, in step S240, based on the historical interaction nodes corresponding to the first node, the probability that the first node and the second node belong to the same cluster is calculated, including: obtaining the similarity of the embedded expression between the first node and the historical interaction node, and determining the importance of any historical interaction node relative to other historical interaction nodes based on the similarity of each historical interaction node; and calculating the time attenuation between the historical interaction node at a preset moment and the second node, wherein the time interval between the image frame time of the historical interaction node and the preset moment is proportional to the time attenuation; based on the importance of the historical interaction node and the time attenuation between the historical interaction node and the second node, the probability that the first node and the second node belong to the same cluster is calculated.
[0073] According to the similarity of the embedded expression between each historical interaction node and the first node, the importance of any historical interaction node relative to other historical interaction nodes is determined, and the importance is used to evaluate the importance of the historical interaction node among all historical interaction nodes of the first node.
[0074] Specifically, the higher the similarity of the embedded expression between the historical interaction node and the first node, the higher its importance relative to other historical interaction nodes.
[0075] Furthermore, the time attenuation of the historical interaction node between the preset moment and the second node is calculated. The influence of the historical interaction node on the second node decays over time. The larger the time interval between the image frame time of the historical interaction node and the preset moment, the smaller the influence of the historical interaction node on the second node, and thus the greater the time attenuation.
[0076] According to the importance of the historical interaction node and the time decay degree between the historical interaction node and the second node, the probability that the first node and the second node belong to the same cluster is calculated.
[0077] For example, the importance of the historical interaction node and the time attenuation between the historical interaction node and the second node are substituted into a preset formula to calculate the probability that the first node and the second node belong to the same cluster.
[0078] Of course, we can further consider the similarity between the embedded expression of the historical interaction node and the embedded expression of the second node, the similarity between the embedded expression of the first node and the embedded expression of the second node, the historical interaction nodes of the second node, etc. This application does not limit this.
[0079] Exemplarily, based on the importance of the historical interaction node and the time decay between the historical interaction node and the second node, the influence strength of each historical interaction node on the second node is calculated; wherein, the importance is directly proportional to the influence strength, and the time decay is inversely proportional to the influence strength; the influence strength of each historical interaction node on the second node is summed to calculate the influence strength of the first node on the second node; based on the influence strength of the first node on the second node, the probability that the first node and the second node belong to the same cluster is calculated.
[0080] For example, the importance of the historical interaction node and the time decay between the historical interaction node and the second node are substituted into a preset formula to calculate the influence strength of the historical interaction node on the second node. For example, the method of calculating the influence strength of the historical interaction node on the second node can refer to the following formula 1:
[0081] (Formula 1)
[0082] Among them, h is the historical interaction node, y is the second node, is the influence strength of the historical interaction node h on the second node y; is the importance of the historical interaction node h in all historical interaction nodes corresponding to the first node; The low-dimensional representation of the historical interaction node h and the second node y, respectively. Represents the time decay between the historical interaction node and the second node.
[0083] The influence strength of each historical interaction node on the second node is summed to calculate the influence strength of the first node on the second node. The relevant calculation formula can be expressed as the following formula 2:
[0084] (Formula 2)
[0085] in, is the influence strength of the first node x on the second node y at the preset time t; is the image frame time corresponding to the historical interaction node h; is the influence strength of the historical interaction node h on the second node y; k is the kernel function, which can be the time decay effect written in the form of an exponential function.
[0086] Optionally, the probability of the existence of an edge between the first node and the second node can be calculated based on the embedding expression of the first node and the embedding expression of the second node; the probability that the first node and the second node belong to the same cluster can be calculated based on the probability of the existence of an edge between the first node and the second node, the importance of the historical interaction node, and the time decay between the historical interaction node and the second node.
[0087] For example, based on the probability of an edge between the first node and the second node, the importance of the historical interaction node, and the time decay between the historical interaction node and the second node, the influence strength of the first node on the second node is calculated. The relevant calculation formula can be expressed as the following formula 3:
[0088] (Formula 3)
[0089] in, is the probability that there is an edge between the first node and the second node at the preset time t, which can be directly calculated according to the similarity between the embedding expression of the first node and the embedding expression of the second node; the interpretation of other symbols can be found in the above formula 2 and will not be repeated here.
[0090] According to the influence strength of the first node on the second node, the probability that the first node and the second node belong to the same cluster is calculated. For example, the influence strength of the first node on the second node can be directly normalized to the range of 0 to 1 to obtain the probability that the first node and the second node belong to the same cluster.
[0091] Of course, in addition to the above-mentioned exemplary embodiments, the allocation probability between the node and each cluster may also be calculated in other ways, which is not limited in the present application.
[0092] In some embodiments, in addition to performing clustering calculations based on the timing information of each node in the time interaction graph, clustering calculations can also be performed based on the similarity between the embedded expressions of the nodes themselves, specifically including: obtaining the cluster center corresponding to the current existing cluster cluster; calculating the similarity between the embedded expression corresponding to the node and the cluster center, and obtaining the probability that the node belongs to the existing cluster cluster based on the similarity between the embedded expression and the cluster center, to obtain a feature-level clustering result; and, using the clustering results of each node obtained based on the time interaction graph as the time-series level clustering result; fusing the feature-level clustering result and the time-series level clustering result to obtain the final clustering result.
[0093] When clustering begins, a preset number of nodes may be randomly selected as initial clusters, and the cluster center may be the average value of the embedding vectors of all nodes in the cluster.
[0094] Each cycle calculates the similarity between the embedding expression corresponding to the unclustered node and the cluster center. The higher the similarity, the greater the probability that the unclustered node belongs to the cluster. Through multiple cycles, the cluster to which each node belongs is determined, and the cluster center corresponding to the cluster is updated after each cycle.
[0095] Specifically, the probability that a node belongs to an existing cluster is obtained based on the similarity between the embedded expression and the cluster center, including: obtaining all clustered nodes contained in the existing cluster; calculating the similarity between the embedded expression corresponding to the node and each clustered node; combining the similarity between the embedded expression and each clustered node and the similarity with the cluster center, to calculate the probability that the node belongs to the existing cluster.
[0096] For example, based on the Student's t distribution, the similarity between each node and each cluster center is measured, and then the similarity between the embedded expression and each clustered node and the similarity between the embedded expression and the cluster center is combined to calculate the probability that the node belongs to the existing cluster. The relevant formula can be found in the following formula 4:
[0097] (Formula 4)
[0098] in, Represents the probability that node x belongs to cluster j, that is, the feature-level clustering result; is the embedding expression of node x; is the cluster center of cluster j; is any node in cluster j; For Node Embedded expression of is the degree of freedom of the Student's t distribution, which can be 1.
[0099] According to step S210 to step S240, the time series level clustering result is obtained, and the feature level clustering result and the time series level clustering result are fused to obtain the final clustering result.
[0100] For example, for a node to be classified, the feature-level clustering result and the time-series-level clustering result of the node are obtained. Both the feature-level clustering result and the time-series-level clustering result contain the probability that the node to be classified belongs to each clustering cluster. The clustering cluster with the largest probability value is selected from the feature-level clustering result and the time-series-level clustering result as the final clustering result.
[0101] For another example, the feature-level clustering results and the time-series-level clustering results are weightedly fused to obtain the fusion probability that the node to be classified belongs to each cluster, and the cluster with the largest fusion probability is selected as the final clustering result. The weight parameter for weighted fusion of the feature-level clustering results and the time-series-level clustering results can be pre-set or flexibly calculated.
[0102] Exemplarily, the number of edges and the number of nodes in the time interaction graph are counted; the ratio between the number of edges and the number of nodes is calculated, and the fusion weight is determined based on the ratio; according to the fusion weight, the feature-level clustering results and the time-series-level clustering results are weightedly fused to obtain the final clustering result.
[0103] For example, according to the ratio between the number of edges and the number of nodes in the time interaction graph, the more edges there are, the more frequent the interactions between nodes in the time interaction graph are, and the more time series information that can be mined, the higher the accuracy of clustering based on the time interaction graph. Therefore, the greater the weight of the time series level clustering result in weighted fusion; conversely, the smaller the weight of the time series level clustering result in weighted fusion.
[0104] The fusion weight is determined according to the ratio, and the feature-level clustering results and the time-series-level clustering results are weightedly fused according to the fusion weight to obtain the final clustering result, thereby improving the accuracy of the clustering result.
[0105] Some of the above steps can be implemented based on a pre-trained clustering model. The training process of the clustering model is explained below:
[0106] Construct a clustering model, which is used to extract the embedded expression of the image or perform further clustering calculations. The clustering model can be implemented based on a network architecture such as a convolutional neural network (CNN) or a recurrent neural network (RNN), and this application does not limit this.
[0107] A time interaction graph between nodes corresponding to the sample image frame sequence is constructed, and a clustering model is trained through the time interaction graph corresponding to the sample image frame sequence to obtain a trained clustering model. The clustering model is implemented based on a deep clustering network.
[0108] In the process of training the clustering model according to the time interaction graph corresponding to the sample image frame sequence, the probability of nodes belonging to the same cluster is controlled as large as possible through the loss function, and the probability of nodes not belonging to the same cluster is controlled as small as possible. The loss function can be the following formula 5:
[0109] (Formula 5)
[0110] in, The loss value for model training; is the predicted probability output by the clustering model that nodes x and y belong to the same cluster at time t; P(x) is the negative sample set, which contains multiple negative samples n; It is the predicted probability output by the clustering model that nodes x and n belong to the same cluster at time t; is the summation symbol.
[0111] Of course, in addition to calculating the model training loss based on the time-level clustering results, you can also calculate the model training loss based on the feature-level clustering results. Combine the model training loss corresponding to the time-level clustering results and the model training loss corresponding to the feature-level clustering results, and adjust the network parameters of the clustering model until a trained clustering model is obtained.
[0112] During the model optimization process, through continuous iterative training, the embedding expression corresponding to the node is made closer to the cluster center of its corresponding cluster cluster, so as to encourage the embedding expression distribution of the node to be as close to the target distribution as possible to make the embedding expression of the node more suitable for clustering.
[0113] It should be noted that in the time interaction graph, since there is a continuous interaction sequence, the distribution of nodes in each interaction needs to be calculated in batches. If a node has multiple interactions, its distribution will be calculated multiple times, so that the characteristics of important nodes can be calibrated multiple times to improve the clustering performance and more effectively capture the information of dynamic node interactions.
[0114] During the training process, the time interaction graph needs to be sliced into multiple training batches to accommodate big data computing. The size of the training batch can be adjusted according to the size of the available memory during the model training process. However, due to the problem of batch training in the time interaction graph, the reconstruction technology based on the adjacency matrix is difficult to apply. Therefore, node clustering can be assisted by simply simulating the adjacency relationship without directly using the adjacency matrix.
[0115] For example, by using cosine similarity to simulate the adjacency relationship between nodes, the interaction relationship between nodes can be reconstructed. In each batch, if the second node and each historical interaction node in the historical interaction node set are to establish a connection edge with the first node, the value of their adjacency relationship can be set to 1, indicating that there is an adjacency relationship between them, and a positive sample node pair is obtained. At the same time, negative sample nodes are introduced into the loss function. These nodes have no edges connected to the first node in the real graph, that is, their adjacency relationship values are 0. In this way, the adjacency relationship can be made closer to the actual situation. In addition, by using cosine similarity, the embedding expression of the positive sample node pair is constrained to be close to 1, and the embedding expression of the negative sample node pair is constrained to be close to 0, so that the embedding expression of the node meets the above constraints during the model training process.
[0116] In some embodiments, see Figure 3 , Figure 3 FIG. 1 is a schematic diagram showing an exemplary embodiment of the present application for performing image clustering on an image frame sequence, as shown in FIG. Figure 3 As shown, a continuous sequence of image frames within a time period is obtained, and target object recognition is performed on each image frame to extract the embedded expression corresponding to the target object. The embedded expression corresponding to the extracted target object is used as a node, and based on the similarity between the nodes, edges are created between nodes whose similarity is greater than a preset similarity threshold, and each node is sorted in the order of the image frame time according to the time of the image frame to which each target object in the image frame sequence belongs, and a time interaction graph is constructed. Timing information mining is performed according to the time interaction graph to cluster the timing information obtained by mining to obtain a clustering result. The clustering results are post-processed, and the post-processing includes but is not limited to the merging of similar clusters, the removal of erroneous nodes in the clusters, etc., which is not limited in this application.
[0117] The image clustering method provided by the present application obtains the embedded expression corresponding to the image frame by extracting the image features of each image frame in the image frame sequence; takes the embedded expression corresponding to the image frame as a node, calculates the similarity between each node, creates an edge between nodes whose similarity is greater than a preset similarity threshold, and sorts each node in the image frame time sequence to obtain a time interaction graph; selects any two nodes in the time interaction graph as the first node and the second node, and obtains the historical interaction node corresponding to the first node at a preset time, where the historical interaction node refers to the node in the time interaction graph whose image frame time is before the preset time and has an edge with the first node; based on the historical interaction node corresponding to the first node, calculates the probability that the first node and the second node belong to the same clustering cluster, records the interaction between nodes in the time dimension through the time interaction graph, and captures the dynamic change information between nodes according to the historical interaction nodes, so as to mine the timing information and improve the accuracy and stability of clustering.
[0118] Figure 4 FIG. 1 is a block diagram of an image clustering device shown in an exemplary embodiment of the present application. Figure 4 As shown, the exemplary image clustering device 400 includes: a feature extraction module 410, an interaction graph construction module 420, a historical node determination module 430 and a temporal clustering module 440. Specifically:
[0119] A feature extraction module 410 is used to extract image features of each image frame in the image frame sequence to obtain an embedded expression corresponding to the image frame; wherein the image frames in the image frame sequence are sorted in chronological order;
[0120] The interaction graph construction module 420 is used to use the embedded expressions corresponding to the image frames as nodes, calculate the similarity between the nodes, create edges between the nodes whose similarity is greater than a preset similarity threshold, and sort each node according to the time sequence of the image frames to obtain a time interaction graph;
[0121] A historical node determination module 430 is used to select any two nodes in the time interaction graph as the first node and the second node, and obtain the historical interaction node corresponding to the first node at a preset time. The historical interaction node refers to a node in the time interaction graph whose image frame time is before the preset time and has an edge with the first node;
[0122] The time series clustering module 440 is used to calculate the probability that the first node and the second node belong to the same cluster based on the historical interaction nodes corresponding to the first node.
[0123] It should be noted that the image clustering device provided in the above embodiment and the image clustering method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In practical applications, the image clustering device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.
[0124] See also Figure 5 , Figure 5 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 500 includes a memory 501 and a processor 502, and the processor 502 is used to execute program instructions stored in the memory 501 to implement the steps in any of the above-mentioned image clustering method embodiments. In a specific implementation scenario, the electronic device 500 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 500 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.
[0125] Specifically, the processor 502 is used to control itself and the memory 501 to implement the steps in any of the above-mentioned image clustering method embodiments. The processor 502 can also be called a central processing unit (CPU). The processor 502 may be an integrated circuit chip with signal processing capabilities. The processor 502 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 502 can be implemented by an integrated circuit chip.
[0126] See also Figure 6 , Figure 6 The computer-readable storage medium 600 stores program instructions 610 that can be executed by a processor, and the program instructions 610 are used to implement the steps in any of the above-mentioned image clustering method embodiments.
[0127] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0128] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.
[0129] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0130] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.
Claims
1. An image clustering method, characterized in that: include: Extracting image features of each image frame in the image frame sequence to obtain an embedded expression corresponding to the image frame; wherein the image frames in the image frame sequence are sorted in chronological order; The embedded expression corresponding to the image frame is used as a node, the similarity between each node is calculated, edges are created between nodes whose similarity is greater than a preset similarity threshold, and each node is sorted according to the time sequence of the image frame to obtain a time interaction graph; Select any two nodes in the time interaction graph as the first node and the second node, and obtain the historical interaction node corresponding to the first node at a preset time, wherein the historical interaction node refers to a node in the time interaction graph whose image frame time is before the preset time and has an edge with the first node; Obtaining the similarity of the embedded expression between the first node and the historical interaction node, determining the importance of any historical interaction node relative to other historical interaction nodes based on the similarity of each historical interaction node; and calculating the time attenuation between the historical interaction node at the preset moment and the second node, wherein the time interval between the image frame time of the historical interaction node and the preset moment is proportional to the time attenuation; Based on the importance of the historical interaction node and the time decay degree between the historical interaction node and the second node, the probability that the first node and the second node belong to the same cluster is calculated.
2. The method according to claim 1, characterized in that: The calculating, based on the importance of the historical interaction node and the time decay degree between the historical interaction node and the second node, the probability that the first node and the second node belong to the same cluster includes: Based on the importance of the historical interaction node and the time decay between the historical interaction node and the second node, calculating the influence strength of each historical interaction node on the second node; wherein the importance is directly proportional to the influence strength, and the time decay is inversely proportional to the influence strength; Sum the influence strength of each historical interaction node on the second node, and calculate the influence strength of the first node on the second node; Based on the influence strength of the first node on the second node, a probability that the first node and the second node belong to the same cluster is calculated.
3. The method according to claim 1, characterized in that The calculating, based on the importance of the historical interaction node and the time decay degree between the historical interaction node and the second node, the probability that the first node and the second node belong to the same cluster includes: Calculating a probability that an edge exists between the first node and the second node based on the embedding expression of the first node and the embedding expression of the second node; Based on the probability of an edge existing between the first node and the second node, the importance of the historical interaction node, and the time decay degree between the historical interaction node and the second node, the probability that the first node and the second node belong to the same cluster is calculated.
4. The method according to claim 1, characterized in that: The method further comprises: Get the cluster center corresponding to the current existing cluster; Calculating the similarity between the embedded expression corresponding to the node and the cluster center, obtaining the probability that the node belongs to the existing cluster based on the similarity between the embedded expression and the cluster center, and obtaining a feature-level clustering result; and The clustering result of each node obtained based on the time interaction graph is used as the time series clustering result; The feature-level clustering result and the time-series-level clustering result are integrated to obtain a final clustering result.
5. The method according to claim 4, characterized in that The obtaining the probability that the node belongs to the existing cluster based on the similarity between the embedded expression and the cluster center includes: Obtaining all clustered nodes contained in the existing cluster; Calculate the similarity between the embedded expression corresponding to the node and each clustered node; The probability that the node belongs to the existing cluster is calculated by combining the similarity between the embedded expression and each clustered node and the similarity between the embedded expression and the cluster center.
6. The method according to claim 4, characterized in that The fusing of the feature-level clustering result and the time-series-level clustering result to obtain the final clustering result includes: Counting the number of edges and the number of nodes in the time interaction graph; calculating a ratio between the number of the edges and the number of the nodes, and determining a fusion weight based on the ratio; According to the fusion weight, the feature-level clustering result and the time-series-level clustering result are weightedly fused to obtain a final clustering result.
7. The method according to claim 1, characterized in that The step of extracting the image features of each image frame in the image frame sequence to obtain an embedded expression corresponding to the image frame includes: Performing target recognition on the image frame to determine the target object contained in the image frame; Image features of the target object contained in the image frame are extracted to obtain an embedded expression corresponding to the target object.
8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, and the processor is used to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1 to 7.