Clustering method based on multimode information, server, and storage medium

By generating a posterior matching probability function in a multimodal clustering method through self-learning, the problems of low accuracy and recall in existing technologies are solved, and more efficient multimodal clustering results are achieved.

WO2024230437A1PCT designated stage expired Publication Date: 2024-11-14CLOUD INTELLIGENCE ASSETS HOLDING (SINGAPORE) PTE LTD

Patent Information

Application Number
PCT/CN2024/088374
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-05
Filing Date
2024-04-17
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Existing multimodal clustering methods have shortcomings in terms of accuracy and recall, mainly because they rely on Euclidean distance or cosine similarity for clustering, resulting in poor clustering performance.

Method used

By generating posterior matching probability functions for each modality through self-learning, and using multimodal feature information to calculate the posterior matching probability of the target object, clustering is performed.

Benefits of technology

It improves the accuracy and recall of clustering, and can more accurately measure the similarity between target objects, making it suitable for clustering tasks in different data distribution scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024088374_14112024_PF_FP_ABST
    Figure CN2024088374_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application provides a clustering method based on multimode information, a server, and a storage medium. According to the method of the present application, a posterior matching probability function in each single mode is generated by means of self-learning according to multimode feature information of target objects to be clustered, labeling information does not need to be used, and the posterior matching probability function is generated by means of self-learning; a posterior matching probability of any two target objects in a multimode is calculated according to the posterior matching probability function in each single mode; the target objects are clustered according to the posterior matching probability of any two target objects in the multimode to obtain a clustering result; by using the posterior matching probability function generated by self-learning, on the basis of the multimode feature information of the target objects, a posterior matching probability of the target objects belonging to a same cluster is calculated as a measure of the similarity between the target objects, so that the similarity between the target objects can be measured more accurately, clustering is carried out on the basis of the posterior matching probability between the target objects, and thus the clustering accuracy and the recall rate can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Clustering method, server and storage medium based on multimodal information

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 5, 2023, with application number 202310516381.8 and invention name “Clustering method, server and storage medium based on multimodal information”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to computer technology, and in particular to a clustering method, server, and storage medium based on multimodal information. Background Art

[0003] With the development of cloud computing and mobile internet, various service platforms have generated massive amounts of data. This data lacks multimodality. Different objects contain data from different modalities, and different modalities have different attributes. Leveraging the complementarity between modalities and eliminating redundancy between them to learn better feature attributes is crucial. Clustering is one important application of this.

[0004] Currently, mainstream multimodal clustering methods usually perform clustering based on Euclidean distance or cosine similarity between features, and the clustering accuracy and recall rate are low.

[0005] Summary of the Invention

[0006] The present application provides a clustering method, server and storage medium based on multimodal information to solve the problems of low clustering accuracy and recall rate of mainstream multimodal clustering methods.

[0007] In a first aspect, the present application provides a clustering method based on multimodal information, comprising: obtaining multimodal feature information of a target object to be clustered; self-learning to generate a posterior matching probability function under each modality based on the multimodal feature information of the target object; calculating the posterior matching probability of any two target objects under multimodality based on the posterior matching probability function under each modality; clustering the target objects based on the posterior matching probability of any two target objects under multimodality to generate a clustering result of the target objects.

[0008] In a second aspect, the present application provides a clustering method based on multimodal information, comprising: obtaining multimodal feature information of an image to be clustered, the multimodal feature information including geographic location information and feature information of at least one modality of the contained object; based on the multimodal feature information of the image, self-learning to generate a posterior matching probability function under each modality; based on the posterior matching probability function under each modality, calculating the posterior matching probability of any two images under multimodality, wherein the posterior matching probability of two images under multimodality is the probability that the two images contain the same object; based on the posterior matching probability of any two images under multimodality, clustering the images containing the same object into one category to obtain a clustering result of the image.

[0009] In the third aspect, the present application provides a clustering method based on multimodal information, which is applied to an end-side device, including: obtaining multimodal feature information of the target object to be clustered; sending the multimodal feature information of the target object to be clustered to a server; receiving the clustering result of the target object sent by the server, and the clustering result of the target object is generated by the method described in the first aspect; according to the clustering result, executing at least one of the following processing logics: information push, information search, and data classification.

[0010] In a fourth aspect, the present application provides a server comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect or the second aspect.

[0011] In a fifth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0013] FIG1 is a diagram of an exemplary system architecture based on multimodal information clustering applicable to the present application;

[0014] FIG2 is a flow chart of a clustering method based on multimodal information provided by an exemplary embodiment of the present application;

[0015] FIG3 is a flow chart of a clustering method based on multimodal information provided by another exemplary embodiment of the present application;

[0016] FIG4 is a framework diagram of determining a multimodal posterior matching probability of a target object based on multimodal information according to an exemplary embodiment of the present application;

[0017] FIG5 is a flow chart of a clustering method based on multimodal information provided by another exemplary embodiment of the present application;

[0018] FIG6 is an interactive diagram of a clustering method based on multimodal information provided by an exemplary embodiment of the present application;

[0019] FIG7 is a schematic structural diagram of a clustering device based on multimodal information provided by an exemplary embodiment of the present application;

[0020] FIG8 is a schematic diagram of the structure of a server provided in an embodiment of the present application.

[0021] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0023] First, let’s explain the terms involved in this application:

[0024] Modality: Each source or form of information can be called a modality. For example, humans have touch, hearing, vision, smell, and taste, which can serve as multiple different modalities. Information media include voice, video, and text, and different media can serve as multiple different modalities. Various sensors, such as radar, infrared, and accelerometers, can serve as different modalities depending on the source device. For example, multimodal feature information about a person can include visual information such as various biometrics, basic biographical information, and geographic information such as time and space. In e-commerce scenarios, multimodal feature information about an item can include image features, text descriptions, item categories, sales figures, and reviews.

[0025] Clustering: The process of dividing a collection of physical or abstract objects into clusters of similar objects is called clustering. A cluster generated by clustering is a collection of objects that are similar to objects in the same cluster and dissimilar to objects in other clusters.

[0026] Multimodal clustering: Clustering objects with multimodal information. In one example scenario, images collected from multiple locations are grouped into the same category based on the multimodal information of the clustered objects. In e-commerce scenarios, items from different stores are grouped into the same category based on their uniqueness, using their multimodal information.

[0027] Missing multimodality: Information on one or more modalities is missing for some objects in a multimodal object set. For example, some objects have no associated people, and some items have no associated reviews.

[0028] Euclidean distance: refers to the distance between two points in Euclidean space.

[0029] Cosine similarity: Measures the similarity between two vectors by calculating the cosine of the angle between them.

[0030] Matching probability: the probability that two objects belong to the same cluster category (cluster).

[0031] Conditional independence: If P(X,Y|Z)=P(X|Z)P(Y|Z), or equivalently P(X|Y,Z)=P(X|Z), then events X and Y are said to be conditionally independent with respect to a given event Z. That is, when Z occurs, whether X occurs or not is irrelevant to whether Y occurs or not. In this application, multiple modalities need to satisfy conditional independence with respect to given category labels and limited scenarios. For example, different biometric similarities satisfy conditional independence in determining whether the same person is the same, and biometric similarity and spatial distance satisfy conditional independence in determining whether the same person is the same.

[0032] The posterior matching probability under a single modality refers to the posterior matching probability determined based on the feature information of a single modality of the object.

[0033] The posterior matching probability under multimodality refers to the posterior matching probability between objects determined based on the multimodal feature information of the objects.

[0034] The posterior matching probability function under a single modality refers to the posterior matching probability function based on the feature information of a single modality.

[0035] Multimodal posterior matching probability function: refers to the posterior matching probability function based on the multimodal feature information of the object, including the cross posterior matching probability function under two modalities and the mixed posterior matching probability function under three or more modalities.

[0036] With the development of cloud computing and mobile internet, various service platforms have generated massive amounts of data. For example, large-scale short video data captured and uploaded by mobile terminals can be clustered online by extracting video content or multimodal information such as images, audio, and text. This can be used for applications such as clustering and searching similar videos. Another example is that people's expressions on social networks are becoming increasingly diverse, with people expressing their emotions and opinions through images, text, and video. This multimodal information, such as images, videos, and text, can be used to cluster dynamic content and dynamic emotions on social networks, recommending similar dynamic content to users. Another example is that more and more shopping platforms offer search services. Different merchants on these platforms may have different descriptions of the same item, including videos, images, text, and prices. By clustering similar items from different merchants together through multimodal information, users can be provided with search services that combine video, image, and text.

[0037] Currently, mainstream multimodal clustering methods typically cluster based on Euclidean distance or cosine similarity between features, resulting in low accuracy and recall. Some multimodal clustering methods use the Euclidean distance or cosine similarity of key features as the similarity between objects, and use feature information from other modalities as rules to optimize similarity between objects. For example, if two images are collected from the same street, the similarity increases by 5%, and if they are collected from the same area, the similarity increases by 2%. These rules are not flexible enough, affecting clustering effectiveness.

[0038] In response to the problem of low clustering accuracy and recall rate of current mainstream multimodal clustering methods, the present application provides a clustering method based on multimodal information. According to the multimodal feature information of the target object to be clustered, the posterior matching probability function under each single modality is self-learned and generated, and the posterior matching probability function under each modality can be automatically learned without using annotation information; according to the posterior matching probability function under each single modality, the posterior matching probability of any two target objects under multimodality is calculated; according to the posterior matching probability of any two target objects under multimodality, the target objects are clustered to generate clustering results of the target objects. By using the posterior matching probability function generated by self-learning, the posterior matching probability that the target objects belong to the same cluster is calculated based on the multimodal feature information of the target objects. This is used as a distance measure of the similarity between target objects. Compared with the Euclidean distance or cosine similarity between features used in mainstream methods, the use of the posterior matching probability between target objects can more accurately measure the similarity between target objects. Clustering based on the posterior matching probability between target objects can greatly improve the accuracy and recall of clustering.

[0039] The clustering method based on multimodal information provided in this application can self-learn to generate the posterior matching probability function under each single modality, so that it can be quickly and unbiasedly applied to different data distribution scenarios, such as the aggregation and organization of image libraries in different regions, the clustering of similar products, etc.

[0040] FIG1 is an exemplary system architecture diagram based on multimodal information clustering applicable to the present application. As shown in FIG1 , the system architecture may specifically include a server, a terminal device, and a data service device.

[0041] The server may be a local server or a server cluster in the cloud. A communication link is provided between the server and each end-side device, enabling communication between the server and each end-side device.

[0042] The terminal-side device refers to an electronic device that provides the server with multimodal feature information of the target object to be clustered. Specifically, it can be a hardware device with network communication function, computing function and information display function, including but not limited to smart terminals, tablet computers, personal computers and other terminal devices used by users, Internet of Things devices, service platforms or institutions' servers, etc.

[0043] A data service device can be a device capable of providing multimodal information to a server. Specifically, the data service device can be a data acquisition device, or a server capable of collecting data in one or more modalities collected by the data acquisition device. The server can obtain multimodal information of the target object from one or more data service devices, perform data preprocessing and feature extraction, and obtain multimodal feature information of the target object.

[0044] The end-side device sends a clustering request / instruction for the target object to the server, or calls an application programming interface provided by the server, to trigger the server to perform clustering based on the multimodal information of the target object. In response to the clustering request / instruction for the target object, or the call of the application programming interface, the server obtains the multimodal information of the target object to be clustered from one or more data service devices and generates multimodal feature information of the target object. Based on the multimodal feature information of the target object, the server self-learns and generates a posterior matching probability function under each modality; based on the posterior matching probability function under each modality, it calculates the posterior matching probability of any two target objects under the multimodality; based on the posterior matching probability of any two target objects under the multimodality, it clusters the target objects and generates a clustering result for the target objects. The server sends the clustering result to the end-side device, and the end-side device may perform at least one of the following processing logics based on the clustering result: information push, information search, and data classification. In another embodiment, the server may perform at least one of the following processing logics based on the clustering result: information push, information search, and data classification, and send the execution result to the end-side device.

[0045] In other embodiments, the terminal side device may obtain the multimodal data of the target object from the data service device. Data of different modalities may come from different devices. The multimodal data of the target object may be preprocessed to generate multimodal feature information of the target object, and the multimodal feature information of the target object may be sent to the server.

[0046] One possible application scenario is the aggregation and organization of image libraries from different regions. Images collected from different regions are grouped together into the same cluster based on the geographic location information of the image and the feature information of at least one modality of the object contained in the image. For example, based on images collected from different regions, multimodal clustering is performed on the images based on the geographic location information of the image, the attribute characteristics and biometric features of the people contained in the images, and images of the same person are grouped together into the same cluster. For example, based on images of motor vehicles collected from different regions, multimodal clustering is performed on the images based on the geographic location information of the motor vehicle, the license plate number, the identity of the owner, and the appearance characteristics of the motor vehicle contained in the motor vehicle images, and images of the same motor vehicle are grouped together into the same cluster.

[0047] In this scenario, the target objects to be clustered are images. The server can be a server of a system platform that has the need for information library aggregation and organization and has obtained authorization, specifically a server deployed locally or on a cloud platform. The server obtains the images to be clustered and their geographic location information from within the system platform or from an image acquisition device that has obtained data authorization from the system platform. Using image recognition technology, the server identifies the objects contained in the images and obtains feature information of at least one modality of the contained objects, thereby obtaining multimodal feature information of the images to be clustered. Based on the multimodal feature information of the images, the server self-learns and generates a posterior matching probability function for each modality. Based on the posterior matching probability function for each modality, the server calculates the posterior matching probability of any two images in the multimodality. The posterior matching probability of two images in the multimodality is the probability that the two images contain the same object. Based on the posterior matching probability of any two images in the multimodality, images containing the same object are clustered into one category, obtaining the image clustering results. Exemplarily, based on the clustering results, the server associates images in the same cluster with the objects contained in the cluster according to their clustering, and stores the images in a classified manner, thus implementing the information library aggregation and organization function. The information to be clustered in the information database may be images, audio, or other types of data information, which is not specifically limited here.

[0048] Another possible application scenario is the e-commerce scenario. Different merchants in the e-commerce platform have different modal information such as videos, pictures, introductory texts, and value information for the same type of items. Through multimodal clustering, similar items from different merchants are clustered into the same cluster, realizing item search, item recommendation and other services based on multimodal information such as videos, pictures, introductory texts, and value information.

[0049] In this scenario, the target objects to be clustered are items, and the server can be an e-commerce platform server, specifically deployed locally or on a cloud platform. The server obtains multimodal information about each item from different merchants on the e-commerce platform, including videos, images, descriptions, and value information. It then performs data preprocessing to generate multimodal feature information for the items. Based on the multimodal feature information, the server self-learns and generates a posterior matching probability function for each modality. Based on this posterior matching probability function, the server calculates the posterior matching probability of any two items in the multimodal context. The posterior matching probability of two items in the multimodal context is the probability that the two items are of the same type. Based on the posterior matching probability of any two items in the multimodal context, similar items are clustered together to obtain a clustering result. For example, based on the clustering results, the server can associate items belonging to the same cluster and, based on the user's query input or the user's current interest (e.g., clicked to browse, added to cart, or followed), search for and recommend similar items to the user.

[0050] The method of this embodiment can be used in scenarios where clustering is required in various application fields, including but not limited to information database aggregation and organization, e-commerce, education, medical care and other fields. It can cluster abstract objects such as data objects, physical objects, etc. There is no specific limitation on the type of target objects to be clustered and the specific application scenarios of the clustering results.

[0051] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0052] Figure 2 is a flow chart of a multimodal information-based clustering method provided by an exemplary embodiment of the present application. The execution entity of this embodiment is the server in the aforementioned multimodal information clustering system architecture. The method provided in this embodiment is used to implement multimodal clustering of target objects based on their multimodal feature information. As shown in Figure 2, the specific steps of this method are as follows:

[0053] Step S201: Acquire multimodal feature information of target objects to be clustered.

[0054] In this embodiment, in order to accurately cluster multiple target objects to be clustered, the server obtains multiple different modal feature information of the target objects. In different application scenarios, the source and method of the server to obtain the multimodal feature information may be different.

[0055] For example, the server may obtain multimodal feature information of the target object from one or more data service devices within the system platform, and feature information of different modalities may come from the same or different data service devices.

[0056] Exemplarily, the server may also obtain multimodal data of the target object from one or more data service devices within the system platform, perform data preprocessing and feature extraction, and generate multimodal feature information of the target object.

[0057] For example, the server may also receive multimodal feature information of a target object from an end-side device. The end-side device may be a device that sends a clustering request / instruction to the server or invokes a clustering application program interface provided by the server. The end-side device may perform data preprocessing and feature extraction on the multimodal data of the target object provided by one or more data service devices to obtain the multimodal feature information of the target object.

[0058] The target object is any one of the following: a physical object, an image containing a physical object, or a text containing a physical object; and the multimodal feature information of the target object includes at least one of the following: an image feature, a text feature, an attribute feature, or a geographic location feature.

[0059] The multimodal feature information of the target object includes but is not limited to: attribute information of one or more modes of the target object (which may include geographic location features, category features, etc.), features of one or more modes of the content contained in the target object (which may include image features, text features, etc.), and features of one or more modes of the target object's associated objects (which may include image features, text features, attribute features, geographic location features, etc.).

[0060] For example, in the image library aggregation and organization scenario, the target object to be clustered can be an image, and the multimodal feature information of the target object includes but is not limited to: the geographic location information of the image acquisition, the feature information of at least one mode of the object contained in the image, such as the biological characteristics of the person contained in the image, the license plate number, owner information, appearance characteristics, etc. of the motor vehicle contained in the image.

[0061] For example, in an e-commerce scenario, the target objects to be clustered may be items, and the multimodal feature information of the target objects includes but is not limited to: videos, pictures, introductory text, and value information.

[0062] Step S202: Based on the multimodal feature information of the target object, self-learn and generate a posterior matching probability function under each modality.

[0063] In this embodiment, a posterior matching probability function for each modality is automatically generated through self-learning based on the multimodal feature information of the target objects. The posterior matching probability function for any modality can be used to calculate the posterior matching probability that any two target objects belong to the same cluster based on the feature information of the target object in that modality. This posterior matching probability reflects the similarity between the target objects based on the feature information of the current modality.

[0064] Step S203: Calculate the posterior matching probability of any two target objects in the multimodal state according to the posterior matching probability function in each modality.

[0065] After self-learning and generating a posteriori matching probability functions for each modality, the algorithm calculates the multimodal posterior matching probability between any two target objects based on these functions and the multimodal feature information of the target objects. This posterior matching probability reflects the overall similarity of the target objects' multimodal feature information. Compared to calculating the posterior matching probability based on feature information from a single modality, using the multimodal posterior matching probability of different target objects to measure target object similarity is more accurate.

[0066] Step S204: cluster the target objects according to the posterior matching probability of any two target objects under multimodality, and generate a clustering result of the target objects.

[0067] After obtaining the posterior matching probability of any two target objects under multimodality, the target objects are clustered based on the posterior matching probability of any two target objects under multimodality. The target objects with a larger posterior matching probability under multimodality are aggregated into the same cluster, and the target objects with a smaller posterior matching probability under multimodality are divided into different clusters, thereby achieving accurate clustering of the target objects.

[0068] In this step, any existing clustering algorithm can be used to perform clustering using the posterior matching probability of the target objects under multimodality as a metric distance of similarity between target objects. The specific clustering process will not be described in detail here.

[0069] The method of this embodiment self-learns and generates a posterior matching probability function under each single modality based on the multimodal feature information of the target objects to be clustered, and the posterior matching probability function under each modality can be automatically learned without using annotation information; the posterior matching probability of any two target objects under multimodality is calculated based on the posterior matching probability function under each single modality; the target objects are clustered based on the posterior matching probability of any two target objects under multimodality to generate a clustering result of the target objects, and the posterior matching probability that the target objects belong to the same cluster is calculated based on the multimodal feature information of the target objects by using the posterior matching probability function generated by self-learning, as a distance measure of the similarity between target objects. Compared with the Euclidean distance or cosine similarity between features used in mainstream methods, the use of the posterior matching probability between target objects can more accurately measure the similarity between target objects. Clustering based on the posterior matching probability between target objects can greatly improve the accuracy and recall rate of clustering.

[0070] FIG3 is a flow chart of a clustering method based on multimodal information provided by another exemplary embodiment of the present application. Based on the aforementioned method embodiment, this embodiment describes the clustering method based on multimodal information in detail. As shown in FIG3 , the specific steps of the method are as follows:

[0071] Step S301: Acquire multimodal feature information of target objects to be clustered.

[0072] This step is implemented in the same way as the aforementioned step S201 and will not be repeated here.

[0073] In an optional embodiment, the server may receive multimodal feature information of the target object to be clustered from the end-side device. Further, after obtaining the final clustering result of the target object, the server outputs the clustering result of the target object to the end-side device.

[0074] After obtaining the multimodal feature information of the target object to be clustered, step S202 is implemented through steps S302-S304, and the posterior matching probability function under each modality is self-learned and generated based on the multimodal feature information of the target object.

[0075] Step S302: Calculate the similarity between any two target objects in the same modality based on the feature information of the target objects.

[0076] Specifically, the feature information of each modality of the target object can be encoded into a feature vector. Based on the feature vector of any modality, the Euclidean distance or cosine similarity between the feature vectors of the modality of any two target objects can be calculated as the similarity between any two target objects under the modality.

[0077] The similarity between any two target objects in any modality can be stored as a matrix to obtain the similarity matrix under that modality. M represents the number of modalities, and the similarity matrix under each modality can be expressed as w (1) , w (2) ,...,w (M) .

[0078] Step S303: construct multiple sample pairs in each modality according to the similarity between any two target objects in each modality, where the sample pairs include two target objects.

[0079] In this step, for any modality, multiple sample pairs are constructed based on the similarity between any two target objects in the modality. Sample pairs in each modality are constructed in the same way to obtain multiple sample pairs in each modality.

[0080] In a possible implementation of this step, when constructing multiple sample pairs under any modality, any two target objects may be combined into a sample pair; or, two different target objects may be randomly selected to form a sample pair.

[0081] In another possible implementation of this step, when constructing multiple sample pairs under any modality, the K nearest neighbor objects of each target object under the modality can also be determined based on the similarity between any two target objects under the modality; any target object under the modality is respectively combined with the K nearest neighbor objects to form a sample pair, and K sample pairs containing the target object are obtained. The number of target objects is expressed as N, and N×K sample pairs under the modality can be obtained. Wherein K and N are both positive integers. By constructing high-quality sample pairs in this way and reducing the number of sample pairs, the quality and efficiency of self-learning to generate the posterior matching probability function under each modality can be improved.

[0082] The K nearest neighbors of any target object in this modality are determined by using the K Nearest Neighbor (KNN) algorithm to determine the K target objects with the highest similarity to the target object, based on the similarity between the target object and other target objects in this modality. K can be set and adjusted based on the needs of the actual application scenario and is not specifically limited here.

[0083] Step S304: Based on the similarity of multiple sample pairs under each modality, an optimization algorithm is used to self-learn and generate a posterior matching probability function under each modality.

[0084] After constructing multiple sample pairs under each modality, an optimization algorithm can be used to self-learn and generate a posterior matching probability function for a single modality under each modality based on the constructed sample pairs and their similarity. This can be achieved through the following steps S3041-S3044:

[0085] Step S3041: Initialize the posterior matching probability function under each mode to a monotonically increasing piecewise function, where the piecewise function contains the piecewise parameters to be solved.

[0086] In practical applications, as the similarity between target objects increases, the posterior matching probability between target objects also increases. In this embodiment, the sample pairs under each modality are arranged in order of similarity from small to large and then divided into I segment sets, where I is a positive integer. T represents the number of sample pairs under a single modality, T = N × K, and the number of sample pairs contained in each segment set is Initialize the posterior matching probability function of a single modality under each modality to a monotonically increasing piecewise function containing I segments. Let M represent the number of modalities, and the posterior matching probability function under the mth modality can be expressed as the following piecewise function:

[0087] Subject to (st):

[0088] Among them, in formula (1), Indicates the i-th mode under the m-th mode m A collection of segments. Indicates the i-th mode under the m-th mode m The number of sample pairs contained in a segment set. Indicates the i-th mode under the m-th mode m Segment Set The set of similarities between pairs of samples in . express The number of similarity values ​​in . Indicates the i-th mode under the m-th mode m The average similarity of the sample pairs in the segment set. Since the segment set is divided into groups by arranging the sample pairs in the order of similarity from small to large, it satisfies represents the i-th posterior matching probability function under the m-th mode m The function value of each piece is the piecewise parameter to be solved. The piecewise function is a monotonically increasing function, so it satisfies f1 (m) <f2 (m) <…<f I (m) The minimum value of the posterior matching probability is 0 and the maximum value is 1, so f1 is satisfied. (m) =0,f I (m) =1. i m [1, 2,…,I], m∈[1, 2,…,M].

[0089] Through formula (1), the posterior matching probability function under each mode can be initialized as a monotonically increasing piecewise function, where the independent variable of each segment is the average value of the similarity of the sample pairs in the corresponding segment set, and the function value is the segment parameter to be solved. There are a total of M×I segment parameters to be solved.

[0090] It should be noted that the number of segments I of the piecewise function can be set according to the actual application scenario, and the value can be different in different application scenarios, and is not specifically limited here. For example, I can be 1000, 200, etc.

[0091] Step S3042: Based on the posterior matching probability function under each mode, construct an objective function, where the objective function includes the segmentation parameters to be solved.

[0092] In this embodiment, the posterior matching probability function under each modality is the posterior matching probability function of a single modality. The posterior matching probability function of a single modality under the mth modality can be expressed as

[0093] In this step, the posterior matching probability function under multimodality is constructed based on the posterior matching probability function of the single modality under each modality.

[0094] Specifically, we can construct a cross-posteriori matching probability function under any two modalities. The cross-posteriori matching probability function under the mth and bth modalities can be expressed as

[0095] Among them, in formula (3), Indicates the i-th mode under the m-th mode m The set of average similarities of sample pairs in the segment set to which they belong under the bth mode. for An average similarity value in i m ,i b ∈[1, 2, ..., I]. m, b∈[1, 2, ..., M]. The meanings of other characters are the same as those in the above content and are not repeated here.

[0096] Furthermore, for any two target objects, the similarity of feature information of given M modalities is The joint probability distribution function of M modes can be expressed as:

[0097] Furthermore, the posterior matching probability function of M modalities is constructed, which can be expressed as:

[0098] The meanings of the characters in formulas (4) and (5) are consistent with those in the above content and will not be repeated here.

[0099] In practical applications, for any set of sample pairs, the mean probability that the sample pairs belong to the same category based on statistics from a single modality should be equal to the mean probability that the sample pairs belong to the same category based on statistics from any multimodal model, and their expected value should be equal to the proportion of samples that actually belong to the same category. That is, for the same set of sample pairs, the posterior matching probability distributions determined based on the posterior matching probability function of a single modality, the cross-posterior matching probability function of two modalities, and the posterior matching probability function of three or more modalities should be consistent. Based on this, a consistency loss function is constructed based on the posterior matching probability function under each modality and the posterior matching probability function under multiple modalities.

[0100] For example, the following consistency loss function can be constructed:

[0101] Where D() represents the distance function, which can be used to calculate Euclidean distance or cosine similarity. The meanings of the other characters are the same as those in the previous content and are not repeated here.

[0102] Given formula (6) affected and Constraints of , it is difficult to learn the consistency loss of these three in formula (6). and Define the following mixed posterior matching probability function:

[0103] The mixed posterior matching probability function of formula (7) is used to constrain and The meanings of other characters are the same as those in the previous content and are not repeated here.

[0104] Furthermore, based on formula (7), the consistency loss function of formula (6) can be replaced by the consistency loss function of formula (8):

[0105] in,

[0106] Among them, L consistency1 Constrain the consistency of the posterior matching probability function of a single modality and the mixed posterior matching probability function, L consistency2 Constrain the consistency of the cross-posterior matching probability function and the mixed posterior matching probability function of the two modalities.

[0107] In practical applications, whether it is the posterior matching probability function of a single modality, the cross-posterior matching probability function of two modalities, or the mixed posterior matching probability function, its minimum value is 0 and its maximum value is 1, based on which the constrained loss function is constructed.

[0108] For example, the following constraint loss function can be constructed:

[0109] Among them, r i ∈[1,2,...,indi], refers to the first indi segments of the piecewise function. j ∈[I-indj,...,I-1,I] refers to the segment after indj+1 in the piecewise function. The values ​​of indi and indj can be set and adjusted according to the actual application scenario. The values ​​vary in different application scenarios and are not specifically limited here. For example, in some scenarios, indi = 10 and indj = 4, while in other scenarios, indi = 10 and indj = 2.

[0110] Furthermore, according to the consistency loss function of formula (8) and the constraint loss function of formula (9), the objective function is constructed as follows: L = λL consistency +L constraint (10)

[0111] In formula (10), λ is a preset weight parameter used to balance consistency loss and constraint loss. The value of λ can be set based on empirical values ​​and is not specifically limited here. Different application scenarios have different values, for example, 80, 20, 2, etc. The meanings of the other characters are consistent with those in the previous content and are not repeated here.

[0112] In an optional embodiment, the objective function may be constructed based solely on the aforementioned constraint loss function.

[0113] Step S3043: Based on the similarity of multiple sample pairs under each modality, an optimization algorithm is used to minimize the objective function and obtain the segmentation parameters.

[0114] After constructing the objective function, an optimization algorithm is used to find the segmentation parameters that minimize the objective function based on the similarity of multiple sample pairs under each modality. This can be achieved using any of the following optimization algorithms: gradient descent, Newton method, quasi-Newton method, momentum optimization, conjugate gradient method, or heuristic optimization algorithm, without specific limitations.

[0115] Step S3044: Substitute the segmentation parameters into the segmentation function to obtain the posterior matching probability function under each mode.

[0116] After solving for the segmentation parameters, the segmentation parameters are substituted into the segmentation function to obtain the posterior matching probability function under each mode.

[0117] In another optional implementation of step S304, the posterior matching probability function under each modality can also be initialized as a continuous monotonic function, which contains unknown parameters to be solved. Based on the continuous monotonic function, a consistency loss function and a constraint loss function are constructed, and an objective function is constructed. The unknown parameters are solved by minimizing the objective function through an optimization algorithm, thereby obtaining a continuous monotonic posterior matching probability function under each modality.

[0118] After learning the posterior matching probability function under each modality, step S203 is implemented through steps S305-S306 to calculate the posterior matching probability of any two target objects under multimodality based on the posterior matching probability function under each modality.

[0119] Step S305 : According to the similarity between any two target objects in any modality, the posterior matching probability function of the modality is used to determine the posterior matching probability between any two target objects in the modality.

[0120] After obtaining the posterior matching probability function under each modality, for any modality, taking the mth modality as an example, according to the similarity of any two target objects under this modality, the I-segmented independent variables of the posterior matching probability function under this modality are In [1], determine the independent variable with the smallest absolute difference in similarity with the two target objects, and use the posterior matching probability corresponding to this independent variable as the posterior matching probability of the two target objects in that modality. Similarly, the posterior matching probability of any two target objects in each modality can be determined.

[0121] Step S306: Calculate the posterior matching probability of any two target objects in multiple modalities based on the posterior matching probability of any two target objects in each modality.

[0122] In this step, after calculating the posterior matching probability of any two target objects in each modality, the posterior matching probability P(i, j) of any two target objects i and j in multimodality can be calculated by the following formula (11):

[0123] in, represents the posterior matching probability of target objects i and j in the mth mode, The meanings of other characters are the same as those in the preceding content and will not be repeated here.

[0124] Optionally, in this step, the posterior matching probabilities of any two target objects in each modality can be averaged to serve as the posterior matching probability of the two target objects in the multimodal state. Optionally, in this step, the posterior matching probabilities of any two target objects in each modality can be weighted and averaged based on the weight coefficients corresponding to each modality, and the calculated result can be used as the posterior matching probability of the two target objects in the multimodal state.

[0125] For example, FIG4 is a framework diagram of determining the posterior matching probability of a target object under multimodal conditions based on multimodal information provided by this embodiment. As shown in FIG4 , the feature information of the target object in M ​​modes is first obtained. For any modality, the similarity between the feature information of any two target objects under the modality is calculated to obtain the similarity matrix w under each modality. (1) ,...,w (M) According to the similarity matrix w under each modality (1) ,...,w (M) The KNN algorithm is used to construct T sample pairs (p1, q1), (p2, q2)…(p T ,q T ), and determine the similarity of T sample pairs. For example, the similarity of (p1, q1) under the mth mode can be expressed as The T sample pairs under each modality are sorted by similarity and divided into I segment sets, using a x , b y ...c z Respectively represent the number of sample pairs in I segment sets, and Respectively represent the first segment set a under the first mode x The similarity of sample pairs, Respectively represent the second segment set b under the first mode y The similarity of sample pairs, Respectively represent the last segment set c under the first mode z The similarity of sample pairs in each segment set under other modalities is expressed in a similar way, for example, Respectively represent a in the first segment set under the second mode xThe similarity of the sample pairs is not listed here one by one. The posterior matching probability function of each single modality is initially a piecewise function, and the cross-posterior matching probability function of the two modalities (i.e., cross-modality) and the posterior matching probability function under M modality are constructed based on the piecewise function. Based on the consistency of various posterior matching probability functions, the posterior matching probability function under each modality is generated by self-learning. Based on the posterior matching probability function under each modality and the similarity of any two target objects under each modality, the posterior matching probability of any two target objects under each single modality can be calculated. Furthermore, the posterior matching probability of any two target objects under each single modality is used to calculate the posterior matching probability of any two target objects under multimodality as the final posterior matching probability of the two target objects. Furthermore, clustering is performed based on the final posterior matching probability of the two target objects to obtain the clustering result of the target objects.

[0126] In an optional embodiment, after calculating the posterior matching probability of any two target objects in multimodality according to the posterior matching probability function under each modality, the posterior matching probability of any two target objects in multimodality can be optimized through the following steps S307-S308 to improve the accuracy of the posterior matching probability of any two target objects in multimodality.

[0127] Step S307: For any two target objects, determine the K nearest neighbor objects of the two target objects in each modality, and determine the common nearest neighbor objects of the two target objects.

[0128] In this step, when determining the K nearest neighbors of any target object in any modality, the KNN algorithm is used to identify the K target objects with the highest similarity to the target object based on the similarity between the target object and other target objects in that modality. K can be set and adjusted based on the needs of the actual application scenario and is not specifically limited here. K is a positive integer.

[0129] Furthermore, after determining the K-nearest neighbors of each target object in each modality, we can obtain the union of the K-nearest neighbor sets of any target object in each modality. This union, serving as the target object's neighbor set, contains the target object's K-nearest neighbors in each modality. For any two target objects, we determine the intersection of their neighbor sets; the neighbors in this intersection are the shared neighbors of the two target objects.

[0130] For example, Represents the set of K nearest neighbor objects of target object i in the mth mode, then the set of nearest neighbor objects of target object i is The union of can be expressed as Similarly, use Represents the set of K nearest neighbor objects of target object j in the mth mode, then the set of nearest neighbor objects of target object j is expressed as The set of common neighbor objects of target object i and target object j is knn ij =knn i ∩knn j .

[0131] Step S308 : Optimizing the posterior matching probability of the two target objects in multimodality according to the posterior matching probability of the two target objects with the same common neighbor object in multimodality.

[0132] In this embodiment, the posterior matching probability of any two target objects in multimodality is optimized based on the posterior matching probability of the two target objects and the same common neighbor object in multimodality, basis consistency and boundary constraints of the posterior matching probability, so as to improve the accuracy of the posterior matching probability of any two target objects in multimodality.

[0133] The first optimization method that can be used in this step is: calculate the product of the posterior matching probabilities of the two target objects and the same common neighbor object under multimodality, and optimize the posterior matching probability of the two target objects under multimodality based on the maximum value of the product and the posterior matching probability of the two target objects under multimodality.

[0134] For example, for the calculated posterior matching probability P(i, j) of any two target objects i and j under multimodality, the following formula (12) can be used for optimization: P(i, j) = max(P(i, j), P(t, h) × P(h, j)) (12)

[0135] In formula (12), h refers to any common neighbor object of target object i and target object j, h∈knn ij . knn ij =knn i ∩kmn j , represents the set of shared neighbors of target objects i and j. max() is the maximum value function. The meanings of the other characters are the same as those in the previous section and are not repeated here.

[0136] A second optimization method that can be used in this step is: taking the sum of the posterior matching probabilities of the two target objects and the same common neighbor object under multimodality as the sum of the posterior matching probabilities corresponding to the common neighbor object; calculating the sum of the posterior matching probabilities corresponding to each common neighbor object to obtain a first summation result; for the first object and the second object of the two target objects, calculating the sum of the posterior matching probabilities of the first object and each common neighbor object under multimodality to obtain a second summation result, calculating the sum of the posterior matching probabilities of the second object and each common neighbor object under multimodality to obtain a third summation result, and calculating the sum of the second summation result and the third summation result to obtain a fourth summation result; according to the ratio of the first summation result to the fourth summation result, optimizing the posterior matching probabilities of the two target objects under multimodality.

[0137] For example, the calculated posterior matching probability P(i, j) between any two target objects i and j in multimodality can be optimized using the following formula (13):

[0138] Among them, h in formula (13) i Refers to any neighbor object in the neighbor object set of the target object i, h i ∈knn i .h j Refers to any neighbor object in the neighbor object set of the target object j, h j ∈knn j The meanings of other characters are the same as those in the above content and will not be repeated here.

[0139] In this step, the two aforementioned optimization methods can also be used to optimize the posterior matching probability of any two target objects in a multimodal environment, and the order in which the two optimization methods are used for optimization is not limited. For example, the posterior matching probability of any two target objects in a multimodal environment can be optimized using the first optimization method first, and then the optimization result of the first optimization method can be optimized a second time using the second optimization method; or, the posterior matching probability of any two target objects in a multimodal environment can be optimized using the second optimization method first, and then the optimization result of the second optimization method can be optimized a second time using the first optimization method.

[0140] Step S309: cluster the target objects according to the posterior matching probability of any two target objects under multimodality, and generate a clustering result of the target objects.

[0141] After obtaining the posterior matching probability of any two target objects under multimodality, the target objects are clustered based on the posterior matching probability of any two target objects under multimodality to generate a clustering result of the target objects.

[0142] In this step, any existing clustering algorithm can be used to perform clustering using the posterior matching probability of the target objects under multimodality as a metric distance of similarity between target objects. The specific clustering process will not be described in detail here.

[0143] In this embodiment, for any two target objects, when the feature information of one or more modes of the target objects is missing, the feature information of the mode can be ignored, and the posterior matching probability of the two target objects under the remaining multimodalities is calculated based on the feature information of the remaining modes without missing information. Therefore, when the feature information of some modes is missing, clustering can still be performed more accurately without supplementing the feature information of the missing modes. At the same time, full use is made of the feature information of the multimodalities for clustering, attention is paid to the importance between different modes, and the accuracy and recall rate of clustering are improved. In addition, T sample pairs under each mode are constructed through the KNN algorithm, and based on the similarity of the T sample pairs under each mode, the posterior matching probability function under each mode is self-learned to generate. This can shorten the time for self-learning to generate the posterior matching probability function under each mode, thereby improving the efficiency of clustering.

[0144] The clustering method based on multimodal information provided in this application can self-learn to generate the posterior matching probability function under each single modality, so that it can be quickly and unbiasedly applied to different data distribution scenarios, such as the aggregation and organization of image libraries in different regions, the clustering of similar products, etc.

[0145] FIG5 is a flow chart of a clustering method based on multimodal information provided by another exemplary embodiment of the present application. The method provided in this embodiment can be applied to the aggregation and sorting scenarios of image libraries in different regions, and the images collected in different regions in different regional information libraries are aggregated into the same cluster based on the geographical location information of the image collection and the feature information of at least one modality of the object contained in the image. For example, based on images collected in different regions, based on the geographical location information of the image collection and the biological features of the people contained in the images, multimodal clustering is performed on the images, and the images of the same person are aggregated into the same cluster. For example, based on motor vehicle images collected in different regions, multimodal clustering is performed on the motor vehicle images based on the geographical location information of the motor vehicle images, the license plate number of the motor vehicle contained in the motor vehicle images, the identity of the owner, the appearance features, etc., and the images of the same motor vehicle are aggregated into the same cluster.

[0146] As shown in Figure 5, the specific steps of the method are as follows:

[0147] Step S501: Acquire multimodal feature information of the image to be clustered, where the multimodal feature information includes geographic location information and feature information of at least one modality of the included objects.

[0148] In this embodiment, the target object to be clustered is an image, and the multimodal feature information of the image includes geographical location information of the image acquisition and feature information of at least one modality of the object contained in the image.

[0149] Exemplarily, in the scenario of clustering and organizing image libraries in different regions, the images to be clustered can be images of specific objects collected from different regions in multiple regional information libraries. Taking human images as an example, the multi-model feature information of the image can include the location coordinates of the image collection, and at least one of the following modal features of the person contained in the image: biometric features, gait features, and identity information.

[0150] Step S502: Generate a posterior matching probability function under each modality through self-learning based on the multi-modal feature information of the image.

[0151] After obtaining the multimodal feature information of the image to be clustered, the posterior matching probability function under each modality is self-learned and generated according to the multimodal feature information of the image. This step can be implemented in a manner similar to step S202 in any of the aforementioned method embodiments. The target object in this step is the image. Please refer to the relevant content in the aforementioned embodiments for details, which will not be repeated here.

[0152] Step S503 : Calculate the posterior matching probability of any two images in multimodality according to the posterior matching probability function in each modality, where the posterior matching probability of two images in multimodality is the probability that the two images contain the same object.

[0153] The specific implementation method of this step is similar to the implementation method of calculating the posterior matching probability of any two target objects in multimodality according to the posterior matching probability function under each modality in the aforementioned step S203. The target object in this step is an image. For details, please refer to the relevant content in the aforementioned embodiment, which will not be repeated here.

[0154] Step S504: Based on the posterior matching probability of any two images under multimodality, the images containing the same object are clustered into one category to obtain a clustering result of the images.

[0155] After obtaining the posterior matching probability of any two images under multimodality, the images are clustered based on the posterior matching probability of any two images under multimodality. Images with a larger posterior matching probability under multimodality are aggregated into the same cluster, and images with a smaller posterior matching probability under multimodality are divided into different clusters, thereby achieving accurate image clustering based on the uniqueness of the objects contained in the images.

[0156] In this step, any existing clustering algorithm can be used to cluster images using the posterior matching probability of images under multimodality as a metric distance for similarity between images. The specific clustering process will not be described here.

[0157] Furthermore, after obtaining the image clustering results, the server sends the clustering results to the end-side device, and the end-side device can execute at least one of the following processing logics based on the image clustering results: information push, information search, and data classification. In another embodiment, the server can execute at least one of the following processing logics based on the image clustering results: information push, information search, and data classification, and send the execution results to the end-side device.

[0158] For example, after obtaining the image clustering result, the server may associate the images of the same cluster with the objects contained therein according to the cluster to which they belong, and classify and store the images to realize the information library aggregation and organization function.

[0159] For example, after obtaining the image clustering result, the server may further analyze and determine the activity range, social area, and other information of the designated object based on the geographic location information of the image associated with the designated object.

[0160] In this embodiment, in a clustering and sorting scenario of different regional information repositories, geographic location information of the images to be clustered and feature information of at least one modality of the objects contained therein are obtained to obtain multimodal feature information of the images to be clustered. A posterior matching probability function for each single modality is generated through self-learning, and the posterior matching probability function for each modality can be automatically learned without using annotation information. Based on the posterior matching probability function for each single modality, the posterior matching probability of any two images in multimodality is calculated. Based on the posterior matching probability of any two images in multimodality, the images are clustered to generate image clustering results. By using the posterior matching probability function generated by self-learning, the posterior matching probability that the images contain the same object is calculated based on the multimodal feature information of the images. This is used as a distance measure of similarity between images. Compared with mainstream methods that use Euclidean distance or cosine similarity between image features, the use of the posterior matching probability between images can more accurately measure the similarity of objects contained in the images. Clustering based on the posterior matching probability between images can significantly improve the accuracy and recall rate of image clustering.

[0161] Figure 6 is an interaction diagram of a clustering method based on multimodal information provided by an exemplary embodiment of the present application. As shown in Figure 6, the interaction process between the server and the end-side device in the clustering method based on multimodal information is as follows:

[0162] Step S601: The terminal-side device obtains multimodal feature information of the target object to be clustered.

[0163] In this embodiment, the terminal-side device obtains the multimodal data of the target object from the data service device. Data of different modalities can come from different devices. The multimodal data of the target object is preprocessed to generate multimodal feature information of the target object, and the multimodal feature information of the target object is sent to the server.

[0164] Among them, the preprocessing of the multimodal data of the target object may include but is not limited to dirty data cleaning, feature extraction, etc. The feature information required to be obtained in different application scenarios is different. The data preprocessing and feature extraction methods can be set according to the needs of the actual application scenarios. The extraction method of various types of feature information can be implemented using commonly used technologies for extracting similar features, which will not be repeated here.

[0165] Step S602: The client device sends multimodal feature information of the target object to be clustered to the server.

[0166] Step S603: The server receives multimodal feature information of the target object to be clustered.

[0167] Step S604: The server self-learns and generates a posterior matching probability function under each modality based on the multi-modal feature information of the target object.

[0168] The specific implementation method of this step is the same as that of the aforementioned step S202. Please refer to the relevant content in the aforementioned embodiment for details, and will not be repeated here.

[0169] Step S605: The server calculates the posterior matching probability of any two target objects in the multimodal environment according to the posterior matching probability function in each modality.

[0170] The specific implementation method of this step is the same as that of the aforementioned step S203. Please refer to the relevant content in the aforementioned embodiment for details, and no further details will be given here.

[0171] Step S606: The server clusters the target objects according to the posterior matching probability of any two target objects under multimodality, and generates a clustering result of the target objects.

[0172] The specific implementation method of this step is the same as that of the aforementioned step S204. Please refer to the relevant content in the aforementioned embodiment for details, and no further details will be given here.

[0173] Step S607: The server sends the clustering result of the target object to the terminal device.

[0174] Step S608: The client device receives the clustering result of the target object sent by the server.

[0175] Step S609: The terminal-side device executes at least one of the following processing logics based on the clustering result: information push, information search, and data classification.

[0176] For example, in an e-commerce scenario, similar items from different merchants are clustered into the same cluster through multimodal clustering. After obtaining the item clustering results, services such as item search and item recommendation can be implemented based on multimodal information such as videos, pictures, introductory text, and value information.

[0177] For example, based on the item the user is currently browsing, we can search for information about similar items and display / push information about similar items to the user, allowing the user to easily select the item they want. Alternatively, based on a user-submitted image, we can search for information about the item in the image and similar items, and display / push information about the item in the image and similar items to the user, allowing the user to easily select the item they want.

[0178] For example, in a clustering and organizing databases across different regions, images collected in different regions belonging to the same object are grouped into the same cluster based on the geographic location of the image and at least one modality of feature information about the object. Furthermore, based on the clustering results, images in the same cluster are associated with the objects they contain, and the images are categorized and stored, thus achieving the database clustering and organization function.

[0179] This embodiment provides an interaction process between a server and an end-side device in a clustering method based on multimodal information.

[0180] FIG7 is a schematic diagram of the structure of a multimodal information-based clustering device provided in an exemplary embodiment of the present application. The multimodal information-based clustering device provided in an embodiment of the present application can execute the processing flow provided in an embodiment of a multimodal information-based clustering method. As shown in FIG7 , a multimodal information-based clustering device 70 includes: a multimodal feature acquisition module 71, a self-learning module 72, a posterior matching probability calculation module 73, and a multimodal clustering module 74.

[0181] Specifically, the multimodal feature acquisition module 71 is used to acquire multimodal feature information of the target objects to be clustered.

[0182] The self-learning module 72 generates a posterior matching probability function under each modality through self-learning based on the multi-modal feature information of the target object.

[0183] The posterior matching probability calculation module 73 calculates the posterior matching probability of any two target objects in the multimodal state according to the posterior matching probability function in each modality.

[0184] The multimodal clustering module 74 clusters the target objects according to the posterior matching probability of any two target objects under multimodality, and generates a clustering result of the target objects.

[0185] In an optional embodiment, when implementing self-learning to generate a posterior matching probability function in each modality based on multimodal feature information of the target object, the self-learning module 72 is further configured to:

[0186] Based on the feature information of the same modality of the target object, the similarity between any two target objects under the modality is calculated; based on the similarity between any two target objects under each modality, multiple sample pairs under each modality are constructed, and the sample pairs include two target objects; based on the similarity of multiple sample pairs under each modality, an optimization algorithm is used to self-learn and generate the posterior matching probability function under each modality.

[0187] In an optional embodiment, when constructing multiple sample pairs in each modality based on the similarity between any two target objects in each modality, and the sample pairs include two target objects, the self-learning module 72 is further configured to:

[0188] For any modality, based on the similarity of any two target objects under the modality, the K nearest neighbor objects of each target object under the modality are determined, where K is a positive integer; each target object under the modality is respectively paired with the K nearest neighbor objects to obtain multiple sample pairs under the modality.

[0189] In an optional embodiment, when implementing the self-learning generation of the posterior matching probability function under each modality based on multiple sample pairs and the similarity of the sample pairs under each modality using an optimization algorithm, the self-learning module 72 is further configured to:

[0190] The posterior matching probability function under each modality is initialized as a monotonically increasing piecewise function, which contains the piecewise parameters to be solved. Based on the posterior matching probability function under each modality, an objective function is constructed, which contains the piecewise parameters to be solved. According to the similarity of multiple sample pairs under each modality, an optimization algorithm is used to minimize the objective function and solve the piecewise parameters. The piecewise parameters are substituted into the piecewise function to obtain the posterior matching probability function under each modality.

[0191] In an optional embodiment, when constructing the target function based on the posterior matching probability function under each modality, the self-learning module 72 is further configured to:

[0192] According to the posterior matching probability function under each modality, the posterior matching probability function under multimodality is constructed; according to the posterior matching probability function under each modality and the posterior matching probability function under multimodality, the consistency loss function and the constraint loss function are constructed;

[0193] The objective function is constructed based on the consistency loss function and the constraint loss function.

[0194] In an optional embodiment, when calculating the posterior matching probability of any two target objects in multimodality according to the posterior matching probability function in each modality, the posterior matching probability calculation module 73 is further configured to:

[0195] According to the similarity of any two target objects in any modality, the posterior matching probability function of the modality is used to determine the posterior matching probability of any two target objects in the modality; according to the posterior matching probability of any two target objects in each modality, the posterior matching probability of any two target objects in multiple modalities is calculated.

[0196] In an optional embodiment, after calculating the posterior matching probability of any two target objects in the multimodal state according to the posterior matching probability function in each modality, the posterior matching probability calculation module 73 is further configured to:

[0197] For any two target objects, determine the K nearest neighbor objects of the two target objects in each modality, and determine the common nearest neighbor objects of the two target objects, where K is a positive integer; based on the posterior matching probability of the two target objects with the same common nearest neighbor object in multimodality, optimize the posterior matching probability of the two target objects in multimodality.

[0198] In an optional embodiment, when optimizing the posterior matching probability of two target objects in multimodality based on the posterior matching probability of the two target objects with the same common neighbor object in multimodality, the posterior matching probability calculation module 73 is further configured to:

[0199] The product of the posterior matching probabilities of two target objects and the same common neighbor object under multimodality is calculated, and the posterior matching probability of the two target objects under multimodality is optimized according to the maximum value of the product and the posterior matching probability of the two target objects under multimodality.

[0200] In an optional embodiment, when optimizing the posterior matching probability of two target objects in multimodality based on the posterior matching probability of the two target objects with the same common neighbor object in multimodality, the posterior matching probability calculation module 73 is further configured to:

[0201] The sum of the posterior matching probabilities of the two target objects with the same shared neighbor object under multimodality is used as the sum of the posterior matching probabilities corresponding to the shared neighbor object; the sum of the posterior matching probabilities corresponding to each shared neighbor object is calculated to obtain a first summation result; for the first object and the second object of the two target objects, the sum of the posterior matching probabilities of the first object with each shared neighbor object under multimodality is calculated to obtain a second summation result, the sum of the posterior matching probabilities of the second object with each shared neighbor object under multimodality is calculated to obtain a third summation result, and the sum of the second summation result and the third summation result is calculated to obtain a fourth summation result; and the posterior matching probabilities of the two target objects under multimodality are optimized according to the ratio of the first summation result to the fourth summation result.

[0202] In an optional embodiment, when acquiring multimodal feature information of target objects to be clustered, the modal feature acquisition module 71 is further configured to: receive multimodal feature information of target objects to be clustered sent by a terminal-side device.

[0203] After generating the clustering result of the target object, the multimodal clustering module 74 is further configured to output the clustering result of the target object to the terminal-side device.

[0204] The device provided in the embodiment of the present application can be specifically used to execute the processing flow executed by the server in any of the above method embodiments, and the specific functions and technical effects that can be achieved are not repeated here.

[0205] Figure 8 is a schematic diagram of the structure of a server provided in an embodiment of the present application. As shown in Figure 8, the server includes: a memory 801 and a processor 802. The memory 801 is used to store computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 802 is communicatively connected to the memory 801 and is used to execute the computer-executable instructions stored in the memory 801 to implement the technical solution provided by any of the above-mentioned method embodiments. Its specific functions and the technical effects that can be achieved are similar and will not be repeated here. Figure 8 takes the server as a cloud server as an example for illustrative explanation, and the server can also be a locally deployed server.

[0206] Optionally, as shown in Figure 8, the server further includes other components such as a firewall 803, a load balancer 804, a communication component 805, and a power supply component 806. Figure 8 only schematically illustrates some components, which does not mean that the server only includes the components shown in Figure 8.

[0207] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement the solution provided by any of the above-mentioned method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0208] The embodiment of the present application also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of the server can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the server executes the solution provided by any of the above-mentioned method embodiments. The specific functions and technical effects that can be achieved are not repeated here. The embodiment of the present application provides a chip, including: a processing module and a communication interface, and the processing module can execute the technical solution of the server in the above-mentioned method embodiment. Optionally, the chip also includes a storage module (such as a memory), the storage module is used to store instructions, the processing module is used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution provided by any of the above-mentioned method embodiments.

[0209] The above storage may be an object storage service (OSS).

[0210] The above-mentioned memory can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0211] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile hotspot (WiFi), a second-generation mobile communication system (2G), a third-generation mobile communication system (3G), a fourth-generation mobile communication system (4G) / long-term evolution (LTE), a fifth-generation mobile communication system (5G) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0212] The power supply assembly provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0213] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0214] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0215] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0216] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0217] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0218] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0219] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0220] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0221] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or in parallel. They are only used to distinguish between different operations, and the sequence number itself does not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit "first" and "second" to different types. The meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0222] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0223] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0224] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A clustering method based on multimodal information, characterized in that: include: Obtain multimodal feature information of the target object to be clustered; According to the multimodal feature information of the target object, self-learning generates a posterior matching probability function under each modality; Calculating the posterior matching probability of any two target objects in multi-modality according to the posterior matching probability function in each modality; According to the posterior matching probability of any two target objects under multimodality, the target objects are clustered to generate clustering results of the target objects.

2. The method according to claim 1, characterized in that The self-learning and generating a posteriori matching probability function in each mode according to the multi-modal feature information of the target object includes: Calculate the similarity between any two target objects in the same mode according to the feature information of the target object; According to the similarity between any two target objects in each modality, construct multiple sample pairs in each modality, wherein the sample pairs include two target objects; According to the similarity of multiple sample pairs under each modality, an optimization algorithm is used to self-learn and generate the posterior matching probability function under each modality.

3. The method according to claim 2, characterized in that According to the similarity of any two target objects in each modality, a plurality of sample pairs in each modality are constructed, wherein the sample pairs include two target objects, including: For any modality, according to the similarity between any two target objects in the modality, determine the K nearest neighbor objects of each target object in the modality, where K is a positive integer; Each target object under the modality is respectively combined with K nearest neighbor objects to form a sample pair, so as to obtain a plurality of sample pairs under the modality.

4. The method according to claim 2, characterized in that: The method of using an optimization algorithm to self-learn and generate a posteriori matching probability function under each modality based on multiple sample pairs under each modality and the similarity of the sample pairs includes: Initialize the posterior matching probability function under each mode to a monotonically increasing piecewise function, wherein the piecewise function contains piecewise parameters to be solved; Based on the posterior matching probability function under each mode, construct an objective function, wherein the objective function includes the segmentation parameters to be solved; According to the similarity of multiple sample pairs under each modality, an optimization algorithm is used to minimize the objective function to obtain the segmentation parameter; Substituting the segmentation parameters into the segmentation function, a posterior matching probability function under each mode is obtained.

5. The method according to claim 4, characterized in that The objective function is constructed based on the posterior matching probability function under each mode, including: According to the posterior matching probability function under each mode, construct the posterior matching probability function under multi-modality; According to the posterior matching probability function under each modality and the posterior matching probability function under multi-modality, a consistency loss function and a constraint loss function are constructed; An objective function is constructed according to the consistency loss function and the constraint loss function.

6. The method according to claim 1, characterized in that The calculating the posterior matching probability of any two target objects in multimodality according to the posterior matching probability function in each modality includes: According to the similarity of any two target objects in any modality, the posterior matching probability function in the modality is used to determine the posterior matching probability of any two target objects in the modality; According to the posterior matching probability of any two target objects in each modality, the posterior matching probability of any two target objects in multi-modality is calculated.

7. The method according to any one of claims 1 to 6, characterized in that After calculating the posterior matching probability of any two target objects in multimodality according to the posterior matching probability function in each modality, the method further includes: For any two target objects, determine the K nearest neighbor objects of the two target objects in each mode, and determine the common nearest neighbor objects of the two target objects, where K is a positive integer; According to the posterior matching probability of the two target objects and the same common neighbor object in multimodality, the posterior matching probability of the two target objects in multimodality is optimized.

8. The method according to claim 7, characterized in that The optimizing the posterior matching probability of the two target objects in multimodality according to the posterior matching probability of the two target objects with the same common neighbor object in multimodality includes: The product of the posterior matching probabilities of the two target objects and the same common neighbor object in multimodality is calculated, and the posterior matching probability of the two target objects in multimodality is optimized according to the maximum value of the product and the posterior matching probability of the two target objects in multimodality.

9. The method according to claim 7, characterized in that: The optimizing the posterior matching probability of the two target objects in multimodality according to the posterior matching probability of the two target objects with the same common neighbor object in multimodality includes: The sum of the posterior matching probabilities of the two target objects and the same shared neighbor object under multimodality is used as the sum of the posterior matching probabilities corresponding to the shared neighbor object; Calculate the sum of the posterior matching probabilities corresponding to the common neighbor objects to obtain a first summation result; For a first object and a second object of the two target objects, calculating the sum of the posterior matching probabilities of the first object and each of the shared neighboring objects under multimodality to obtain a second summation result, calculating the sum of the posterior matching probabilities of the second object and each of the shared neighboring objects under multimodality to obtain a third summation result, and calculating the sum of the second summation result and the third summation result to obtain a fourth summation result; According to the ratio of the first summation result to the fourth summation result, the posterior matching probability of the two target objects in multimodality is optimized.

10. The method according to any one of claims 1 to 6, characterized in that: The step of obtaining multimodal feature information of target objects to be clustered includes: The multimodal feature information of the target object to be clustered is sent by the receiving end-side device; After generating the clustering result of the target object, the method further includes: The clustering result of the target object is output to the terminal side device.

11. A clustering method based on multimodal information, characterized in that: include: Acquire multimodal feature information of the images to be clustered, wherein the multimodal feature information includes geographic location information and feature information of at least one mode of the included objects; According to the multi-modal feature information of the image, self-learning is used to generate a posterior matching probability function in each modality; Calculating the posterior matching probability of any two images in multimodality according to the posterior matching probability function in each modality, wherein the posterior matching probability of two images in multimodality is the probability that the two images contain the same object; According to the posterior matching probability of any two images under multimodality, images containing the same object are clustered into one category to obtain a clustering result of the images.

12. A clustering method based on multimodal information, characterized in that: Applied to terminal devices, including: Obtain multimodal feature information of the target object to be clustered; Sending multimodal feature information of target objects to be clustered to the server; receiving a clustering result of the target object sent by the server, where the clustering result of the target object is generated by the method according to any one of claims 1 to 10; According to the clustering result, at least one of the following processing logics is executed: information push, information search, data classification.

13. A server, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 11 when executed by a processor.

Citation Information

Patent Citations

  • Integrated multi-classifier fusion classification method and integrated multi-classifier fusion classification system based on graph clustering label propagation

    CN103605990A

  • Data processing method, device and equipment and readable storage medium

    CN110020022A

  • Method and device for processing multi-modal information of target object and storage medium

    CN114357263A

  • Information matching method and apparatus, computer device and readable storage medium

    WO2023024413A1

Cited By

  • Image clustering method, electronic equipment and computer readable storage medium

    CN121280752A