Processing method and device of multi-modal information of target object and storage medium

Abstract

This invention discloses a method, apparatus, and storage medium for processing multimodal information of target objects. The method includes: acquiring multimodal information of multiple target objects in multiple modalities, wherein the multimodal information includes: the metric distance of any target object in each modality; calculating the posterior matching probability between any two target objects based on the multimodal information, wherein the target objects are objects to be clustered; and clustering all target objects into multimodal objects based on the posterior matching probability between any two target objects, generating clustering results. This invention solves the technical problem in related clustering methods that are difficult to efficiently and accurately achieve object clustering.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically, to a method, apparatus, and storage medium for processing multimodal information of a target object. Background Technology

[0002] With the development of cloud computing and mobile internet, massive amounts of data are generated constantly across different platforms. This data lacks multimodality; different objects contain different modalities, each with distinct attributes. Leveraging the complementarity between multimodalities and eliminating redundancy is crucial for learning better feature attributes. Clustering is one important application. For example, the massive amounts of photos and videos of pedestrians, motor vehicles, and non-motorized vehicles generated constantly in cities can be clustered online to obtain important information such as movement trajectories, activity ranges, and social areas. Another example is the large-scale short video data uploaded by mobile phones. By acquiring multimodal information such as video content, images, audio, and text, online clustering can be used for applications such as similar video clustering and retrieval. Furthermore, people's expressions on social networks are becoming increasingly diverse, using images, text, and videos to express their emotions and opinions. Utilizing multimodal information such as images, videos, and text for dynamic content clustering and dynamic sentiment clustering on social networks can recommend similar content to users. For example, more and more shopping platforms offer search services. Different merchants on shopping platforms have different descriptions of the same type of items, such as videos, pictures, text, and prices. By using multimodal clustering to group similar items from different merchants together, users can be provided with a search service that combines videos, pictures, and text.

[0003] In related technologies, the currently used missing multimodal clustering methods are mainly based on matrix factorization strategies, spectral clustering methods, multi-kernel learning methods, and contrastive learning methods, which are difficult to perform multimodal clustering simply and efficiently.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, and storage medium for processing multimodal information of a target object, in order to at least solve the technical problem that it is difficult to efficiently and accurately achieve object clustering in clustering methods of related technologies.

[0006] According to one aspect of the present invention, a method for processing multimodal information of target objects is provided, comprising: acquiring multimodal information of multiple target objects in multiple modalities, wherein the multimodal information includes: metric distance of any target object in each modality; calculating the posterior matching probability between any two target objects based on the multimodal information, wherein the target objects are objects to be clustered; and performing multimodal object clustering on all target objects based on the posterior matching probability between any two target objects to generate clustering results.

[0007] Optionally, based on the multimodal information, calculating the posterior matching probability between any two target objects includes: extracting the metric distance between the two target objects based on the multimodal information of the two target objects, wherein the metric distance includes at least one of the following: Euclidean distance and spatial distance; collecting Bayesian formula and conditional independence equation, calculating the metric distance between the two target objects, and obtaining the posterior matching probability between the two target objects.

[0008] Optionally, based on the posterior matching probability between any two target objects, multimodal object clustering is performed on all target objects to generate clustering results, including: determining at least two target objects to be clustered from the plurality of target objects and counting the number of target objects to be clustered, wherein the target objects to be clustered are labeled with clustering labels; performing probabilistic clustering on the target objects to be clustered based on the posterior matching probability of the target objects to be clustered, and generating the clustering results.

[0009] Optionally, based on the posterior matching probability of the target objects to be clustered, probabilistic clustering is performed on the target objects to be clustered to generate the clustering result, including: constructing K nearest neighbor objects for each target object to be clustered according to the posterior matching probability of the target objects to be clustered, and initializing each target object to be clustered as a separate category; traversing each target object to be clustered, classifying each target object to be clustered into its original category, and classifying the maximum likelihood probability loss function of the corresponding K nearest neighbor object's category as the lowest category; if the traversal of each target object to be clustered is complete, summarizing the classification results into the clustering result.

[0010] Optionally, constructing the K nearest neighbor objects for each target object to be clustered includes: sorting the target objects to be clustered according to the posterior matching probability between each target object and other target objects, and extracting the K objects with the highest posterior matching probability as the K nearest neighbor objects.

[0011] Optionally, the method further includes: analyzing the clustering labels of the target objects to be clustered based on the joint probability density function to determine the probability loss of the probabilistic clustering.

[0012] Optionally, the method further includes: if there is missing modal information in one or more modalities of the target object, supplementing the modal information with the missing data based on the clustering results.

[0013] According to one aspect of the present invention, a method for processing multimodal information of a target object is provided, comprising: acquiring multimodal information of multiple target objects in a multimodal context, wherein, when the multimodality is the face modality and geographic modality of any target object, the multimodal information includes at least: face similarity and spatial distance; calculating the posterior matching probability between any two target objects based on the face similarity and spatial distance, wherein the target objects are objects to be clustered; and clustering all target objects into multimodal objects based on the posterior matching probability between any two target objects to generate a clustering result.

[0014] According to one aspect of the present invention, a method for processing multimodal information of a target object is provided, comprising: acquiring multimodal information of multiple target objects in a multimodal context, wherein, when the multimodality is a face feature modality and a body feature modality of any target object, the multimodal information includes at least: face similarity and body similarity; calculating a posterior matching probability between any two target objects based on the face similarity and body similarity, wherein the target objects are objects to be clustered; and performing multimodal object clustering on all target objects based on the posterior matching probability between any two target objects to generate a clustering result.

[0015] According to one aspect of the present invention, a processing apparatus for multimodal information of target objects is provided, comprising: a first acquisition module, configured to acquire multimodal information of multiple target objects in multiple modalities, wherein the multimodal information includes: a metric distance of any target object in each modality; a first calculation module, configured to calculate the posterior matching probability between any two target objects based on the multimodal information, wherein the target objects are objects to be clustered; and a first clustering module, configured to perform multimodal object clustering on all target objects based on the posterior matching probability between any two target objects, and generate a clustering result.

[0016] According to one aspect of the present invention, a processing apparatus for multimodal information of target objects is provided, comprising: a second acquisition module, configured to acquire multimodal information of multiple target objects in a multimodal context, wherein, when the multimodality is a face modality and a geographic modality of any target object, the multimodal information includes at least: face similarity and spatial distance; a second calculation module, configured to calculate a posterior matching probability between any two target objects based on the face similarity and spatial distance, wherein the target objects are objects to be clustered; and a second clustering module, configured to cluster all target objects into multimodal objects based on the posterior matching probability between any two target objects, thereby generating a clustering result.

[0017] According to one aspect of the present invention, a processing apparatus for multimodal information of a target object is provided, comprising: a third acquisition module, configured to acquire multimodal information of multiple target objects in a multimodal context, wherein, when the multimodality is a face feature modality and a body feature modality of any target object, the multimodal information includes at least: face similarity and body similarity; a third calculation module, configured to calculate a posterior matching probability between any two target objects based on the face similarity and body similarity, wherein the target objects are objects to be clustered; and a third clustering module, configured to cluster all target objects into multimodal objects based on the posterior matching probability between any two target objects, and generate a clustering result.

[0018] According to one aspect of the present invention, a storage medium is provided, the storage medium including a stored program, wherein, when the program is executed, the device where the storage medium is located is controlled to perform the multimodal information processing method for the target object as described in any one of the above embodiments.

[0019] According to one aspect of the present invention, a processor is provided, characterized in that the processor is configured to run a program, wherein the program executes the multimodal information processing method for the target object as described in any one of the preceding embodiments.

[0020] In this embodiment of the invention, multimodal information of multiple target objects in a multimodal manner is acquired. Based on this multimodal information, the posterior matching probability between any two target objects is calculated. Then, based on the posterior matching probability between any two target objects, multimodal object clustering is performed on all target objects to generate clustering results. Because the clustering results are obtained based on the posterior matching probability between any two target objects, and the posterior matching probability is obtained based on multimodal information, the posterior matching probability can be obtained based on more comprehensive information. This solves the technical problem in related clustering methods that are difficult to efficiently and accurately achieve object clustering. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0022] Figure 1 A hardware structure block diagram of a mobile device for processing multimodal information of a target object is shown.

[0023] Figure 2 This is a flowchart of a method for processing multimodal information of a target object according to Embodiment 1 of the present invention;

[0024] Figure 3 This is a flowchart of a second method for processing multimodal information of a target object according to Embodiment 1 of the present invention;

[0025] Figure 4 This is a flowchart of a third method for processing multimodal information of a target object according to Embodiment 1 of the present invention;

[0026] Figure 5 This is a schematic diagram of heterogeneous bimodal posterior matching probability calculation provided by an optional embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of homogeneous bimodal posterior matching probability calculation provided by an optional embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of probabilistic clustering provided by an optional embodiment of the present invention;

[0029] Figure 8 This is a structural block diagram of a device for processing multimodal information of a target object according to Embodiment 2 of the present invention;

[0030] Figure 9 This is a structural block diagram of the second device for processing multimodal information of a target object according to Embodiment 3 of the present invention;

[0031] Figure 10 This is a structural block diagram of the third device for processing multimodal information of a target object according to Embodiment 4 of the present invention;

[0032] Figure 11 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0036] Modality: Every source or form of information can be called a modality. For example, humans have touch, hearing, vision, smell, and taste; information media include voice, video, and text; and there are various sensors such as radar, infrared, and accelerometers. Each of these can be called a modality. In the context of urban brains, the multimodal nature of people includes visual information such as human features / gait features / facial features, attribute information, geographic information such as time and space, and metadata such as video detection, tracking, and ID. In e-commerce scenarios, the multimodal nature of products includes image features / text representation, product categories, sales volume / reviews, and other information.

[0037] Clustering: The process of dividing a collection of physical or abstract objects into multiple classes composed of similar objects is called clustering. A cluster generated by clustering is a set of data objects that are similar to objects in the same cluster and different from objects in other clusters.

[0038] Multimodal clustering: Clustering objects with multimodal information. In the context of a smart city system, this means grouping all captured images of the same person / vehicle / non-motorized vehicle into one category. In e-commerce, it means grouping products sold in different stores into the same category based on the uniqueness of the product.

[0039] Missing multimodal data: Data for one or more modalities is missing from a collection of multimodal objects. For example, in a city brain scenario, video streams contain detection and tracking information, while image streams do not, and some non-motorized vehicles are not associated with human bodies.

[0040] Euclidean distance: refers to the distance between two points in Euclidean space. In this scheme, it is used to calculate the Euclidean distance between the eigenvectors of a certain mode between two objects.

[0041] Feature vector: A one-dimensional array calculated for an image. The similarity between two images can usually be obtained by calculating the Euclidean distance between their feature vectors.

[0042] Cosine similarity: It evaluates the similarity between two vectors by calculating the cosine of the angle between them. In this scheme, it is used to calculate the similarity between feature data of a certain modality between two objects.

[0043] Matching probability: The probability that a pair (two) objects belong to the same cluster category.

[0044] Indicator function: The letter I represents the indicator function, I(True) = 1, I(False) = 0.

[0045] Detection Tracking ID: In video parsing, the target is tracked across multiple frames. The detection tracking ID identifies the uniqueness of an object in the video over a period of time. However, model judgment errors can lead to the IDs of different objects being swapped in adjacent time periods. In most cases, it can be inferred that if the detection tracking IDs are the same, the images are likely of the same object.

[0046] Bayes' theorem:

[0047] Conditional independence: If P(X,Y|Z)=P(X|Z)P(Y|Z), or equivalently P(X|Y,Z)=P(X|Z), then events X and Y are said to be conditionally independent with respect to a given event Z. That is, when Z occurs, whether X occurs or not is irrelevant to whether Y occurs. In this scheme, multimodal interactions need to satisfy conditional independence with respect to a given category label and a defined scene. For example, whether a given face feature distance / body feature distance is the same person satisfies conditional independence; whether a given face / body feature distance / spatial distance is the same person satisfies conditional independence; however, whether a given detection / tracking ID / spatial distance ID is the same person does not satisfy conditional independence because the spatial distance is very close when the detection / tracking ID is the same. Therefore, when introducing a modality with a detection / tracking ID, it must be limited to the same spatiotemporal range. Within adjacent spatiotemporal ranges, whether the detection / tracking ID and the given spatial distance are the same person satisfies conditional independence.

[0048] Example 1

[0049] According to an embodiment of the present invention, a method embodiment for processing multimodal information of a target object is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0050] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for processing multimodal information of a target object is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0051] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0052] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for processing multimodal information of the target object in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, to implement the method for processing multimodal information of the target object of the application described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0053] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of computer terminal 10. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0054] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0055] Under the aforementioned operating environment, this application provides the following: Figure 2 Method 1 for processing multimodal information of the target object shown. Figure 2 This is a flowchart of a method for processing multimodal information of a target object according to Embodiment 1 of the present invention, as shown below. Figure 2 As shown, the method includes the following steps:

[0056] Step S202: Obtain multimodal information of multiple target objects in a multimodal manner, wherein the multimodal information includes: the metric distance of any target object in each modality;

[0057] Step S204: Based on multimodal information, calculate the posterior matching probability between any two target objects, where the target objects are the objects to be clustered;

[0058] Step S206: Based on the posterior matching probability between any two target objects, perform multimodal object clustering on all target objects to generate clustering results.

[0059] Through the above steps, multimodal information of multiple target objects in a multimodal context is acquired. Based on this multimodal information, the posterior matching probability between any two target objects is calculated. Then, based on this posterior matching probability, all target objects can be clustered in a multimodal manner, generating clustering results. Because the clustering results are obtained based on the posterior matching probability between any two target objects, and the posterior matching probability is derived from multimodal information, it is possible to obtain the posterior matching probability based on more comprehensive information. This solves the technical problem in related clustering methods of difficulty in efficiently and accurately implementing object clustering.

[0060] It should be noted that the target objects mentioned above are the objects to be clustered, i.e., the objects to be classified. A modality refers to any source or form of information; multimodal information refers to information encompassing multiple sources or forms of information. Distance metrics can include Euclidean distance, spatial distance, etc., and can be used to calculate the similarity of a particular modality between two objects.

[0061] As an optional implementation, multimodal information of multiple target objects in a multimodal context is acquired. This multimodal information includes the metric distance of any target object in each modality, with each modality's metric distance condition being independent. It should be noted that the multimodal information should not be identical depending on the scenario. For example, in a city brain scenario where the target object is a person, the acquired multimodal information includes visual information such as human features / gait features / facial features, attribute information, geographic information such as time and space, and metadata such as video detection and tracking IDs. These multiple pieces of information can be referred to as multiple modalities, and the multimodal information includes the metric distances across these multiple modalities. In an e-commerce scenario where the target object is a product, the acquired multimodal information includes image features / text representation, product category, sales volume / reviews, etc. These multiple pieces of information can also be referred to as multiple modalities, and the multimodal information includes the metric distances across these multiple modalities. By acquiring multimodal information of multiple target objects in a multimodal context, the importance of different modalities can be effectively assessed based on this information.

[0062] As an optional implementation, the posterior matching probability between any two target objects is calculated based on multimodal information. This posterior matching probability can be obtained from the metric distance between any two target objects in each modality, as included in the multimodal information. Specifically, the calculation can be performed as follows: extract the metric distance between any two target objects based on their multimodal information; use Bayes' theorem and conditional independence equations to calculate the metric distance between the two target objects, thus obtaining the posterior matching probability. By using Bayes' theorem and conditional independence properties to calculate the posterior matching probability of multimodal data, i.e., calculating the probability that two objects belong to the same class, and because the modalities satisfy conditional independence for a given category and scenario, the matching probability calculation for any multimodal data is achieved. Furthermore, this method replaces the pure Euclidean distance or cosine similarity used in related technologies. The matching probability calculated using multimodal data is more comprehensive than that calculated using Euclidean distance or cosine similarity, significantly improving clustering accuracy and recall. In addition, during the calculation using Bayes' theorem and the conditional independence property, the objects supported for calculation may lack data for one or more modalities. While making full use of the complete multimodal information of the objects to be clustered, attention is also paid to the importance between different modalities.

[0063] As an optional embodiment, multimodal clustering is performed on all target objects based on the posterior matching probability between any two target objects to generate clustering results. Specifically, the process of generating clustering results can be as follows: At least two target objects to be clustered are determined from multiple target objects, and the number of target objects to be clustered is counted, where each target object is labeled with a clustering tag; based on the posterior matching probability of the target objects to be clustered, probabilistic clustering is performed on the target objects to be clustered to generate clustering results. Probabilistic clustering of the target objects to be clustered is achieved by calculating the posterior matching probability. Because it involves acquiring multimodal information of multiple target objects in a multimodal context, processing a large amount of multimodal information to obtain posterior matching probabilities and thus generate clustering results, this application can handle situations with a large amount of modal data. Compared to contrastive learning methods in related technologies, which cannot handle situations with a large number of modalities, this application represents a significant improvement.

[0064] As an optional implementation, the cluster labels of the target objects to be clustered are analyzed based on the joint probability density function to determine the probability loss of probabilistic clustering. It should be noted that because the probability loss is obtained by analyzing the cluster labels of the target objects to be clustered—that is, through the cluster labels marked on the target objects—when determining the probability loss of probabilistic clustering, if there are two target objects to be clustered among multiple target objects, the probability loss for the two target objects belonging to the same category can be calculated only, greatly reducing computational costs.

[0065] As an optional embodiment, the process of probabilistically clustering the target objects to be clustered based on their posterior matching probabilities to generate clustering results may further include the following steps: constructing K-nearest neighbor objects for each target object to be clustered based on their posterior matching probabilities, and initializing each target object to be clustered as a separate category; traversing each target object to be clustered, classifying each target object to be clustered back to its original category, and classifying the category of the corresponding K-nearest neighbor object with the maximum likelihood probability loss function as the lowest category; if the traversal of each target object to be clustered is complete, summarizing the classification results into a clustering result. Specifically, when constructing the K-nearest neighbor objects for each target object to be clustered, each target object to be clustered can be sorted according to its posterior matching probability with other target objects, and the K objects with the highest posterior matching probabilities can be extracted as K-nearest neighbor objects. The above process involves constructing K nearest neighbors for each object based on its posterior matching probability. These K nearest neighbors are sorted in descending order of their posterior matching probability with other objects. The K objects with the highest posterior matching probabilities (K <= m) are selected. Each object is initialized to a separate category. Each object is then iterated through, and each object is adjusted to the category that minimizes the maximum likelihood loss function between its original category and the categories of its K nearest neighbors. This iteration is repeated until no further updates are needed, at which point the probabilistic clustering ends. The results are then summarized into a clustering result. Through these steps, probabilistic clustering can be achieved, reducing the traditional requirement to consider the matching probabilities between all objects to only requiring calculation of the matching probabilities between objects belonging to the same category. This reduces the time complexity of probabilistic clustering from O(N^2) to O(NK), significantly improving clustering efficiency.

[0066] As an optional implementation, if modal information for one or more modalities of the target object is missing, the missing modal information can be supplemented based on clustering results. By observing the relationships between different modalities, the similarity of the missing data can be predicted, thereby enabling the supplementation of missing modal information.

[0067] This application also provides, as follows: Figure 3Method 2 for processing multimodal information of the target object shown. Figure 3 This is a flowchart of a second method for processing multimodal information of a target object according to Embodiment 1 of the present invention, as shown below. Figure 3 As shown, the method includes the following steps:

[0068] Step S302: Obtain multimodal information of multiple target objects in a multimodal context. In the case where the multimodality is the face modality and geographic modality of any target object, the multimodal information includes at least: face similarity and spatial distance.

[0069] Step S304: Based on face similarity and spatial distance, calculate the posterior matching probability between any two target objects, where the target objects are the objects to be clustered;

[0070] Step S306: Based on the posterior matching probability between any two target objects, perform multimodal object clustering on all target objects to generate clustering results.

[0071] Through the above steps, multimodal information of multiple target objects in a multimodal context is obtained. In the case where the multimodality comprises the facial and geographic modalities of any target object, the multimodal information includes at least facial similarity and spatial distance. Based on this multimodal information, the posterior matching probability between any two target objects is calculated. Furthermore, based on this posterior matching probability, all target objects can be clustered into multimodal clusters, generating clustering results. Because the clustering results are derived from the posterior matching probability between any two target objects, and the posterior matching probability is obtained from multimodal information, it is possible to obtain the posterior matching probability based on more comprehensive information. This solves the technical problem in related clustering methods of difficulty in efficiently and accurately achieving object clustering.

[0072] This application also provides, as follows: Figure 4 Method 3 for processing multimodal information of the target object shown. Figure 4 This is a flowchart of a third method for processing multimodal information of a target object according to Embodiment 1 of the present invention, as shown below. Figure 4 As shown, the method includes the following steps:

[0073] Step S402: Obtain multimodal information of multiple target objects in a multimodal context. In the case where the multimodality is the face feature modality and body feature modality of any target object, the multimodal information includes at least: face similarity and body similarity.

[0074] Step S404: Based on face similarity and human body similarity, calculate the posterior matching probability between any two target objects, where the target objects are the objects to be clustered;

[0075] Step S406: Based on the posterior matching probability between any two target objects, perform multimodal object clustering on all target objects to generate clustering results.

[0076] Through the above steps, multimodal information of multiple target objects is obtained in a multimodal manner. In the case where the multimodality consists of facial feature modality and body feature modality for any target object, the multimodal information includes at least facial similarity and body similarity. Based on this multimodal information, the posterior matching probability between any two target objects is calculated. Furthermore, based on this posterior matching probability, all target objects can be clustered in a multimodal manner to generate clustering results. Because the clustering results are obtained based on the posterior matching probability between any two target objects, and the posterior matching probability is derived from multimodal information, it is possible to obtain the posterior matching probability based on more comprehensive information. This solves the technical problem in related clustering methods of difficulty in efficiently and accurately achieving object clustering.

[0077] Based on the above embodiments and optional embodiments, an optional implementation method is provided, which is described in detail below.

[0078] In an optional embodiment of the present invention, an arbitrary multimodal clustering method based on probabilistic clustering is provided. This method uses multimodal posterior matching probability calculation and maximum likelihood clustering of matching probability to cluster multimodal objects, which solves the problems of traditional clustering methods being unable to handle weak modal data, unable to handle missing data, and having high time complexity.

[0079] The method provided in the optional embodiments of the present invention can be widely applied to multimodal clustering scenarios, such as large-scale online clustering of pedestrians, motor vehicles, and non-motor vehicles in cities, large-scale online clustering of short videos, real-time clustering of dynamic data published on social media platforms, and product categorization on shopping platforms. The optional embodiments of the present invention are described in detail below.

[0080] Figure 5 This is a schematic diagram of heterogeneous bimodal posterior matching probability calculation provided by an optional embodiment of the present invention, as shown below. Figure 5 As shown. In face clustering scenarios, some faces often possess geographical information. When an image contains both facial features and geographical information, the posterior matching probability can be calculated based on the Euclidean distance (negatively correlated with cosine similarity) and spatial distance of the facial feature vectors. This can be derived as an atomic distribution using Bayes' theorem and the conditional independence equation. After statistically analyzing the atomic distribution on the labeled data, the posterior probabilities under various combinations of facial similarity / spatial distance are obtained, as shown below. Figure 5 The partial representation of face and spatiotemporal posterior probability is shown, while the posterior matching probability of pure face similarity is as follows. Figure 5The partial posterior probability of pure faces is shown. The effect of adding spatial distance on the pure face matching probability is as follows. Figure 5 As shown in the comparison of the mid-posterior probability difference, it can be seen that when the facial similarity is not high and the spatial distance is close, the addition of geographical information will increase the matching probability; while when the facial similarity is not high and the spatial distance is far, the addition of spatial information will decrease the matching probability; when the facial similarity is very high, the matching probability remains basically unchanged (close to 1.0) regardless of the spatial distance.

[0081] Among them, such as Figure 5 The probability calculation method shown is as follows:

[0082]

[0083] Figure 6 This is a schematic diagram illustrating the calculation of homogeneous bimodal posterior matching probability provided by an optional embodiment of the present invention, as shown below. Figure 6 As shown. In human body clustering scenarios, some human bodies are often associated with faces. When an image contains both face and human body features, the posterior matching probability based on the Euclidean distance (negatively correlated with cosine similarity) of the face / human body feature vectors can be calculated. This can be derived as an atomic distribution using Bayes' theorem and the conditional independence equation. After statistically analyzing the atomic distribution on the labeled data, the posterior probabilities for various combinations of face / human body similarity are obtained, as shown below. Figure 6 The partial posterior probability of face-body association is shown, and the posterior matching probability of pure face similarity is as follows: Figure 6 The partial posterior probability of pure face association is shown. The impact of adding human body information on the pure face matching probability is as follows. Figure 6 As shown in the comparison of the mid-posterior probability difference, it can be seen that when the face similarity is not high and the body similarity is high, the addition of body information will increase the matching probability; when the similarity between the two is not high, the addition of body information will decrease the matching probability; when the face similarity is very high, the matching probability remains basically unchanged (close to 1.0) regardless of the body similarity.

[0084] Among them, such as Figure 6 The probability calculation method shown is as follows:

[0085]

[0086] Figure 7 This is a schematic diagram of probabilistic clustering provided by an optional embodiment of the present invention. In the diagram, solid and hollow circle objects represent two categories, the numerical values ​​between objects represent the similarity between them, and the sets enclosed by circles, ellipses, and triangles represent possible clustering results. Figure 7 As can be seen, according to the above method, the clustering accuracy of mode 1 + mode 2 is higher after using the matching probability of multimodal.

[0087] (I) The following is a detailed explanation of the multimodal posterior matching probability calculation process:

[0088] Define d k e is the metric distance for the k-th mode (the metric distances for each mode are conditionally independent). ij Indicate whether object i and object j belong to the same category. Calculate the posterior matching probability using Bayesian probability and conditional independence.

[0089] Taking two modes as an example, the calculation is as follows:

[0090]

[0091]

[0092] Taking three modes as an example, the calculation is as follows:

[0093]

[0094] Extending to any modality, the calculation is as follows:

[0095]

[0096] As mentioned above, the posterior matching probability form for any modality only contains P(d k |e ij =0 / 1) and P(e ij =0 / 1|d k From the data of the objects to be clustered, a portion of labeled samples are taken, and kernel density estimation is performed on these labeled samples to solve for P(d). k |e ij =0 / 1), use monotonic regression to train and solve P(e) ij =0 / 1|d k ).

[0097] Therefore, the posterior matching probability between two objects of any modality can be solved using the above formula, replacing the distance metric for a single modality.

[0098] For detection tracking IDs, in the case of the same location / adjacent time, detection tracking ID distance information is introduced: d track =0 (if the two images have the same tracking ID) / 1 (if the tracking IDs are different), P(e ij |d track =0 / 1) can be estimated based on the reliability of the detection and tracking.

[0099] (II) The following is a detailed introduction to probabilistic clustering:

[0100] The general steps of probabilistic clustering are as follows: First, recall the top m similar objects using a partial single model (such as feature vectors), calculate the posterior matching probability between the target object and the top m using multimodal posterior matching probabilities, construct the K nearest neighbors of each object based on the posterior matching probability (sorted from largest to smallest according to the posterior matching probability between the object and other objects, and take the K objects with the highest posterior matching probability, K<=m), and initialize each object as a separate category; Second, traverse each object, and adjust each object to the category that minimizes the maximum likelihood probability loss function among the original category and the categories of the K nearest neighbors; Third, repeat step two until there are no more updates, and the probabilistic clustering ends.

[0101] The calculation process for probabilistic clustering is as follows: Define π: [z1, z2, ..., z n ] represents the clustering label, where n is the number of samples to be clustered, and z is the number of clustering objects. i Let be the cluster label for the i-th object. The joint probability density function is defined as follows:

[0102]

[0103] Where I is the indicator function, indicating whether two labels are the same label, and Ω is the normalization function. The maximum likelihood probability is defined as follows:

[0104]

[0105] Based on the definition of maximum natural probability, the probabilistic clustering loss function is defined as follows:

[0106] loss=-logP(X|π=-∑ i，j I(z i =z j )(logP(e ij =1)-logP(e ij =0))+const;

[0107] It should be noted that the calculation of similarity between multimodal objects can also train a machine learning model. The input is the multimodal information of the two objects or the similarity information of each modality. Missing modal information is specially labeled, and the output is the matching probability. Traditional machine learning methods or neural networks can be used for prediction.

[0108] The probabilistic clustering method provided by the optional embodiments of this invention solves the problem of calculating all similarities between pairs of objects in traditional probabilistic clustering methods. During the optimization process, the loss function definition is optimized as follows, reducing the computational complexity from O(N^2) in loss function definition 1 to O(NK) in loss function definition 2.

[0109] In the loss function definition 2, the optional implementation of the present invention divides the pairwise relationship into two parts: inter-class and intra-class. The product of the probabilities of all pairs of objects that do not belong to the same class is split into a constant term to obtain the loss function definition. Only the matching probability between pairs of objects within the same class needs to be calculated.

[0110] Define π as: [z1, z2, ..., z n [The result is the cluster category ID.]

[0111] Define the joint probability density:

[0112] I is an indicator function

[0113] Using maximum likelihood probability:

[0114]

[0115] Loss function definition 1:

[0116]

[0117] Let {A1, A2, ..., A} k If the K classes are obtained by π-segmentation, then the loss can be rewritten as:

[0118]

[0119] Loss function definition 2:

[0120]

[0121]

[0122] The following beneficial effects can be achieved through the above optional implementation methods:

[0123] (1) An optional embodiment of the present invention enables clustering of objects with arbitrary multimodality and missing data.

[0124] (2) The optional embodiments of the present invention can be widely applied to multimodal clustering scenarios, such as large-scale online clustering of pedestrians, motor vehicles and non-motor vehicles in cities, large-scale online clustering of short videos, real-time clustering of dynamics published on social platforms, and product classification on shopping platforms.

[0125] (3) In an optional embodiment of the present invention, the posterior matching probability between objects to be clustered is calculated using Bayesian probability and conditional independence based on the multimodal information of the objects. That is, the probability that two objects belong to the same class is calculated, replacing the pure Euclidean distance or cosine similarity used by mainstream methods. The matching probability calculated using multimodal methods is more sufficient than that calculated by Euclidean distance or cosine similarity, and the clustering accuracy and recall are significantly improved.

[0126] (4) In the optional implementation of the present invention, the KNN graph is constructed using the posterior matching probability. When using maximum likelihood clustering, in the definition of the maximum likelihood probability loss function, it is only necessary to calculate the probability loss when two objects in the same class belong to the same category, which reduces the complexity of the probability clustering algorithm from O(N^2) to O(NK). Compared with the solution in related technologies (N^2 computational complexity), the running time is greatly reduced, the efficiency is improved, and big data clustering is supported.

[0127] (5) The optional embodiments of the present invention utilize multimodal information to calculate the posterior matching probability. As can be seen from the formula for calculating the posterior matching probability of any modality, the formula supports the calculation of the object for which one or more modal data are missing. It can make full use of the complete multimodal information of the object to be clustered, while also paying attention to the importance between different modalities.

[0128] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that the method for processing multimodal information of the target object according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0130] Example 2

[0131] According to an embodiment of the present invention, an apparatus for implementing the above-described method for processing multimodal information of a target object is also provided. Figure 8 This is a structural block diagram of a multimodal information processing device for a target object provided in Embodiment 2 of the present invention, as shown in the figure. Figure 8 As shown, the device includes a first acquisition module 802, a first calculation module 804, and a first clustering module 806. The device will be described below.

[0132] The first acquisition module 802 is used to acquire multimodal information of multiple target objects in a multimodal context, wherein the multimodal information includes: the metric distance of any target object in each modality; the first calculation module 804 is connected to the first acquisition module 802 and is used to calculate the posterior matching probability between any two target objects based on the multimodal information, wherein the target objects are objects to be clustered; the first clustering module 806 is connected to the first calculation module 804 and is used to perform multimodal object clustering on all target objects based on the posterior matching probability between any two target objects, and generate clustering results.

[0133] It should be noted that the first acquisition module 802, the first calculation module 804, and the first clustering module 806 mentioned above correspond to steps S202 to S206 in Embodiment 1. Multiple modules implement the same instances and application scenarios as their corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal provided in Embodiment 1.

[0134] Example 3

[0135] According to an embodiment of the present invention, an apparatus for implementing the second method for processing multimodal information of the target object is also provided. Figure 9 This is a structural block diagram of the second multimodal information processing device for the target object provided in Embodiment 3 of the present invention, as shown in the figure. Figure 9 As shown, the device includes: a second acquisition module 902, a second calculation module 904, and a second clustering module 906. The device will be described below.

[0136] The second acquisition module 902 is used to acquire multimodal information of multiple target objects in a multimodal context. When the multimodality is the face modality and geographic modality of any target object, the multimodal information includes at least: face similarity and spatial distance. The second calculation module 904, connected to the second acquisition module 902, is used to calculate the posterior matching probability between any two target objects based on face similarity and spatial distance, where the target objects are objects to be clustered. The second clustering module 906, connected to the second calculation module 904, is used to cluster all target objects in a multimodal context based on the posterior matching probability between any two target objects, generating clustering results.

[0137] It should be noted that the second acquisition module 902, the second calculation module 904, and the second clustering module 906 mentioned above correspond to steps S302 to S306 in Embodiment 1. Multiple modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal provided in Embodiment 1.

[0138] Example 4

[0139] According to an embodiment of the present invention, an apparatus for implementing the third method for processing multimodal information of the target object described above is also provided. Figure 10 This is a structural block diagram of the device three for processing multimodal information of a target object according to Embodiment 4 of the present invention, as shown in the figure. Figure 10 As shown, the device includes a third acquisition module 1002, a third calculation module 1004, and a third clustering module 1006. The device will be described below.

[0140] The third acquisition module 1002 is used to acquire multimodal information of multiple target objects in a multimodal context. When the multimodality is the face feature modality and body feature modality of any target object, the multimodal information includes at least: face similarity and body similarity. The third calculation module 1004, connected to the third acquisition module 1002, is used to calculate the posterior matching probability between any two target objects based on face similarity and body similarity, where the target objects are objects to be clustered. The third clustering module 1006, connected to the third calculation module 1004, is used to cluster all target objects using multimodal objects based on the posterior matching probability between any two target objects, generating clustering results.

[0141] It should be noted that the third acquisition module 1002, the third calculation module 1004, and the third clustering module 1006 mentioned above correspond to steps S402 to S406 in Embodiment 1. Multiple modules implement the same instances and application scenarios as their corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal provided in Embodiment 1.

[0142] Example 5

[0143] Embodiments of the present invention can provide a computer terminal, which can be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the computer terminal can also be replaced by a mobile terminal or other terminal device.

[0144] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0145] In this embodiment, the computer terminal described above can execute the following steps of the method for processing multimodal information of target objects in an application: obtaining multimodal information of multiple target objects in multiple modes, wherein the multimodal information includes: the metric distance of any target object in each mode; calculating the posterior matching probability between any two target objects based on the multimodal information, wherein the target objects are objects to be clustered; and clustering all target objects into multimodal objects based on the posterior matching probability between any two target objects to generate clustering results.

[0146] Optionally, Figure 11 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Figure 11 As shown, the computer terminal may include one or more (only one is shown in the figure) processors 111, memory 112, and other modules.

[0147] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and apparatus for processing and detecting multimodal information of the target object in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned method for processing multimodal information of the target object. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0148] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring multimodal information of multiple target objects in a multimodal manner, wherein the multimodal information includes: the metric distance of any target object in each modality; calculating the posterior matching probability between any two target objects based on the multimodal information, wherein the target objects are objects to be clustered; and clustering all target objects into multimodal objects based on the posterior matching probability between any two target objects to generate clustering results.

[0149] Optionally, the processor may also execute program code that performs the following steps: Calculates the posterior matching probability between any two target objects based on multimodal information, including: extracting the metric distance between any two target objects based on the multimodal information of any two target objects, wherein the metric distance includes at least one of the following: Euclidean distance and spatial distance; collecting Bayes' formula and conditional independence equation, calculating the metric distance between any two target objects, and obtaining the posterior matching probability between any two target objects.

[0150] Optionally, the processor may also execute program code that performs the following steps: clustering all target objects into multimodal objects based on the posterior matching probability between any two target objects, generating clustering results including: determining at least two target objects to be clustered from multiple target objects, and counting the number of target objects to be clustered, wherein the target objects to be clustered are labeled with clustering labels; performing probabilistic clustering of the target objects to be clustered based on the posterior matching probability of the target objects to be clustered, and generating clustering results.

[0151] Optionally, the processor may also execute program code with the following steps: performing probabilistic clustering on the target objects to be clustered based on their posterior matching probabilities, and generating clustering results, including: constructing K-nearest neighbor objects for each target object to be clustered based on their posterior matching probabilities, and initializing each target object to be clustered as a separate category; traversing each target object to be clustered, classifying each target object to be clustered back to its original category, and classifying the maximum likelihood probability loss function of the corresponding K-nearest neighbor object's category as the lowest category; and if the traversal of each target object to be clustered is complete, summarizing the classification results into clustering results.

[0152] Optionally, the processor may also execute program code that performs the following steps: constructing K nearest neighbor objects for each target object to be clustered, including: sorting each target object to be clustered according to the posterior matching probability between each target object and other target objects, and extracting the K objects with the highest posterior matching probability as K nearest neighbor objects.

[0153] Optionally, the processor may also execute program code that performs the following steps: The method further includes: analyzing the cluster labels of the target objects to be clustered based on the joint probability density function, and determining the probability loss of probabilistic clustering.

[0154] Optionally, the processor may also execute program code that performs the following steps: The method further includes: if there is missing modal information in one or more modalities of the target object, supplementing the modal information with missing data based on the clustering results.

[0155] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring multimodal information of multiple target objects in a multimodal context, wherein, in the case where the multimodality is the face modality and geographic modality of any target object, the multimodal information includes at least: face similarity and spatial distance; calculating the posterior matching probability between any two target objects based on face similarity and spatial distance, wherein the target objects are objects to be clustered; and clustering all target objects into multimodal objects based on the posterior matching probability between any two target objects to generate clustering results.

[0156] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring multimodal information of multiple target objects in a multimodal context, wherein, in the case where the multimodality is the facial feature modality and the human body feature modality of any target object, the multimodal information includes at least: facial similarity and human body similarity; calculating the posterior matching probability between any two target objects based on facial similarity and human body similarity, wherein the target objects are objects to be clustered; and clustering all target objects into multimodal objects based on the posterior matching probability between any two target objects to generate clustering results.

[0157] Those skilled in the art will understand that Figure 11 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 11 This does not limit the structure of the aforementioned electronic device. For example, computer terminal 11 may also include components that are more... Figure 11 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 11 The different configurations shown.

[0158] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0159] Example 6

[0160] Embodiments of the present invention also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the method for processing multimodal information of the target object provided in Embodiment 1.

[0161] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0162] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining multimodal information of multiple target objects in a multimodal manner, wherein the multimodal information includes: the metric distance of any target object in each modality; calculating the posterior matching probability between any two target objects based on the multimodal information, wherein the target objects are objects to be clustered; and clustering all target objects into multimodal objects based on the posterior matching probability between any two target objects to generate clustering results.

[0163] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: calculating the posterior matching probability between any two target objects based on multimodal information, including: extracting the metric distance between any two target objects based on the multimodal information of any two target objects, wherein the metric distance includes at least one of the following: Euclidean distance and spatial distance; collecting Bayes' formula and conditional independence equation, calculating the metric distance between any two target objects, and obtaining the posterior matching probability between any two target objects.

[0164] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: clustering all target objects into multimodal objects based on the posterior matching probability between any two target objects, and generating clustering results, including: determining at least two target objects to be clustered from multiple target objects, and counting the number of target objects to be clustered, wherein the target objects to be clustered are labeled with clustering labels; performing probabilistic clustering on the target objects to be clustered based on the posterior matching probability of the target objects to be clustered, and generating clustering results.

[0165] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: performing probabilistic clustering on the target objects to be clustered based on the posterior matching probability of the target objects to be clustered, and generating clustering results, including: constructing K nearest neighbor objects for each target object to be clustered according to the posterior matching probability of the target objects to be clustered, and initializing each target object to be clustered as a separate category; traversing each target object to be clustered, classifying each target object to be clustered back to its original category, and classifying the maximum likelihood probability loss function of the corresponding K nearest neighbor object's category as the lowest category; if the traversal of each target object to be clustered is complete, summarizing the classification results into a clustering result.

[0166] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: constructing K nearest neighbor objects for each target object to be clustered, including: sorting each target object to be clustered according to the posterior matching probability between each target object to be clustered and other target objects, and extracting the K objects with the highest posterior matching probability as K nearest neighbor objects.

[0167] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the method further includes: analyzing the clustering labels of the target objects to be clustered based on the joint probability density function to determine the probability loss of probabilistic clustering.

[0168] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the method further includes: if there is data loss in modal information under one or more modalities of the target object, supplementing the modal information with data loss based on the clustering results.

[0169] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the method further includes: acquiring multimodal information of multiple target objects in a multimodal context, wherein, in the case where the multimodality is the face modality and geographic modality of any one target object, the multimodal information includes at least: face similarity and spatial distance; calculating the posterior matching probability between any two target objects based on face similarity and spatial distance, wherein the target objects are objects to be clustered; and clustering all target objects into multimodal objects based on the posterior matching probability between any two target objects to generate clustering results.

[0170] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the method further includes: acquiring multimodal information of multiple target objects in a multimodal context, wherein, when the multimodality is the face feature modality and body feature modality of any one target object, the multimodal information includes at least: face similarity and body similarity; calculating the posterior matching probability between any two target objects based on face similarity and body similarity, wherein the target objects are objects to be clustered; and clustering all target objects into multimodal objects based on the posterior matching probability between any two target objects to generate clustering results.

[0171] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0172] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0174] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0176] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0177] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for processing multimodal information of a target object, characterized in that, include: Acquire multimodal information of multiple target objects in a multimodal context, wherein the multimodal information includes: the metric distance of any target object in each modality; Based on the multimodal information, the posterior matching probability between any two target objects is calculated, wherein the target objects are objects to be clustered; Based on the posterior matching probability between any two target objects, multimodal object clustering is performed on all target objects by calculating the matching probability between objects belonging to the same class, generating clustering results. This includes: constructing K-nearest neighbor objects for each target object to be clustered based on the posterior matching probability of the target objects to be clustered, and initializing each target object to be clustered as a separate category; traversing each target object to be clustered, classifying each target object to be clustered back to its original category, and classifying the category with the maximum likelihood probability loss function of the corresponding K-nearest neighbor object as the lowest-level category; if the traversal of each target object to be clustered is complete, summarizing the classification results into the clustering results. The calculation of the posterior matching probability between any two target objects based on the multimodal information includes: when there is partial missing multimodal information of one of the target objects, the calculation of the posterior matching probability between the two target objects based on the partially missing multimodal information.

2. The method according to claim 1, characterized in that, Based on the multimodal information, the posterior matching probability between any two target objects is calculated, including: Based on the multimodal information of any two target objects, extract the metric distance between the two target objects, wherein the metric distance includes at least one of the following: Euclidean distance and spatial distance; By collecting Bayes' theorem and the conditional independence equation, the metric distance between any two target objects is calculated to obtain the posterior matching probability between the two target objects.

3. The method according to claim 1 or 2, characterized in that, Based on the posterior matching probability between any two target objects, multimodal object clustering is performed on all target objects, generating clustering results including: From the plurality of target objects, at least two target objects to be clustered are determined, and the number of target objects to be clustered is counted, wherein the target objects to be clustered are labeled with clustering labels; Based on the posterior matching probability of the target objects to be clustered, probabilistic clustering is performed on the target objects to be clustered to generate the clustering result.

4. The method according to claim 1, characterized in that, Constructing the K nearest neighbor objects for each target object to be clustered includes: sorting the target objects for each target object to be clustered according to the posterior matching probability between each target object and other target objects, and extracting the K objects with the highest posterior matching probability as the K nearest neighbor objects.

5. The method according to claim 3, characterized in that, The method further includes: Based on the joint probability density function, the clustering labels of the target objects to be clustered are analyzed to determine the probability loss of the probabilistic clustering.

6. The method according to claim 1, characterized in that, The method further includes: if there is data loss in modal information under one or more modalities of the target object, supplementing the modal information with data loss based on the clustering results.

7. A method for processing multimodal information of a target object, characterized in that, include: Acquire multimodal information of multiple target objects in a multimodal context, wherein, when the multimodality is the facial modality and geographic modality of any target object, the multimodal information includes at least: facial similarity and spatial distance; Based on the facial similarity and spatial distance, the posterior matching probability between any two target objects is calculated, wherein the target objects are objects to be clustered; Based on the posterior matching probability between any two target objects, multimodal object clustering is performed on all target objects by calculating the matching probability between objects belonging to the same class, generating clustering results. This includes: constructing K-nearest neighbor objects for each target object to be clustered based on the posterior matching probability of the target objects to be clustered, and initializing each target object to be clustered as a separate category; traversing each target object to be clustered, classifying each target object to be clustered back to its original category, and classifying the category with the maximum likelihood probability loss function of the corresponding K-nearest neighbor object as the lowest-level category; if the traversal of each target object to be clustered is complete, summarizing the classification results into the clustering results. Specifically, the posterior matching probability between any two target objects is calculated based on the facial similarity and spatial distance, including: when there is a partial missing facial similarity and spatial distance between the target objects, the posterior matching probability between the two target objects is calculated based on the partially missing facial similarity and spatial distance.

8. A method for processing multimodal information of a target object, characterized in that, include: Acquire multimodal information of multiple target objects in a multimodal context, wherein, when the multimodality is the facial feature modality and human body feature modality of any target object, the multimodal information includes at least: facial similarity and human body similarity; Based on the face similarity and body similarity, the posterior matching probability between any two target objects is calculated, wherein the target objects are objects to be clustered; Based on the posterior matching probability between any two target objects, multimodal object clustering is performed on all target objects by calculating the matching probability between objects belonging to the same class, generating clustering results. This includes: constructing K-nearest neighbor objects for each target object to be clustered based on the posterior matching probability of the target objects to be clustered, and initializing each target object to be clustered as a separate category; traversing each target object to be clustered, classifying each target object to be clustered back to its original category, and classifying the category with the maximum likelihood probability loss function of the corresponding K-nearest neighbor object as the lowest-level category; if the traversal of each target object to be clustered is complete, summarizing the classification results into the clustering results. Specifically, the posterior matching probability between any two target objects is calculated based on the face similarity and body similarity, including: when some of the face similarity and body similarity of the target objects are missing, the posterior matching probability between the two target objects is calculated based on the partially missing face similarity and body similarity.

9. A device for processing multimodal information of a target object, characterized in that, include: The first acquisition module is used to acquire multimodal information of multiple target objects in a multimodal manner, wherein the multimodal information includes: the metric distance of any target object in each modality; The first calculation module is used to calculate the posterior matching probability between any two target objects based on the multimodal information, wherein the target objects are objects to be clustered; The first clustering module is used to perform multimodal object clustering on all target objects based on the posterior matching probability between any two target objects, by calculating the matching probability between objects belonging to the same class, and generating clustering results. The first clustering module is further used to construct K-nearest neighbor objects for each target object to be clustered based on the posterior matching probability of the target objects to be clustered, and initialize each target object to be clustered as a separate category; iterate through each target object to be clustered, classify each target object to be clustered back to its original category, and classify the category with the maximum likelihood probability loss function of the corresponding K-nearest neighbor object as the lowest category; if the iteration of each target object to be clustered is complete, summarize the classification results into the clustering results. The first calculation module is further configured to calculate the posterior matching probability between any two target objects based on the partially missing multimodal information when the multimodal information of one of the target objects is partially missing.

10. A device for processing multimodal information of a target object, characterized in that, include: The second acquisition module is used to acquire multimodal information of multiple target objects in a multimodal context. In the case where the multimodality is the face modality and geographic modality of any target object, the multimodal information includes at least: face similarity and spatial distance. The second calculation module is used to calculate the posterior matching probability between any two target objects based on the face similarity and spatial distance, wherein the target objects are objects to be clustered; The second clustering module is used to perform multimodal object clustering on all target objects based on the posterior matching probability between any two target objects, by calculating the matching probability between objects belonging to the same class, and generating clustering results. The second clustering module is also used to construct the K nearest neighbor objects for each target object to be clustered based on the posterior matching probability of the target objects to be clustered, and initialize each target object to be clustered as a separate class; iterate through each target object to be clustered, classify each target object to be clustered back to its original class, and classify the class with the maximum likelihood probability loss function of the corresponding K nearest neighbor object's class as the lowest class; if the iteration of each target object to be clustered is complete, summarize the classification results into the clustering results. The second calculation module is further configured to calculate the posterior matching probability between any two target objects based on the partially missing face similarity and spatial distance when the face similarity and spatial distance of the target objects are partially missing in any two target objects.

11. A device for processing multimodal information of a target object, characterized in that, include: The third acquisition module is used to acquire multimodal information of multiple target objects in a multimodal context. In the case where the multimodality is the face feature modality and body feature modality of any target object, the multimodal information includes at least: face similarity and body similarity. The third calculation module is used to calculate the posterior matching probability between any two target objects based on the face similarity and body similarity, wherein the target objects are objects to be clustered; The third clustering module is used to perform multimodal object clustering on all target objects based on the posterior matching probability between any two target objects, by calculating the matching probability between objects belonging to the same class, and generating clustering results. The third clustering module is also used to construct K-nearest neighbor objects for each target object to be clustered based on the posterior matching probability of the target objects to be clustered, and initialize each target object to be clustered as a separate category; iterate through each target object to be clustered, classify each target object to be clustered back to its original category, and classify the category with the maximum likelihood probability loss function of the corresponding K-nearest neighbor object as the lowest category; if the iteration of each target object to be clustered is complete, summarize the classification results into the clustering results. The third calculation module is further configured to calculate the posterior matching probability between any two target objects based on the partially missing face similarity and body similarity when the face similarity and body similarity of the target objects are partially missing in any two target objects.

12. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the method for processing multimodal information of the target object as described in any one of claims 1 to 8.

13. A computer device, characterized in that, include: Memory and processor The memory stores computer programs; The processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, the processor performs the method for processing multimodal information of the target object as described in any one of claims 1 to 8.

Images