Data processing method and device, computer device, and storage medium

Abstract

The application discloses a data processing method and device, computer equipment and a storage medium, and belongs to the technical field of computers. The application uses the average values of the feature data under various modes to complete the missing feature values in the incomplete multi-modal data, so that the completed feature data and the original feature data can be used to align the feature data of different modes, facilitate accurate reconstruction of shared features representing common information between the feature data of different modes, and thus the processing accuracy for the incomplete multi-modal data can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the rapid development of computer and multimedia technologies, feature data of the same entity can be collected from different data sources or channels, giving rise to multimodal data. In this context, each source or form of information can be called a modality. For example, a webpage typically possesses feature data in two modalities: text and links; a news article typically possesses feature data in multiple modalities, consisting of translated versions in different languages; and an image typically possesses feature data in multiple modalities, such as texture, color, and histograms.

[0003] In the application of multimodal data, data gaps often occur, such as missing feature data of a certain modality or missing part of the feature data of a certain modality. This results in the data to be processed often appearing in an incomplete and multimodal form, leading to insufficient usable information. Therefore, how to process incomplete multimodal data has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a data processing method, apparatus, computer device, and storage medium capable of accurately extracting shared features among incomplete multimodal data. The technical solution is as follows:

[0005] On the one hand, a data processing method is provided, which includes:

[0006] Acquire multiple feature data belonging to multiple modalities, wherein at least one modality among the multiple modalities contains missing feature values ​​in its feature data, and any one modality among the multiple modalities corresponds to one or more feature data;

[0007] For the at least one modality containing missing feature values, based on the average value of the feature data corresponding to the at least one modality, the missing feature values ​​in the feature data corresponding to the at least one modality are filled in to obtain the filled feature of the at least one modality;

[0008] Based on the completion features of the at least one modality and the feature data of the modalities other than the at least one modality, a shared feature of the feature data of the multiple modalities is obtained. The shared feature is used to represent the common information among the feature data of the multiple modalities respectively.

[0009] On one hand, a data processing apparatus is provided, the apparatus comprising:

[0010] The first acquisition module is used to acquire multiple feature data belonging to multiple modalities, wherein the feature data corresponding to at least one modality among the multiple modalities contains missing feature values, and any one modality among the multiple modalities corresponds to one or more feature data.

[0011] The completion module is used to, for the at least one modality containing missing feature values, complete the missing feature values ​​in the feature data corresponding to the at least one modality based on the average value of the feature data corresponding to the at least one modality, so as to obtain the completed feature of the at least one modality;

[0012] The second acquisition module is used to acquire shared features of the multiple feature data of the multiple modalities based on the completion features of the at least one modality and the feature data of the modalities other than the at least one modality. The shared features are used to represent common information among the multiple feature data belonging to the multiple modalities respectively.

[0013] In one possible implementation, the completion module includes:

[0014] The acquisition submodule is used to acquire the average value of the feature data corresponding to any one of the at least one modalities containing missing feature values.

[0015] The filling submodule is used to fill the missing feature values ​​in the feature data corresponding to any modality with the average value of the feature data corresponding to any modality, so as to obtain the complete feature of any modality.

[0016] In one possible implementation, the device further includes:

[0017] The regularization module is used to perform regularization processing on the multiple feature data to obtain multiple regularized feature data.

[0018] The completion module is further configured to perform the step of completing missing feature values ​​and obtaining completed features based on the multiple regularized feature data.

[0019] In one possible implementation, the feature data corresponding to any one of the plurality of modalities corresponds to a plurality of categories respectively;

[0020] The second acquisition module includes:

[0021] The linear combination submodule is used to linearly combine the feature values ​​or the average value of the completions belonging to the same category for the feature data or completion features of any of the multiple modalities, so as to obtain the self-expressive features of the multiple categories under any of the modalities.

[0022] The clustering submodule is used to cluster the multiple self-expressive features of the multiple categories under the multiple modalities to obtain the shared features.

[0023] In one possible implementation, the clustering submodule includes:

[0024] The construction unit is used to construct multiple affinity graphs of the multiple categories under the multiple modalities based on the multiple self-expressive features. Each affinity graph includes multiple nodes and multiple weighted edges. Each node is used to represent a self-expressive feature of a category under the corresponding modality. The weight carried by any weighted edge is used to represent the similarity between the two self-expressive features of the two categories corresponding to the two nodes of the weighted edge under the corresponding modality.

[0025] The spectral clustering unit is used to perform spectral clustering operations on the multiple self-expressed features based on the multiple affinity graphs to obtain the shared features.

[0026] In one possible implementation, the spectral clustering unit includes:

[0027] The first acquisition subunit is used to acquire multiple graph Laplacian matrices of the multiple affinity graphs for any given clustering process;

[0028] The second acquisition subunit is used to acquire multiple clustering indicator matrices of the multiple modalities based on the multiple graph Laplacian matrices;

[0029] The third acquisition subunit is used to acquire the loss function value of any clustering process based on the reconstruction error of the multiple self-expressive features, the clustering error of any clustering process, and the similarity difference between the multiple clustering indicator matrices and the features to be solved.

[0030] An iterative adjustment subunit is used to iteratively adjust the parameters of the feature to be solved until the loss function value meets the stopping condition, at which point the iteration stops and the feature to be solved at the last iteration is obtained as the shared feature.

[0031] In one possible implementation, the device further includes:

[0032] The third acquisition module is used to acquire a target weight matrix corresponding to any one of the multiple modalities, wherein the weight coefficients of the feature values ​​that are not missing in any one modality in the target weight matrix are greater than the weight coefficients of the feature values ​​that are missing.

[0033] The third acquisition subunit is further configured to, during the process of acquiring the loss function value, call the target weight matrix to weight the reconstruction error of the self-expressive feature corresponding to any modality.

[0034] In one possible implementation, the target weight matrix is ​​a diagonal weight matrix, the weight coefficient of the non-missing feature values ​​is 1, and the weight coefficient of the missing feature values ​​is the ratio of the number of samples with non-missing feature values ​​in any modality to the total number of samples.

[0035] In one possible implementation, the first acquisition subunit is used to:

[0036] Based on any one of the multiple affinity graphs, obtain the similarity matrix and diagonal matrix of any one affinity graph;

[0037] The difference between the diagonal matrix and the similarity matrix is ​​obtained as the graph Laplacian matrix of any affinity graph.

[0038] In one possible implementation, the iterative adjustment subunit is used for:

[0039] Based on the alternating direction multiplier method, the values ​​of multiple intermediate variables in the loss function are iteratively adjusted to obtain the adjusted intermediate variables. When adjusting the value of any intermediate variable, the values ​​of the intermediate variables other than the adjusted intermediate variable are kept unchanged.

[0040] Keeping the adjusted values ​​of the intermediate variables unchanged, perform the step of iteratively adjusting the parameters of the feature to be solved.

[0041] In one possible implementation, the stopping condition is that the difference between the loss function value of any clustering process and the loss function value of the previous clustering process is less than a loss threshold.

[0042] In one possible implementation, the feature data corresponding to any one of the multiple modalities corresponds to multiple categories, the multiple categories being multiple target objects, and the multiple modalities of the multiple target objects at least include visual information and near-infrared information of the face images of each of the multiple target objects.

[0043] On one hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the at least one computer program being loaded and executed by the one or more processors to implement the data processing method as described above.

[0044] On the one hand, a storage medium is provided that stores at least one computer program, which is loaded and executed by a processor to implement the data processing method described above.

[0045] On one hand, a computer program product or computer program is provided, the computer program product or computer program comprising one or more lines of program code stored in a computer-readable storage medium. One or more processors of a computer device are capable of reading the one or more lines of program code from the computer-readable storage medium, and the one or more processors execute the one or more lines of program code, enabling the computer device to perform the aforementioned data processing method.

[0046] The beneficial effects of the technical solutions provided in this application include at least the following:

[0047] By utilizing the average value of feature data under each modality, missing feature values ​​in incomplete multimodal data can be supplemented. By using the supplemented feature and the original feature data, feature data of different modalities can be aligned, making it easier to accurately reconstruct shared features that represent common information between feature data of different modalities, thereby improving the processing accuracy of incomplete multimodal data. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of the implementation environment of a data processing method provided in an embodiment of this application;

[0050] Figure 2 This is a flowchart of a data processing method provided in an embodiment of this application;

[0051] Figure 3 This is a flowchart of a data processing method provided in an embodiment of this application;

[0052] Figure 4 This is a schematic flowchart illustrating the principle of a data processing method provided in an embodiment of this application;

[0053] Figure 5 This is a schematic flowchart illustrating the principle of a data processing method provided in an embodiment of this application;

[0054] Figures 6 to 12 The selection of model parameter λ1 is shown respectively;

[0055] Figures 13 to 19 The choices of model parameters λ3 and λ4 are shown respectively;

[0056] Figure 20 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0057] Figure 21 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0059] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0060] In this application, the term "at least one" means one or more, and "multiple" means two or more, for example, multiple first positions means two or more first positions.

[0061] Before introducing the embodiments of this application, it is necessary to introduce some basic concepts in the field of cloud technology, which will be introduced below.

[0062] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. It encompasses network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on cloud computing business models. These technologies can form resource pools, allowing for on-demand, flexible, and convenient use. Cloud computing technology will become a crucial support in the cloud technology field. Backend services of technical network systems require substantial computing and storage resources, such as those for video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring data to be transmitted to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, all of which can be achieved through cloud computing.

[0063] Cloud computing is a computing model that distributes computing tasks across a large pool of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, the resources in the "cloud" are infinitely scalable, readily available, on-demand, and expandable, with payment based on usage.

[0064] As a provider of fundamental cloud computing capabilities, we establish a cloud resource pool (referred to as a cloud platform, generally called IaaS, or Infrastructure as a Service) platform. Within this resource pool, we deploy various types of virtual resources for external customers to choose from. The cloud resource pool primarily includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices.

[0065] Based on logical function, a PaaS (Platform as a Service) layer can be deployed on top of the IaaS layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. Alternatively, SaaS can be deployed directly on top of IaaS. PaaS is a platform for running software, such as databases and web (web page) containers. SaaS refers to various types of business software, such as web portals and bulk SMS senders. Generally speaking, SaaS and PaaS are upper layers compared to IaaS.

[0066] Cloud storage is a new concept that has been extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology and distributed storage file systems to bring together a large number of storage devices of various types in the network (storage devices are also called storage nodes) to work together through application software or application interfaces to provide data storage and business access functions to the outside world.

[0067] Currently, the storage method of storage systems is as follows: Logical volumes are created. During the creation of a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID entity, ID). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0068] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0069] Artificial Intelligence as a Service (AIaaS): Also known as AI (Artificial Intelligence) as a service, this is currently a mainstream service model for artificial intelligence platforms. Specifically, AIaaS platforms break down several common AI services and provide them as independent or packaged services in the cloud. This service model is similar to opening an AI-themed marketplace: all developers can access and use one or more AI services provided by the platform through API interfaces. Some experienced developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and maintain their own dedicated cloud AI services.

[0070] This application's embodiments involve the intersection of cloud computing, cloud storage, and AIaaS. Users can upload incomplete multimodal data (i.e., feature data of multiple modalities with missing feature values) to a cloud server. The cloud server, in a cloud computing mode, uses this incomplete multimodal data to solve for the shared features of the overall multimodal data, which can then be used in various downstream sub-tasks. For example, depending on the type of multimodal data, sub-tasks include, but are not limited to, audio recognition, multimodal sentiment analysis, terminal authentication, image semantic recognition, image classification, and text classification. Furthermore, the incomplete multimodal data and the shared features obtained from its analysis can be stored in a distributed cloud storage system for persistent storage and maintenance. In addition, the extraction of shared features from multimodal data and various downstream sub-tasks can be provided as independent or packaged services in the cloud, thereby enabling artificial intelligence cloud services.

[0071] Therefore, before introducing the embodiments of this application, it is necessary to explain some terms in the field of AI:

[0072] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0073] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include audio processing, computer vision, natural language processing, and machine learning / deep learning.

[0074] Enabling computers to hear, see, speak, and feel represents the future direction of human-computer interaction. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0075] In this application embodiment, a data processing method is involved, which can utilize machine learning theory to extract shared features that can accurately represent the common information among the entire multimodal data based on incomplete multimodal data with missing feature values, so as to be put into various downstream sub-tasks. This ensures that the shared features can integrate comprehensive information from various modalities and reflect the characteristics of each sample more comprehensively and in detail.

[0076] Figure 1 This is a schematic diagram illustrating the implementation environment of a data processing method provided in an embodiment of this application. See also... Figure 1 The implementation environment includes terminal 101 and server 102.

[0077] Terminal 101 has applications installed and running that support multimodal data acquisition services. These applications include, but are not limited to, browser applications, social applications, food ordering applications, payment applications, ride-hailing applications, image processing applications, and short video applications.

[0078] Terminal 101 can be directly or indirectly connected to server 102 via wired or wireless communication. This embodiment of the application does not limit the connection method.

[0079] Server 102 is used to provide background multimodal data processing services for the aforementioned applications. Server 102 includes at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. Optionally, server 102 undertakes the primary computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the primary computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.

[0080] In some embodiments, server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0081] In some embodiments, terminal 101 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, MP3 (Moving Picture Experts Group Audio Layer III) player, MP4 (Moving Picture Experts Group Audio Layer IV) player, e-book reader, etc., but is not limited to these.

[0082] Those skilled in the art will understand that the number of terminals 101 described above can be more or less. For example, there may be only one terminal 101, or there may be dozens or hundreds of terminals 101, or even more. This application does not limit the number or type of terminals 101 in the embodiments.

[0083] With the rapid development of computer and multimedia technologies, feature data of the same entity (an entity can be considered a sample, i.e., the object to be tested) can be collected from different data sources or channels, giving rise to multimodal data. Each source or channel generates feature data of different modalities. For example, web pages typically possess feature data of both text and links; each international news article has translations in multiple languages ​​(each translation is also a type of feature data); images possess feature data of texture, color, and histograms, among other modalities. These multimodal data exhibit complementarity and consistency. Effectively fusing information from different modalities can yield more comprehensive and discriminative features of the data entity (i.e., shared features), providing more reliable feature support for various downstream tasks. However, in many application scenarios, data gaps are often unavoidable, meaning feature data is frequently presented in incomplete, multimodal forms. For example, some feature values ​​may be missing from the feature data of a certain modality. This not only leads to insufficient usable information but also disrupts the consistency of instance dimensions across modalities, rendering traditional multimodal fusion algorithms inapplicable. Considering that a large amount of unlabeled feature data also exists in real life, in recent years, how to solve the problem of fusion of incomplete multimodal data has become a research hotspot in the field of multimodal applications. Therefore, there is an urgent need for a data processing method for incomplete multimodal data.

[0084] This application provides a data processing method, namely a fusion algorithm for incomplete multimodal data. If data samples (i.e., data instances) can represent each other, it means that a data sample can be represented as a linear combination of other data samples in the same category, so that the similarity between data samples can be measured according to the strength of their correlation, rather than simply the distance measurement.

[0085] Figure 2 This is a flowchart of a data processing method provided in an embodiment of this application. See also... Figure 2 This embodiment is applied to a computer device and includes the following steps:

[0086] 201. A computer device acquires multiple feature data belonging to multiple modalities, wherein the feature data corresponding to at least one of the multiple modalities contains missing feature values, and any one of the multiple modalities corresponds to one or more feature data.

[0087] In some embodiments, the plurality of feature data includes at least the following dimensions of description:

[0088] 1) Modality: Used to represent the data source or channel from which feature data is collected. For example, feature data collected from different data sources have different modalities, feature data collected from different channels have different modalities, and feature data collected from different data sources or different channels all have different modalities.

[0089] 2) Sample: also known as instance. For the same sample, it is collected from different data sources or channels to obtain feature data of multiple modalities belonging to the sample. The phenomenon of missing feature values ​​can refer to the overall absence of feature data of a certain sample in a certain modality.

[0090] 3) Category: This refers to the target object to which the sample belongs. The same category may include one or more samples, and each sample includes feature data belonging to multiple modalities.

[0091] In some embodiments, the feature data corresponding to any one of the multiple modalities corresponds to multiple categories, which are multiple target objects. The multiple modalities of the multiple target objects include at least the visual information and near-infrared information of the face images of the multiple target objects. The following will use face images as an example for illustrative explanation.

[0092] In an exemplary scenario, let's take a multimodal face dataset as an example. Assume this dataset includes feature data from two modalities: visual information and near-infrared information (from different acquisition channels) of the face image. The total number of modalities is m = 2. If the dataset contains N face images, the total number of instances (i.e., the sample size of the dataset) is N. These N face images belong to k actual target objects, resulting in a total of k categories. Therefore, the multimodal face dataset can be represented as follows: in, Indicates having d v The original matrix formed by the feature data of the v-th modality of the dimensional attribute, where R indicates that all feature data belong to the real number domain. For example, in d... v Given that X = 100, (1) X represents a visual information matrix with 100-dimensional feature data. (2) This represents a near-infrared information matrix with 100-dimensional feature data. Optionally, since some samples lack visual or near-infrared information, this constitutes an incomplete multimodal face dataset.

[0093] It should be noted that in other embodiments, in addition to selecting the face image of the target object as a sample, the body image of the target object, the video of the target object, the audio of the target object, and the descriptive text of various information can also be selected as samples. The embodiments of this application do not specifically limit the type of multimodality and the type of feature data.

[0094] 202. For the at least one mode containing missing feature values, the computer device completes the missing feature values ​​in the feature data corresponding to the at least one mode based on the average value of the feature data corresponding to the at least one mode, thereby obtaining the completed feature of the at least one mode.

[0095] In some embodiments, since not every modality's feature data has missing feature values, the computer device only needs to use the average value of the existing feature data to fill in the missing feature values ​​in each modality with missing feature values ​​to construct a complete multimodal feature dataset (e.g., a multimodal face dataset). For feature data in modalities that do not have missing feature values, they are skipped and do not need to be processed, thereby saving the processing resources of the computer device.

[0096] 203. The computer device obtains a shared feature of the multiple feature data of the multiple modalities based on the completion feature of the at least one modality and the feature data of the multiple modalities other than the at least one modality. The shared feature is used to represent the common information among the multiple feature data belonging to the multiple modalities respectively.

[0097] In some embodiments, a computer device can regard the completed features of each modality and the feature data of each modality that does not have missing feature values ​​as a completed multimodal feature dataset. In the completed multimodal feature dataset, there are no longer any missing feature values. Therefore, it is possible to align the feature data of different modalities, which facilitates the subsequent accurate extraction of shared features of the feature data of each modality, thereby achieving the effect of multimodal data fusion.

[0098] In some embodiments, when extracting shared features based on the completed multimodal feature dataset, low-rank sparse representation techniques can be used to learn the global and local graph structures of each modality in the self-expression subspace, construct each category's self-expression features in each modality, and use each self-expression feature as an affinity graph in the spectral clustering algorithm to perform spectral clustering operations on each self-expression feature, thereby constructing a more accurate shared feature. The above operations will be described in detail in the next embodiment.

[0099] The method provided in this application uses the average value of feature data under each modality to complete the missing feature values ​​in incomplete multimodal data. By using the completed feature and the original feature data, the feature data of different modalities can be aligned, which facilitates the accurate reconstruction of shared features that represent the common information between feature data of different modalities, thereby improving the processing accuracy of incomplete multimodal data.

[0100] Figure 3 This is a flowchart of a data processing method provided in an embodiment of this application. Please refer to it. Figure 3 This embodiment is applied to computer devices. The following description uses a server as an example of a computer device. This embodiment includes the following steps:

[0101] 301. The server obtains multiple feature data belonging to multiple modalities, wherein the feature data corresponding to at least one of the multiple modalities contains missing feature values, and any one of the multiple modalities corresponds to one or more feature data.

[0102] In some embodiments, the plurality of feature data includes at least the following dimensions of description:

[0103] 1) Modality: Used to represent the data source or channel from which feature data is collected. For example, feature data collected from different data sources have different modalities, feature data collected from different channels have different modalities, and feature data collected from different data sources or different channels all have different modalities.

[0104] 2) Sample: also known as instance. For the same sample, it is collected from different data sources or channels to obtain feature data of multiple modalities belonging to the sample. The phenomenon of missing feature values ​​can refer to the overall absence of feature data of a certain sample in a certain modality.

[0105] 3) Category: This refers to the target object to which the sample belongs. The same category may include one or more samples, and each sample includes feature data belonging to multiple modalities.

[0106] In some embodiments, the feature data corresponding to any one of the multiple modalities corresponds to multiple categories, which are multiple target objects. The multiple modalities of the multiple target objects include at least the visual information and near-infrared information of the face images of the multiple target objects. The following will use face images as an example for illustrative explanation.

[0107] In an exemplary scenario, let's take a multimodal face dataset as an example. Assume this dataset includes feature data from two modalities: visual information and near-infrared information (from different acquisition channels) of the face image. The total number of modalities is m = 2. If the dataset contains N face images, the total number of instances (i.e., the sample size of the dataset) is N. These N face images belong to k actual target objects, resulting in a total of k categories. Therefore, the multimodal face dataset can be represented as follows: in, Indicates having d v The original matrix formed by the feature data of the v-th modality of the dimensional attribute, where R indicates that all feature data belong to the real number domain. For example, in d... v Given that X = 100, (1) X represents a visual information matrix with 100-dimensional feature data. (2) This represents a near-infrared information matrix with 100-dimensional feature data. Optionally, since some samples lack visual or near-infrared information, this constitutes an incomplete multimodal face dataset.

[0108] It should be noted that in other embodiments, in addition to selecting the face image of the target object as a sample, the body image of the target object, the video of the target object, the audio of the target object, and the descriptive text of various information can also be selected as samples. The embodiments of this application do not specifically limit the type of multimodality and the type of feature data.

[0109] Optionally, the server can read the multiple feature data from a local database, receive the multiple feature data uploaded by the terminal from a cloud computing platform or AIaaS platform, or call the multiple feature data from a distributed cloud storage system. This application embodiment does not specifically limit the method of obtaining the multiple feature data.

[0110] 302. The server performs regularization on the multiple feature data to obtain multiple regularized feature data.

[0111] In some embodiments, the server may use various regularization methods to regularize the multiple feature data to obtain the multiple regularized feature data. For example, it may use a regularization method based on the L0 norm, or a regularization method based on the L1 norm, or a regularization method based on the L2 norm. This application embodiment does not specifically limit the regularization method of the feature data.

[0112] In an exemplary scenario, taking the aforementioned incomplete multimodal face dataset as an example, assuming that L2 norm-based regularization is applied to each data sample (i.e., data instance) in the multimodal face dataset, the regularization principle is equivalent to performing the operation shown in the following formula (1):

[0113]

[0114] Where, the symbol st represents the constraint condition, X i，j Let represent the element in the i-th row and j-th column of matrix X, which represents the j-th attribute value of the i-th data sample in the multimodal face dataset X. N represents the sample size of the multimodal face dataset X. v d represents the attribute dimension of a modality in a multimodal face dataset X. For example, for the modality of visual information in face images, d represents the attribute dimension of the visual information matrix. v It can be 100-dimensional, in which case the visual information of each data sample is represented as a vector of length 100. The visual information vectors of N data samples constitute the visual information matrix of the entire multimodal face dataset. This visual information matrix is ​​a matrix of size N×d. v The matrix.

[0115] In the above process, by performing regularization on each feature data, it is equivalent to preprocessing the entire multimodal feature dataset. This preprocessing method of performing regularization on each feature data of each modality can eliminate the dimensional influence between feature data of different modalities and accelerate the convergence rate when adjusting shared features in subsequent iterations.

[0116] In some embodiments, the server may skip step 302 and directly execute step 303 after obtaining the original feature data. That is, for each modality containing missing feature values, the average value of the unregularized feature data is obtained, and the average value is filled into the missing feature values ​​of each modality to obtain the intermediate features of each modality. This can greatly simplify the data processing flow.

[0117] 303. For any modality in the at least one modality containing missing feature values, the server obtains the average value of the regularized feature data corresponding to that modality.

[0118] In some embodiments, the server determines at least one modality with missing feature values ​​from the plurality of modalities, obtains the average value of the regularized feature data corresponding to each modality in the at least one modality, and performs the following step 304.

[0119] In some embodiments, if the individual feature data are not regularized, the server only needs to obtain the average value of the unregularized feature data for each modality containing missing feature values ​​before performing step 304.

[0120] 304. The server fills the missing feature values ​​in the regularized feature data corresponding to any modality with the average value of the regularized feature data corresponding to any modality to obtain the complete feature of any modality.

[0121] In some embodiments, the server can assign the missing feature values ​​(NULL or empty) in the regularized feature data of each modality to the average value of the regularized feature data of each modality obtained in step 303 above, thereby obtaining a complete multimodal feature dataset after completion. In this multimodal feature dataset, each modality without missing feature values ​​retains the original feature data obtained in step 301 above, while each modality with missing feature values ​​is updated to the completed intermediate features obtained in step 304 above.

[0122] In some embodiments, if no regularization is performed on each feature data, the server only needs to obtain the average value of the unregularized feature data and then assign the missing feature value (NULL or empty) in the unregularized feature data of each modality to the average value of the unregularized feature data of each modality obtained in step 303 above. This will result in a complete multimodal feature dataset after completion, and since no regularization step needs to be performed, the data processing flow can be simplified.

[0123] In one exemplary embodiment, server regularization processes an incomplete multimodal feature dataset. Then, for each modality v, the average value of the feature data corresponding to each modality with missing feature value is obtained, and the missing feature value in each modality is filled with the average value. The filled intermediate features can be successfully aligned in dimensions in each modality, so as to facilitate further multimodal data fusion.

[0124] In steps 303-304 above, the server performs the step of filling in missing feature values ​​and obtaining the filled feature based on the multiple regularized feature data, which can accelerate the convergence rate. In some embodiments, step 302 above may not be performed. In this case, the steps of filling in missing feature values ​​and obtaining the filled feature are directly performed based on the original feature data that has not been regularized, thereby simplifying the data processing flow.

[0125] In the above process, by obtaining the completion features of each modality, the server can ensure that after the padding operation, the missing feature values ​​of each modality in the incomplete multimodal feature dataset are completed, so that the final completion features of each modality can be aligned with the dimensions, which is beneficial to subsequent feature fusion operations.

[0126] 305. For any one of the multiple modalities, the server obtains the target weight matrix corresponding to that modality, wherein the weight coefficients of the non-missing feature values ​​in the target weight matrix under that modality are greater than the weight coefficients of the missing feature values.

[0127] Optionally, the target weight matrix is ​​a diagonal weight matrix, the weight coefficient of the non-missing feature value is 1, and the weight coefficient of the missing feature value is the ratio of the number of samples with non-missing feature values ​​in any modality to the total number of samples.

[0128] It should be noted that the target weight matrix is ​​not limited to the form of a diagonal weight matrix. As long as the weight coefficient of the non-missing feature values ​​is greater than the weight coefficient of the missing feature values ​​(which have been filled with the average value), the embodiment of this application does not specifically limit the form of the target weight matrix.

[0129] In some embodiments, the server constructs a target weight matrix corresponding to each of the multiple modalities. This target weight matrix is ​​used to weight the reconstruction error of the self-expressive features of the corresponding modality during subsequent learning of shared features, thereby reducing the weight of the average value of the padding in the construction of shared features and thus improving the expressive power of the shared features. Optionally, the above weighting process also involves element-wise multiplying each target weight matrix with the reconstruction error of the self-expressive features of the corresponding modality.

[0130] In one exemplary embodiment, the server constructs a diagonal weight matrix G for the feature data of each modality. (v) ∈R N×N (i.e., the target weight matrix), the diagonal weight matrix is ​​defined as follows in formula (2):

[0131]

[0132] in, The weight matrix G represents mode v. (v) The element located in the i-th row and i-th column (i.e., the i-th diagonal element), n v This represents the total number of data instances in modality v that are not missing, and N represents the total number of data instances in modality v (i.e., the sample size).

[0133] In the above process, for the at least one modality containing missing feature values, the server fills in the missing feature values ​​in the feature data corresponding to the at least one modality based on the average value of the feature data corresponding to the at least one modality, and obtains the completed features of the at least one modality. By constructing a target weight matrix, the missing feature values ​​can be assigned lower weights, thereby ensuring that the non-missing data instances of each modality are represented by real data as much as possible (rather than the filled average value).

[0134] 306. For the feature data or completion features of any of the multiple modalities, the server linearly combines the average value of the feature values ​​or completions belonging to the same category to obtain the self-expressive features of the multiple categories in any modality.

[0135] In some embodiments, for each of the multiple modalities, the server determines the feature values ​​(or the padded average values) of all data samples belonging to each category in each modality, and performs a linear combination of the feature values ​​of all data samples belonging to each category to obtain the self-expressive features corresponding to each category in each modality.

[0136] In some embodiments, during the linear combination process described above, the self-expression features corresponding to each category can be gradually approximated by reducing the reconstruction error. When obtaining the reconstruction error of each self-expression feature, the target weight matrix constructed in step 305 can be used for weighting to reduce the contribution of the filled average value to the self-expression features.

[0137] In the above process, that is, for each category in each modality, the server linearly combines the feature values ​​(or the average values ​​of the completed features) belonging to that modality and that category in a certain combination method to obtain a linearly combined feature and obtain the reconstruction error of the combination method. If the reconstruction error is greater than the reconstruction threshold, it is necessary to iteratively adjust the combination method (such as adjusting the coefficient of each feature value in the linear combination) until the reconstruction error is less than or equal to the reconstruction threshold, and then the self-expressive feature of that category in that modality can be obtained.

[0138] In some embodiments, the server combines low-rank sparse representation techniques and weighting mechanisms, using the feature data (or completed features) of each modality as a dictionary, and reconstructs its own feature data (or completed features) to learn a low-rank and sparse self-expressive feature Z. (v) ∈R N×N For example, in the case where the feature data is a face image, that is, the self-expressive features make the facial features of the same person linearly represented by their own face image as much as possible.

[0139] 307. The server constructs multiple affinity graphs for the multiple categories under the multiple modalities based on the multiple self-expressive features of the multiple categories under the multiple modalities.

[0140] In this context, any one of the multiple affinity graphs includes multiple nodes and multiple weighted edges. Any one of the multiple nodes is used to represent the self-expression feature of a category in the corresponding modality. The weight carried by any one of the multiple weighted edges is used to represent the similarity between the two self-expression features of the two categories corresponding to the two nodes of the weighted edge in the corresponding modality.

[0141] In some embodiments, the server treats the self-expression features of each category as a node in the affinity graph, connects each node to at least one of its nearest neighboring nodes to form at least one undirected edge, and uses the similarity between the two self-expression features corresponding to the two endpoints of each undirected edge as the weight of the undirected edge, thus obtaining at least one weighted edge. By repeating the above operation, the affinity graph of each category in each modality can be obtained.

[0142] In the above process, the server combines the target weight matrix obtained in step 305 and uses low-rank sparse representation technology to learn the global and local graph structures of the self-expression subspace of each modality. Under the constraint of low weight in the target weight matrix, the missing feature values ​​in each modality hardly participate in the feature representation of other data instances. Therefore, the information of other data feature representations is not diluted by the average value of the filling, thus ensuring the accuracy of other data feature representations.

[0143] 308. Based on the multiple affinity graphs, the server performs spectral clustering on the multiple self-expressive features to obtain the shared features of the multiple feature data of the multiple modalities. The shared features are used to represent the common information among the multiple feature data belonging to the multiple modalities respectively.

[0144] In some embodiments, the server applies graph clustering techniques to the low-rank sparse affinity graphs of each modality to obtain the corresponding low-dimensional feature representations (i.e., clustering indicator matrices). Finally, a kernel alignment technique is used to fuse the shared low-dimensional feature representations among the learned modalities using the clustering indicator matrices of each modality, outputting the final shared features. In other words, an incomplete multimodal data fusion model based on low-rank sparse graph learning can be constructed, which will be detailed below.

[0145] 3081. For any given clustering process, the server obtains multiple graph Laplacian matrices for the multiple affinity graphs.

[0146] Optionally, the server obtains the similarity matrix and diagonal matrix of any one of the multiple affinity graphs; and obtains the difference between the diagonal matrix and the similarity matrix as the graph Laplacian matrix of any one affinity graph.

[0147] In some embodiments, it is assumed that the affinity graph constructed from the self-expression features obtained in step 307 above is represented as Z. (v) Then the expression for the graph Laplace matrix is: Among them, L Z It is a graph Laplace matrix. Based on affinity diagram Z (v) To ensure the symmetry of the similarity matrix, we define... The superscript T denotes the transpose of the matrix, while It is the affinity diagram Z (v) A diagonal matrix, where the diagonal elements are defined as similarity matrices. The sum of the elements in the corresponding row.

[0148] 3082. Based on the multiple graph Laplacian matrices, the server obtains multiple clustering indicator matrices for the multiple modalities.

[0149] In some embodiments, the server continuously optimizes the clustering indicator matrices by reducing the value of the objective function. Alternatively, the objective function can be expressed as shown in formula (3) below:

[0150]

[0151] The first three terms of the objective function are expressions for low-rank sparse representation under the weighting mechanism, the fourth term is the spectral clustering formula, and the symbol st represents the constraint condition.

[0152] Among the constraints, E (v) =X (v) -X (v) Z (v) Represents the reconstruction error of the self-expression process (i.e., the reconstruction error of multiple self-expression features in the v-th mode). That is, the error matrix and the affinity diagram Z. (v) Used in subsequent spectral clustering processes, and Z (v) 1 = 1 is about Z (v) The three additional constraints, conditions The aim is to avoid excessive self-representation of data instances, with a non-negativity constraint 0 ≤ Z. (v) ≤1 makes the affinity graph Z (v) It has better interpretability because positive values ​​are more meaningful in typical applications; finally, to ensure that all data participate in self-expressive learning, it is subjected to Z-factors.(v) The constraint 1 = 1, where 1 represents a vector whose elements are all 1s.

[0153] Alternatively, for minimizing the objective function, ||·|| F Denotes the Frobenius norm. This represents the square of the current Frobenius norm in the weight matrix G. (v) Guided by information, minimize ||E (v) G (v) || F The term can effectively reduce the contribution of missing feature values ​​in low-rank sparse representation (i.e., self-expressing features), so as to ensure that non-missing feature values ​​can accurately represent self-expressing features.

[0154] Among them, ||·|| * The nuclear norm is used to represent the self-expression matrix Z. (v) Applying low-rank constraints allows us to learn its global structure (representing the overall relationship between face images of different categories), where ||·||1 denotes the 1-norm, aiming to constrain Z. (v) The sparsity of the matrix allows us to capture its local structural information (representing important feature terms for each face representation). Tr(·) denotes the trace of the matrix. Finally, spectral clustering is used to obtain low-dimensional feature representations for each modality. That is, the clustering indicator matrix.

[0155] Where λ1, λ2, and λ3 are all penalty parameters.

[0156] Optionally, as can be seen from the above formula (3), in the process of obtaining the loss function value, the server calls the target weight matrix to weight the reconstruction error of the self-expression feature corresponding to any modality, thereby effectively reducing the contribution of the filling average value to the overall self-expression feature.

[0157] In some embodiments, the server can successfully obtain low-dimensional feature representations of all modalities in the case of missing data by minimizing the above objective function. Traditional multimodal fusion techniques force all modalities to share the same low-dimensional feature matrix, and its model is shown below:

[0158]

[0159] 3083. Based on the reconstruction error of the multiple self-expressive features, the clustering error of the clustering process, and the similarity difference between the multiple clustering indicator matrices and the features to be solved, the server obtains the loss function value of the clustering process.

[0160] In some embodiments, since the above formula (4) is equivalent to minimizing the following model:

[0161]

[0162] In other words, the determination of the clustering indicator matrix F depends on the sum of the similarity matrices of all modalities.

[0163] In the case of missing modalities, these similarity matrices contain non-true elements (filled averages). Accumulating these elements will exacerbate the error, leading to inaccurate learned shared feature representations. Therefore, this application employs a kernel alignment technique to effectively fuse incomplete multimodal features and, by combining the objective function, obtain the fused model loss function:

[0164]

[0165] Where λ4 is the trade-off parameter, U represents the target fusion feature matrix, and ω(F (v) U) is a kernel alignment technique that measures the F of each mode v based on the idea of ​​kernels. (v) Similarity to U. Optionally, the kernel alignment function is defined as follows:

[0166]

[0167] In some embodiments, the server employs a linear core. K U =UU T ,because Therefore, the above kernel alignment formula (6) can be rewritten as the following formula (7):

[0168]

[0169] In some embodiments, for a given dataset, the number of categories k is typically a fixed constant; therefore, ignoring this constant term yields the final objective function expression:

[0170]

[0171] In the above process, the server designs the objective function based on the model content, performs joint optimization on the matrix variables in the objective function, and obtains the final iterative formula (8). After setting the convergence threshold (i.e. the loss threshold) and initializing all parameters and variables, the final accurate clustering indicator matrix can be obtained by continuous iteration.

[0172] In some embodiments, the server may use either formula (5) or formula (8) as the objective function to obtain the clustering indicator matrix. That is, the server may optimize the clustering indicator matrix by formula (8) based on kernel alignment technology to improve the expressive power of the clustering indicator matrix, or it may optimize the clustering indicator matrix directly by formula (5) without using kernel alignment technology to simplify the data processing flow. This application embodiment does not specifically limit this.

[0173] 3084. The server iteratively adjusts the parameters of the feature to be solved until the loss function value meets the stopping condition, then stops the iteration and obtains the feature to be solved at the last iteration as the shared feature.

[0174] In some embodiments, the server iteratively adjusts the values ​​of multiple intermediate variables in the loss function based on the alternating direction multiplier method to obtain adjusted intermediate variables. When adjusting the value of any intermediate variable, the values ​​of the intermediate variables other than the adjusted intermediate variable are kept unchanged. The server then performs the step of iteratively adjusting the parameters of the feature to be solved while keeping the adjusted intermediate variables unchanged.

[0175] In some embodiments, the stopping condition is that the difference between the loss function value of any clustering process and the loss function value of the previous clustering process is less than a loss threshold. Optionally, the stopping condition may also be that the loss function value of any clustering process is less than a target threshold. Optionally, the stopping condition may also be that the number of iterations is greater than a threshold; this embodiment of the application does not specifically limit this.

[0176] In an exemplary scenario, when variable E (v) Z (v) F (v) When coupled with U, the minimum optimization of the above algorithm becomes a non-convex problem, making it very difficult to find the global optimum. Furthermore, the matrix variable Z... (v) The constraints are numerous, and optimization is almost impossible.

[0177] In view of this, the embodiments of this application respectively introduce intermediate variables S (v) P (v) and Q (v) To realize matrix variable Z (v) For multi-constraint problems, optionally, based on the ADMM (Alternating Direction Method of Multipliers) technique, the original objective function is transformed into multiple sub-problems. By iteratively optimizing the variables in each sub-problem, the local optimum of the model can be found separately.

[0178] The Lagrangian form of the overall objective function is given below:

[0179]

[0180] in, and All are Lagrange multipliers of mode v, and μ is a penalty parameter.

[0181] In the ADMM algorithm, when updating one variable, other variables remain unchanged. Optionally, the update process is as follows:

[0182] (1) Fix all other variables and update Z. (v)

[0183] Regarding Z (v) The Lagrange Days problem is:

[0184]

[0185] Taking the partial derivative of the above equation and setting the partial derivative formula to 0, we can obtain the optimized formula (11) for Z(v):

[0186]

[0187] in,

[0188] (2) Fix all other variables and update S. (v)

[0189] Regarding S (v) The Lagrange Days problem is:

[0190]

[0191] To solve the above subproblem, the singular value threshold shrinkage operator Θ is applied to update S. (v) :

[0192]

[0193] (3) Fix all other variables and update Q. (v) and

[0194] The Lagrange day problem is as follows:

[0195]

[0196] because Therefore, equation (14) above can be rewritten as an optimization problem of the following element form:

[0197]

[0198] in, Taking the partial derivative of the above formula (15) with respect to Q (v) The optimization problem is equivalent to Therefore, the above formula (15) can be updated to the following formula (16):

[0199]

[0200] To ensure The nonnegativity of this application embodiment is added. Operation. That is to say, it can then be updated according to the following formula (17).

[0201]

[0202] (4) Fix all other variables and update P. (v)

[0203] Its Lagrange day problem is:

[0204]

[0205] Sparsity-constrained problems can be solved using a soft thresholding operation θ:

[0206]

[0207] (5) Fix all other variables and update E. (v)

[0208] Its Lagrange day problem is:

[0209]

[0210] definition Then matrix E (v) The update formula is:

[0211] E (v)＝ μM (v) (2G (v) G (v)T +μI) -1 (twenty one)

[0212] in,(·) -1 This represents the matrix inversion operation.

[0213] (6) Fix all other variables and minimize F. (v) The problem can be transformed into the following maximization problem:

[0214]

[0215] The above problem can be optimized using the eigenvalue decomposition method, and the updated F (v) Depend on The eigenvalue vectors are formed by the k largest eigenvalues ​​after eigenvalue decomposition.

[0216] (7) Fix other variables and update and μ

[0217] Optionally, the relevant parameters in these ADMMs can be updated according to the following formula:

[0218]

[0219]

[0220]

[0221]

[0222] μ=min(μ0,ρμ) (27)

[0223] Here, the constants μ0 and ρ are preset parameters.

[0224] (8) Finally, fixing all other variables, update the shared spectral clustering indicator matrix U of the incomplete multimodal feature dataset using the same method as in step (6). The subproblem with respect to U is:

[0225]

[0226] The i-th row of matrix U is updated to The eigenvalue vector corresponding to the i-th (descending order) eigenvalue after eigenvalue decomposition.

[0227] Optionally, after obtaining the update formulas for all variables, a convergence threshold of 10 can be pre-set. -4 Assign values ​​to all parameters; optionally, let μ = 0.01, μ0 = 1e8, ρ = 1.1; randomly initialize all variable matrices.

[0228] In some embodiments, the server updates the variable matrix alternately according to the above optimization formula and calculates the objective function value after each iteration. It determines whether the difference between the objective function values ​​of two adjacent iterations meets the stopping condition. If it does, the iteration stops, and the shared low-dimensional feature matrix obtained in this calculation becomes the final target output of the algorithm. If the convergence condition is not met, step 3084 is repeated.

[0229] In steps 307-308 above, the server clusters the multiple self-expressive features of the multiple categories under the multiple modalities to obtain the shared feature. By applying low-rank and sparse joint constraints to the self-expressive affinity graph, both a global graph structure reflecting the relationships of all data instances and a local graph structure reflecting the features of individual data instances can be obtained simultaneously.

[0230] In steps 306-308 above, the server obtains shared features of the multiple modalities' feature data based on the completion features of the at least one modality and the feature data of the modalities other than the at least one modality. This constructs an effective model to address the information loss problem in multimodal data fusion. Extensive experimental verification shows that the shared features obtained in this application's embodiments exhibit good performance. Furthermore, a weighting mechanism based on the target weight matrix is ​​proposed, combined with low-rank sparse representation technology, which can effectively learn decentralized multimodal data in the case of missing modalities and is easily extended to scenarios with any number of modalities.

[0231] Figure 4 This is a schematic flowchart illustrating the principle of a data processing method provided in an embodiment of this application. Please refer to it. Figure 4 Taking a total of 3 modalities as an example, for an incomplete multimodal feature dataset, the average value is used to fill the missing feature values ​​in each modality. Then, combined with the weighting mechanism of the target weight matrix, the self-expression features of each category in each modality are obtained. The clustering indicator matrix of each modality is obtained by using spectral clustering technology. Finally, the kernel alignment technology is combined to iteratively adjust the final shared features.

[0232] Figure 5 This is a schematic flowchart illustrating the principle of a data processing method provided in an embodiment of this application. Please refer to it. Figure 5 Step 1: Preprocess the incomplete multimodal data, which is equivalent to regularizing each feature data. Step 2: Fill in the missing feature values ​​with averages and construct the corresponding target weight matrix. Step 3: Learn the self-expressive features of each modality's feature data and use them as affinity maps for spectral clustering to obtain a clustering indicator matrix, then fuse them using kernel alignment technology. Step 4: Solve the optimization formula for each variable by minimizing the algorithm's objective function, set a convergence threshold, and initialize all parameters and variables. Step 5: Update all variables according to the optimization formula and calculate the current loss function value. Step 6: Calculate the difference between the current and previous loss function values. If it is greater than or equal to the given loss threshold, return to Step 5; otherwise, proceed to Step 7. Step 7: End the iteration and obtain the shared features.

[0233] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0234] The method provided in this application uses the average value of feature data under each modality to complete the missing feature values ​​in incomplete multimodal data. By using the completed feature and the original feature data, the feature data of different modalities can be aligned, which facilitates the accurate reconstruction of shared features that represent the common information between feature data of different modalities, thereby improving the processing accuracy of incomplete multimodal data.

[0235] Figures 6 to 12 The different choices of model parameter λ1 are shown respectively. Figures 6 to 12 The standardized mutual information index varies with different values ​​of λ1 for seven multimodal datasets: BUAA, 3sources, Yale, SensIT, Wikipedia, Aloi, and Webkb, when the data missing rate is 30%, while the parameters λ3 and λ4 are fixed.

[0236] Figures 13 to 19 The choices of model parameters λ3 and λ4 are shown respectively. Figures 13 to 19 The figures show the variation of standardized mutual information with λ3 and λ4 for seven multimodal datasets (BUAA, 3sources, Yale, SensIT, Wikipedia, Aloi, and Webkb) with a data missing rate of 30%, while parameter λ1 is fixed.

[0237] To verify the effectiveness of the model proposed in this application, it is compared and analyzed with seven representative multimodal feature learning models: PVC (Partial Multi-View Clustering), MIC (Multi-Incomplete-view Clustering), IMG (Incomplete Multi-modality Grouping), DAIMC (Doubly Aligned Incomplete Multi-view Clustering), MLRSSC (Multi-view Low-Rank Sparse Subspace Clustering), APMC (Anchor-based Partial Multi-view Clustering), and IMSCAGL (Incomplete Multi-view Spectral Clustering with Adaptive Graph Learning). The performance of each model is verified in terms of accuracy, standardized mutual information, and purity using K-means clustering technology. In addition to using the BUAA face image dataset from the examples, the experiment also used the news dataset 3sources, the web page dataset Webkb, the sound signal dataset SensIT, the Wikipedia dataset, and the image datasets Yale and Aloi. The specific dataset information is shown in Table 1.

[0238] Table 1

[0239]

[0240]

[0241] In the experiments, each model first learned the fusion features of different multimodal datasets with missing data rates ranging from 0.1 to 0.5, and then used K-means clustering analysis. Tables 2 to 8 show the comparison results of clustering accuracy, standardized mutual information, and purity between the proposed model and other comparative models on seven datasets. The experiments show that the proposed model significantly outperforms other models in clustering on all datasets. This fully demonstrates that the model effectively acquires the inherent geometric structure between missing data through low-rank sparse graph learning under the weight mechanism, and its kernel-aligned fusion features have high discriminative accuracy. Furthermore, since the proposed model achieves the best performance on various datasets, it demonstrates its broad application value.

[0242] Table 2 shows the comparison results of the average clustering performance (accuracy, normalized mutual information, purity) of the BUAA dataset on various models:

[0243] Table 2

[0244]

[0245]

[0246] Table 3 shows the comparison results of the average clustering performance (accuracy, normalized mutual information, purity) of the 3sources dataset on various models:

[0247] Table 3

[0248]

[0249] Table 4 shows the comparison results of the average clustering performance (accuracy, normalized mutual information, purity) of the Yale dataset on various models:

[0250] Table 4

[0251]

[0252]

[0253] Table 5 shows the comparison results of the average clustering performance (accuracy, normalized mutual information, purity) of the SensIT dataset on various models:

[0254] Table 5

[0255]

[0256]

[0257] Table 6 shows the comparison results of the average clustering performance (accuracy, normalized mutual information, purity) of the Wikipedia dataset on various models:

[0258] Table 6

[0259]

[0260] Table 7 shows the comparison results of the average clustering performance (accuracy, normalized mutual information, purity) of the Aloi dataset on various models:

[0261] Table 7

[0262]

[0263] Table 8 shows the comparison results of the average clustering performance (accuracy, normalized mutual information, purity) of the Webkb dataset on various models:

[0264] Table 8

[0265]

[0266]

[0267] In the comparative experiments, all model parameters used the recommended values ​​from the relevant original texts. For the model parameters proposed in this application, the optimal value of λ1 was selected within the range of {0.1, 0.3, 0.5, 0.7, 0.9}, and λ2 was set to 1-λ1, with values ​​for λ3 and λ4 selected within the range of {1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1}. To find the best match for the above parameters, this application adopted the following strategy: first, λ1 was set to 0.3 to find the optimal combination of λ3 and λ4, and then the value of λ1 was adjusted by fixing it in reverse. Finally, using the standardized mutual information value as the evaluation criterion, the parameter values ​​with the best clustering performance were selected as the final parameter values ​​for model calculation.

[0268] The above provides a detailed description of an incomplete multimodal data fusion algorithm based on low-rank sparse graph learning provided in the embodiments of this application.

[0269] Figure 20 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Please refer to it. Figure 20 The device includes:

[0270] The first acquisition module 2001 is used to acquire multiple feature data belonging to multiple modalities, wherein at least one modality among the multiple modalities contains missing feature values ​​in its feature data, and any one modality among the multiple modalities corresponds to one or more feature data.

[0271] The completion module 2002 is used to, for the at least one mode containing missing feature values, complete the missing feature values ​​in the feature data corresponding to the at least one mode based on the average value of the feature data corresponding to the at least one mode, and obtain the completed features of the at least one mode;

[0272] The second acquisition module 2003 is used to acquire shared features of the multiple feature data of the multiple modalities based on the completion features of the at least one modality and the feature data of the modalities other than the at least one modality. The shared features are used to represent the common information among the multiple feature data belonging to the multiple modalities respectively.

[0273] The apparatus provided in this application uses the average value of feature data under each modality to complete the missing feature values ​​in incomplete multimodal data. By using the completed feature and the original feature data, the feature data of different modalities can be aligned, which facilitates the accurate reconstruction of shared features that represent the common information between feature data of different modalities, thereby improving the processing accuracy of incomplete multimodal data.

[0274] In one possible implementation, based on Figure 20 The device comprises the following components: the completion module 2002 includes:

[0275] The acquisition submodule is used to acquire the average value of the feature data corresponding to any one of the at least one modalities containing missing feature values.

[0276] The fill submodule is used to fill the missing feature values ​​in the feature data corresponding to any mode with the average value of the feature data corresponding to any mode, so as to obtain the complete feature of any mode.

[0277] In one possible implementation, based on Figure 20 The device comprises:

[0278] The regularization module is used to perform regularization processing on the multiple feature data to obtain multiple regularized feature data.

[0279] The completion module 2002 is also used to perform the step of completing missing feature values ​​and obtaining completed features based on the multiple regularized feature data.

[0280] In one possible implementation, the feature data corresponding to any one of the multiple modalities corresponds to multiple categories respectively;

[0281] based on Figure 20 The device comprises, wherein the second acquisition module 2003 includes:

[0282] The linear combination submodule is used to linearly combine the feature values ​​or the average value of the complements belonging to the same category for the feature data or complements of any of the multiple modalities, so as to obtain the self-expressive features of the multiple categories in any modality.

[0283] The clustering submodule is used to cluster multiple self-expressive features of multiple categories under multiple modalities to obtain the shared feature.

[0284] In one possible implementation, based on Figure 20 The device consists of the following components: the clustering submodule includes:

[0285] The construction unit is used to construct multiple affinity graphs of multiple categories under multiple modalities based on the multiple self-expressive features. Each affinity graph includes multiple nodes and multiple weighted edges. Each node is used to represent a self-expressive feature of a category under the corresponding modality. The weight carried by any weighted edge is used to represent the similarity between the two self-expressive features of the two categories corresponding to the two nodes of the weighted edge under the corresponding modality.

[0286] The spectral clustering unit is used to perform spectral clustering operations on the multiple self-expressed features based on the multiple affinity graphs to obtain the shared feature.

[0287] In one possible implementation, based on Figure 20 The device comprises: the spectral clustering unit includes:

[0288] The first acquisition subunit is used to acquire multiple graph Laplacian matrices of the multiple affinity graphs for any given clustering process;

[0289] The second acquisition subunit is used to acquire multiple clustering indicator matrices of the multiple modalities based on the multiple graph Laplacian matrices;

[0290] The third acquisition subunit is used to acquire the loss function value of any clustering process based on the reconstruction error of the multiple self-expressive features, the clustering error of any clustering process, and the similarity difference between the multiple clustering indicator matrices and the features to be solved.

[0291] The iterative adjustment sub-unit is used to iteratively adjust the parameters of the feature to be solved until the loss function value meets the stopping condition. Then the iteration stops and the feature to be solved at the last iteration is obtained as the shared feature.

[0292] In one possible implementation, based on Figure 20 The device comprises:

[0293] The third acquisition module is used to acquire a target weight matrix corresponding to any one of the multiple modes, wherein the weight coefficients of the non-missing feature values ​​in the target weight matrix under any one mode are greater than the weight coefficients of the missing feature values.

[0294] The third acquisition subunit is also used to call the target weight matrix during the process of acquiring the loss function value to weight the reconstruction error of the self-expressive feature corresponding to any mode.

[0295] In one possible implementation, the target weight matrix is ​​a diagonal weight matrix, the weight coefficient of the non-missing feature value is 1, and the weight coefficient of the missing feature value is the ratio of the number of samples with non-missing feature values ​​in any modality to the total number of samples.

[0296] In one possible implementation, the first acquisition subunit is used for:

[0297] Based on any one of the multiple affinity graphs, obtain the similarity matrix and diagonal matrix of that affinity graph;

[0298] The difference between the diagonal matrix and the similarity matrix is ​​taken as the graph Laplacian matrix of any affinity graph.

[0299] In one possible implementation, the iterative adjustment subunit is used for:

[0300] Based on the alternating direction multiplier method, the values ​​of multiple intermediate variables in the loss function are iteratively adjusted to obtain the adjusted intermediate variables. When adjusting the value of any intermediate variable, the values ​​of the intermediate variables other than the adjusted intermediate variable are kept unchanged.

[0301] Keeping the adjusted values ​​of the intermediate variables unchanged, perform the step of iteratively adjusting the parameters of the feature to be solved.

[0302] In one possible implementation, the stopping condition is that the difference between the loss function value of any clustering process and the loss function value of the previous clustering process is less than a loss threshold.

[0303] In one possible implementation, the feature data corresponding to any one of the multiple modalities corresponds to multiple categories, which are multiple target objects. The multiple modalities of the multiple target objects include at least the visual information and near-infrared information of the face images of each of the multiple target objects.

[0304] It should be noted that the data processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the data processing method embodiments, which will not be repeated here.

[0305] Figure 21 This is a schematic diagram of a computer device provided in an embodiment of this application. The computer device 2100 can vary significantly due to differences in configuration or performance. The computer device 2100 includes one or more Central Processing Units (CPUs) 2101 and one or more memories 2102. The memories 2102 store at least one computer program, which is loaded and executed by the one or more processors 2101 to implement the data processing methods provided in the various embodiments described above. Optionally, the computer device 2100 also includes wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The computer device 2100 also includes other components for implementing device functions, which will not be elaborated here.

[0306] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program, which can be executed by a processor in a terminal to perform the data processing methods in the various embodiments described above. For example, the computer-readable storage medium includes ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0307] In an exemplary embodiment, a computer program product or computer program is also provided, including one or more lines of program code stored in a computer-readable storage medium. One or more processors of a computer device are capable of reading the one or more lines of program code from the computer-readable storage medium, and the one or more processors execute the one or more lines of program code, enabling the computer device to perform the data processing method described above.

[0308] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. Optionally, the program is stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0309] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Multiple feature data belonging to multiple modalities are obtained, wherein at least one modality in the multiple modalities contains missing feature values ​​in its feature data, each modality in the multiple modalities corresponds to one or more feature data, the feature data corresponding to each modality in the multiple modalities corresponds to multiple categories, the multiple categories are multiple target objects, and the multiple modalities of the multiple target objects include at least the visual information and near-infrared information of the face images of each of the multiple target objects; For the at least one modality containing missing feature values, based on the average value of the feature data corresponding to the at least one modality, the missing feature values ​​in the feature data corresponding to the at least one modality are filled in to obtain the filled features of the at least one modality; Based on the completion features of at least one modality and the feature data of modalities other than the at least one modality among the plurality of modalities, multiple self-expressive features of the plurality of categories under the plurality of modalities are obtained; Based on the multiple self-expressive features, multiple affinity graphs of the multiple categories are constructed under the multiple modalities. Each affinity graph includes multiple nodes and multiple weighted edges. Each node is used to represent a self-expressive feature of a category under the corresponding modality. The weight carried by any weighted edge is used to represent the similarity between the two self-expressive features of the two categories corresponding to the two nodes of the weighted edge under the corresponding modality. Based on the multiple affinity graphs, spectral clustering is performed on the multiple self-expressive features to obtain shared features of the multiple feature data of the multiple modalities. The shared features are used to represent the common information among the multiple feature data belonging to the multiple modalities respectively.

2. The method according to claim 1, characterized in that, For the at least one modality containing missing feature values, based on the average value of the feature data corresponding to the at least one modality, the missing feature values ​​in the feature data corresponding to the at least one modality are filled in to obtain the filled features of the at least one modality, including: For any of the at least one modalities containing missing feature values, obtain the average value of the feature data corresponding to that modality; The average value of the feature data corresponding to any modality is filled into the missing feature values ​​in the feature data corresponding to any modality to obtain the complete feature of any modality.

3. The method according to claim 1, characterized in that, Before, for the at least one modality containing missing feature values, the method further includes, based on the average value of the feature data corresponding to the at least one modality, filling in the missing feature values ​​in the feature data corresponding to the at least one modality to obtain the completed features of the at least one modality: The multiple feature data are subjected to regularization processing to obtain multiple regularized feature data; Based on the multiple regularized feature data, the steps of completing missing feature values ​​and obtaining completed features are performed.

4. The method according to claim 1, characterized in that, The acquisition of multiple self-expressive features of the multiple categories under the multiple modalities based on the completion features of the at least one modality and the feature data of the modalities other than the at least one modality includes: For the feature data or completion features of any of the multiple modalities, the average value of the feature values ​​or completions belonging to the same category is linearly combined to obtain the self-expressive features of the multiple categories under any of the modalities.

5. The method according to claim 1, characterized in that, The step of performing spectral clustering on the multiple self-expressed features based on the multiple affinity graphs to obtain the shared features of the multiple feature data of the multiple modalities includes: For any given clustering process, obtain multiple graph Laplacian matrices of the multiple affinity graphs; Based on the multiple graph Laplacian matrices, multiple clustering indicator matrices of the multiple modalities are obtained; Based on the reconstruction error of the multiple self-expressive features, the clustering error of any clustering process, and the similarity difference between the multiple clustering indicator matrices and the features to be solved, the loss function value of any clustering process is obtained. The parameters of the feature to be solved are iteratively adjusted until the loss function value meets the stopping condition. The iteration is then stopped, and the feature to be solved at the last iteration is obtained as the shared feature.

6. The method according to claim 5, characterized in that, For the at least one modality containing missing feature values, after filling in the missing feature values ​​in the feature data corresponding to the at least one modality based on the average value of the feature data corresponding to the at least one modality to obtain the completed features of the at least one modality, the method further includes: For any one of the multiple modalities, obtain the target weight matrix corresponding to the any one modality, wherein the weight coefficients of the feature values ​​that are not missing in the any one modality in the target weight matrix are greater than the weight coefficients of the feature values ​​that are missing. During the process of obtaining the loss function value, the target weight matrix is ​​invoked to weight the reconstruction error of the self-expressive feature corresponding to any modality.

7. The method according to claim 6, characterized in that, The target weight matrix is ​​a diagonal weight matrix, the weight coefficient of the non-missing feature values ​​is 1, and the weight coefficient of the missing feature values ​​is the ratio of the number of samples with non-missing feature values ​​in any modality to the total number of samples.

8. The method according to claim 5, characterized in that, The process of obtaining the multiple graph Laplacian matrices of the multiple affinity graphs includes: Based on any one of the multiple affinity graphs, obtain the similarity matrix and diagonal matrix of any one affinity graph; The difference between the diagonal matrix and the similarity matrix is ​​obtained as the graph Laplacian matrix of any affinity graph.

9. The method according to claim 5, characterized in that, The iterative adjustment of the parameters of the feature to be solved includes: Based on the alternating direction multiplier method, the values ​​of multiple intermediate variables in the loss function are iteratively adjusted to obtain the adjusted intermediate variables. When adjusting the value of any intermediate variable, the values ​​of the intermediate variables other than the adjusted intermediate variable are kept unchanged. Keeping the adjusted values ​​of the intermediate variables unchanged, perform the step of iteratively adjusting the parameters of the feature to be solved.

10. The method according to claim 5, characterized in that, The stopping condition is that the difference between the loss function value of any clustering process and the loss function value of the previous clustering process is less than the loss threshold.

11. A data processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire multiple feature data belonging to multiple modalities, wherein at least one modality in the multiple modalities contains missing feature values ​​in the feature data, each modality in the multiple modalities corresponds to one or more feature data, the feature data corresponding to each modality in the multiple modalities corresponds to multiple categories, the multiple categories are multiple target objects, and the multiple modalities of the multiple target objects include at least the visual information and near-infrared information of the face images of each of the multiple target objects; The completion module is used to, for the at least one modality containing missing feature values, complete the missing feature values ​​in the feature data corresponding to the at least one modality based on the average value of the feature data corresponding to the at least one modality, so as to obtain the completed feature of the at least one modality; The second acquisition module includes a linear combination submodule and a clustering submodule; The linear combination submodule is used to obtain multiple self-expressive features of the multiple categories under the multiple modalities based on the completion features of the at least one modality and the feature data of the modalities other than the at least one modality among the multiple modalities; The clustering submodule includes: The construction unit is used to construct multiple affinity graphs of the multiple categories under the multiple modalities based on the multiple self-expressive features. Each affinity graph includes multiple nodes and multiple weighted edges. Each node is used to represent a self-expressive feature of a category under the corresponding modality. The weight carried by any weighted edge is used to represent the similarity between the two self-expressive features of the two categories corresponding to the two nodes of the weighted edge under the corresponding modality. The spectral clustering unit is used to perform spectral clustering operations on the multiple self-expressive features based on the multiple affinity graphs to obtain shared features of the multiple feature data of the multiple modalities. The shared features are used to represent the common information among the multiple feature data belonging to the multiple modalities respectively.

12. The apparatus according to claim 11, characterized in that, The completion module includes: The acquisition submodule is used to acquire the average value of the feature data corresponding to any one of the at least one modalities containing missing feature values. The filling submodule is used to fill the missing feature values ​​in the feature data corresponding to any modality with the average value of the feature data corresponding to any modality, so as to obtain the complete feature of any modality.

13. The apparatus according to claim 11, characterized in that, The device further includes: The regularization module is used to perform regularization processing on the multiple feature data to obtain multiple regularized feature data. The completion module is further configured to perform the step of completing missing feature values ​​and obtaining completed features based on the multiple regularized feature data.

14. The apparatus according to claim 11, characterized in that, The linear combination submodule is used to linearly combine the feature values ​​or the average value of the completions belonging to the same category for the feature data or completion features of any of the multiple modalities, so as to obtain the self-expressive features of the multiple categories under any modality.

15. The apparatus according to claim 11, characterized in that, The spectral clustering unit includes: The first acquisition subunit is used to acquire multiple graph Laplacian matrices of the multiple affinity graphs for any given clustering process; The second acquisition subunit is used to acquire multiple clustering indicator matrices of the multiple modalities based on the multiple graph Laplacian matrices; The third acquisition subunit is used to acquire the loss function value of any clustering process based on the reconstruction error of the multiple self-expressive features, the clustering error of any clustering process, and the similarity difference between the multiple clustering indicator matrices and the features to be solved. An iterative adjustment subunit is used to iteratively adjust the parameters of the feature to be solved until the loss function value meets the stopping condition, at which point the iteration stops and the feature to be solved at the last iteration is obtained as the shared feature.

16. The apparatus according to claim 15, characterized in that, The device further includes: The third acquisition module is used to acquire a target weight matrix corresponding to any one of the multiple modalities, wherein the weight coefficients of the feature values ​​that are not missing in any one modality in the target weight matrix are greater than the weight coefficients of the feature values ​​that are missing. The third acquisition subunit is further configured to, during the process of acquiring the loss function value, call the target weight matrix to weight the reconstruction error of the self-expressive feature corresponding to any modality.

17. The apparatus according to claim 16, characterized in that, The target weight matrix is ​​a diagonal weight matrix, the weight coefficient of the non-missing feature values ​​is 1, and the weight coefficient of the missing feature values ​​is the ratio of the number of samples with non-missing feature values ​​in any modality to the total number of samples.

18. The apparatus according to claim 15, characterized in that, The first acquisition subunit is used for: Based on any one of the multiple affinity graphs, obtain the similarity matrix and diagonal matrix of any one affinity graph; The difference between the diagonal matrix and the similarity matrix is ​​obtained as the graph Laplacian matrix of any affinity graph.

19. The apparatus according to claim 15, characterized in that, The iterative adjustment subunit is used for: Based on the alternating direction multiplier method, the values ​​of multiple intermediate variables in the loss function are iteratively adjusted to obtain the adjusted intermediate variables. When adjusting the value of any intermediate variable, the values ​​of the intermediate variables other than the adjusted intermediate variable are kept unchanged. Keeping the adjusted values ​​of the intermediate variables unchanged, perform the step of iteratively adjusting the parameters of the feature to be solved.

20. The apparatus according to claim 15, characterized in that, The stopping condition is that the difference between the loss function value of any clustering process and the loss function value of the previous clustering process is less than the loss threshold.

21. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the data processing method as described in any one of claims 1 to 10.

22. A storage medium, characterized in that, The storage medium stores at least one computer program, which is loaded and executed by a processor to implement the data processing method as described in any one of claims 1 to 10.

23. A computer program product comprising one or more lines of program code stored in a computer-readable storage medium, wherein one or more processors of a computer device are capable of reading the one or more lines of program code from the computer-readable storage medium, and wherein the one or more processors execute the one or more lines of program code to enable the computer device to perform a data processing method as described in any one of claims 1 to 10.

Images