Multi-modal data processing method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a multi-modal data processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The multi-modal data processing method and device are applied to various data processing scenes for feature alignment, such as cloud technology, artificial intelligence, intelligent transportation, games, virtual reality and vehicle-mounted equipment. The multi-modal data processing method comprises the steps that features of source modal sample data in a target modal feature space are extracted through a first to-be-trained model, target modal estimation features are obtained, and the first to-be-trained model is a to-be-trained artificial neural network model; the first to-be-trained model is used for extracting features of the data of the source mode in a target mode feature space; calculating a feature loss function value between the target modal estimation feature and the target modal sample feature by adopting P loss functions; and training the first to-be-trained model based on the feature loss function value to obtain a feature alignment model. According to the invention, the accuracy of model training can be improved.

Description

Technical Field

[0001] This application relates to data processing technology in the field of artificial intelligence, and more particularly to a multimodal data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Feature alignment refers to the process of acquiring data features of a target modality based on data from a source modality. Feature alignment enhances the flexibility of multimodal data processing. Generally, feature alignment is achieved using artificial neural network models. However, when training these models, a single loss function is typically used to calculate the loss value, which affects the accuracy of the loss function and consequently, the accuracy of the model training. Summary of the Invention

[0003] This application provides a multimodal data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product that can improve the accuracy of model training.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a multimodal data processing method, the method comprising:

[0006] Obtain source modal sample data and target modal sample data, wherein the source modal sample data and the target modal sample data are sample data describing the same object from different modalities;

[0007] The first model to be trained is used to extract the features of the source modality sample data in the target modality feature space to obtain the target modality prediction features. The first model to be trained is an artificial neural network model to be trained. The first model to be trained is used to extract the features of the source modality data in the target modality feature space.

[0008] P loss functions are used to calculate the feature loss function value between the predicted features of the target modality and the features of the target modality samples. The P loss functions include two or three of the following: error loss function, similarity loss function and mutual exclusion loss function, where P>1 and P is an integer.

[0009] The first training model is trained based on the feature loss function value to obtain a feature alignment model. This application embodiment provides a multimodal data processing apparatus, the multimodal data processing apparatus comprising:

[0010] The data acquisition module is used to acquire source modal sample data and target modal sample data, wherein the source modal sample data and the target modal sample data are sample data describing the same object from different modalities;

[0011] The feature alignment module is used to extract the features of the source modality sample data in the target modality feature space using a first model to be trained, so as to obtain the target modality prediction features. The first model to be trained is an artificial neural network model to be trained. The first model to be trained is used to extract the features of the source modality data in the target modality feature space.

[0012] The loss calculation module is used to calculate the feature loss function value between the target modality prediction feature and the target modality sample feature using P loss functions. The P loss functions include two or three of the following: error loss function, similarity loss function and mutual exclusion loss function, where P>1 and P is an integer.

[0013] The model training module is used to train the first model to be trained based on the feature loss function value to obtain the feature alignment model.

[0014] In this embodiment, the loss calculation module is further configured to perform the following processing for each of the M feature channels corresponding to the target modality prediction features and the target modality sample features, where M>1 and M is an integer: obtaining the channel prediction features corresponding to the feature channel from the target modality prediction features; obtaining the channel sample features corresponding to the feature channel from the target modality sample features; calculating the sub-channel loss function value between the channel prediction features and the channel sample features using the loss function; obtaining the channel loss function value based on the P sub-channel loss function values ​​corresponding to the P loss functions; and obtaining the feature loss function value based on the M channel loss function values ​​corresponding to the M feature channels.

[0015] In this embodiment of the application, when the loss function is an error loss function, the loss calculation module is further used to calculate the channel feature difference between the channel prediction feature and the channel sample feature; and to determine the sub-channel loss function value based on the channel feature difference.

[0016] In this embodiment of the application, when the loss function is a similarity loss function, the loss calculation module is further configured to fuse the channel prediction feature and the channel sample feature to obtain a first fused feature; fuse the absolute value of the channel prediction feature and the absolute value of the channel sample feature to obtain a second fused feature; and obtain the sub-channel loss function value that is positively correlated with the first fused feature and negatively correlated with the second fused feature.

[0017] In this embodiment of the application, when the loss function is a mutually exclusive loss function, the loss calculation module is further configured to perform the following processing on the k-th channel prediction feature and the corresponding k-th channel sample feature of the k-th object to be processed, based on the K channel prediction features corresponding to the K objects to be processed and the K channel sample features corresponding to the K objects to be processed, where 1≤k≤K, K>1, and K is an integer: obtain a first total fusion result of fusing the k-th channel prediction feature with the K channel sample features respectively; obtain a second total fusion result of fusing the k-th channel sample feature with the K channel prediction features respectively; and determine the sub-channel loss function value by combining the k-th first fusion feature, the first total fusion result, and the second total fusion result, wherein the k-th first fusion feature is the fusion result of the k-th channel prediction feature and the k-th channel sample feature.

[0018] In this embodiment of the application, the loss calculation module is further configured to obtain a first mutually exclusive loss value that is positively correlated with the kth first fusion feature and negatively correlated with the first total fusion result; obtain a second mutually exclusive loss value that is positively correlated with the kth first fusion feature and negatively correlated with the second total fusion result; and combine the first mutually exclusive loss value and the second mutually exclusive loss value to determine the sub-channel loss function value.

[0019] In this embodiment, the loss calculation module is further configured to perform J downsampling operations on the target modality prediction features and the target modality sample features respectively, to obtain the downsampling alignment features and downsampling sample features corresponding to each downsampling operation, where J is a positive integer; use P loss functions to calculate the downsampling loss function values ​​between the downsampling alignment features and the downsampling sample features, to obtain J downsampling loss function values ​​corresponding to the J downsampling operations; and obtain the feature loss function values ​​based on the J downsampling loss function values.

[0020] In this embodiment of the application, the loss calculation module is further configured to use P loss functions to calculate the original scale loss function value between the target modality prediction feature and the target modality sample feature.

[0021] In this embodiment of the application, the loss calculation module is further configured to combine J of the downsampling loss function values ​​and the original scale loss function values ​​to obtain the feature loss function value.

[0022] In this embodiment, the feature alignment module is further configured to: obtain the channel feature dimension of each of the M feature channels corresponding to the target modality sample features; extract the features of the source modality sample data in the target modality feature space using the initial feature extraction module of the first model to be trained, to obtain initial alignment features; extract the channel prediction features of the initial alignment features based on the channel feature dimension in each of the M feature channels using the scaling module of the first model to be trained, to obtain M channel prediction features; and determine the M channel prediction features as the target modality prediction features.

[0023] In this embodiment of the application, the multimodal data processing device further includes a model application module, which is used to obtain the data to be aligned of the source modality in response to an information prediction request; to extract the features of the data to be aligned in the feature space of the target modality using the feature alignment model to obtain the alignment features to be processed; and to perform information prediction on the alignment features to be processed using the information prediction model to obtain a first information prediction result, wherein the information prediction model is used to perform information prediction on the data features of the target modality.

[0024] In this embodiment, the multimodal data processing device further includes a modality determination module, used to collect an information prediction dataset of a second model to be trained, wherein the second model to be trained is an artificial neural network model to be trained for information prediction of data features of the target modality; in the information prediction dataset, the sample data volume corresponding to Q data modalities is counted, where Q>1 and Q is an integer, and the Q data modalities include one or more of the following: text, image, audio, video, and point cloud; the data modality with the largest sample data volume is determined as the target modality; a target dataset corresponding to the target modality is obtained from the information prediction dataset; the second model to be trained is trained based on the target dataset to obtain the information prediction model.

[0025] In this embodiment of the application, the model training module is further configured to: use an information prediction model to predict the target modality prediction features to obtain a second information prediction result; use the information prediction model to predict the target modality sample features to obtain an information prediction label; combine the difference between the second information prediction result and the information prediction label to calculate an information prediction loss value; and combine the feature loss function value and the information prediction loss value to train the first model to be trained to obtain the feature alignment model.

[0026] In this embodiment, the information prediction includes virtual object generation, entity segmentation, virtual reality enhancement, virtual reality detection, and information querying. Virtual object generation refers to generating virtual objects based on the data features of the target modality; entity segmentation refers to segmenting entities based on the data features of the target modality; virtual reality enhancement refers to virtually enhancing real-world information based on the data features of the target modality; virtual reality detection refers to determining the objects to be enhanced based on the data features of the target modality; and information querying refers to querying related information based on the data features of the target modality.

[0027] In this embodiment of the application, the multimodal data processing device further includes a model application module, which is used to extract features from the data to be processed of the target modality to obtain features of the data to be processed; and to perform data processing on the data to be aligned and the data to be processed by combining the alignment features and the features of the data to be processed.

[0028] This application provides an electronic device for data processing, the electronic device comprising:

[0029] Memory is used to store executable instructions or computer programs.

[0030] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the multimodal data processing method provided in the embodiments of this application.

[0031] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the multimodal data processing method provided in this application.

[0032] This application provides a computer program product, including computer-executable instructions or a computer program, which, when executed by a processor, implements the multimodal data processing method provided in this application.

[0033] The embodiments of this application have at least the following beneficial effects: When training a first training model for feature alignment, the first training model is first used to perform feature alignment on the source modality sample data to obtain the target modality predicted features; then, P loss functions are used to calculate the feature loss function value between the target modality predicted features and the target modality sample features of the target modality sample data; finally, the first training model is trained based on the feature loss function value to obtain the feature alignment model. Since the feature loss function value calculated during the training process of the first training model is calculated using P loss functions, the diversity of loss functions used to calculate the loss function value is improved, and using multiple loss functions to calculate the loss function value can improve the accuracy of the feature loss function value; and the P loss functions include two or three of error loss functions, similarity loss functions, and mutual exclusion loss functions, which can improve the accuracy of the feature loss function value from at least two dimensions of numerical value, similarity, and mutual exclusion; thus, when the model is trained based on accurate feature loss function values, the accuracy of model training can be improved. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the architecture of the multimodal data processing system provided in the embodiments of this application;

[0035] Figure 2 This is one of the embodiments provided in this application. Figure 1 A schematic diagram of the server structure in the diagram;

[0036] Figure 3 This is a flowchart illustrating the multimodal data processing method provided in the embodiments of this application. Figure 1 ;

[0037] Figure 4 This is a flowchart illustrating the multimodal data processing method provided in the embodiments of this application. Figure 2 ;

[0038] Figure 5 This is a schematic diagram of the process for obtaining feature loss function values ​​provided in an embodiment of this application;

[0039] Figure 6 This is a flowchart illustrating the multimodal data processing method provided in the embodiments of this application. Figure 3 ;

[0040] Figure 7 This is a flowchart illustrating the multimodal data processing method provided in the embodiments of this application. Figure 4 ;

[0041] Figure 8 This is a training diagram of an exemplary feature alignment model provided in an embodiment of this application;

[0042] Figure 9This is an exemplary training diagram of feature alignment provided in an embodiment of this application;

[0043] Figure 10 This is another exemplary feature alignment diagram provided in the embodiments of this application;

[0044] Figure 11 This is a schematic diagram illustrating an exemplary information generation result provided in an embodiment of this application;

[0045] Figure 12 This is an exemplary feature alignment application diagram provided in an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0048] In the following description, the terms "first" and "second" are used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0049] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0050] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for descriptive purposes only and is not intended to limit the scope of this application.

[0051] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0052] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0053] 1) Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science used to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0054] It should be noted that artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large models or foundational models, can be widely applied to downstream tasks in various areas of AI after fine-tuning. AI software technologies include computer vision, speech processing, natural language processing, and machine learning / deep learning. In the embodiments of this application, feature alignment can be achieved using AI technology.

[0055] 2) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It is used to study how computers simulate or implement human learning behavior to acquire new knowledge or skills; and to reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence. Machine learning applications are found in all areas of artificial intelligence. Machine learning / deep learning typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning. Large models represent the latest development in machine learning / deep learning, integrating these techniques. In the embodiments of this application, feature alignment can be achieved by combining machine learning / deep learning.

[0056] 3) Artificial neural networks are mathematical models that mimic the structure and function of biological neural networks. Exemplary structures of artificial neural networks in this application include Graph Convolutional Networks (GCNs, a type of neural network for processing graph-structured data), Deep Neural Networks (DNNs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Neural State Machines (NSMs), and Phase-Functioned Neural Networks (PFNNs). In this application, feature alignment can be achieved using an artificial neural network model; and the first training model, feature alignment model, second training model, and information prediction model provided in this application can all be artificial neural network models.

[0057] 4) Feature alignment refers to the process of acquiring data features of the target modality based on the data of the source modality. In other words, feature alignment means transforming the data of the source modality into features in the feature space of the target modality. For example, when the source modality data is image data and the target modality is a point cloud, feature alignment means acquiring the features corresponding to the point cloud space from the image data.

[0058] 5) Feature channels refer to the dimensions used to represent data features in the field of artificial intelligence, typically referring to filters or kernels in convolutional layers. Each feature channel corresponds to a different feature learned by the artificial neural network model, such as edge features, texture features, and color features. The number of feature channels is positively correlated with the feature representation capability; therefore, the number of feature channels is usually determined based on the complexity of the task.

[0059] 6) Mean Absolute Error (MAE), also known as the L1 loss function, measures the prediction error of an artificial neural network model by calculating the absolute difference between the predicted and true features. For a single sample, the absolute difference between the predicted and true features of that sample is the MAE; for multiple samples, the average of the multiple absolute differences corresponding to the multiple samples is the MAE. See formula (6) for an example calculation process.

[0060] 7) Mean squared error, also known as the L2 loss function, is a measure of the prediction error of an artificial neural network model by calculating the squared difference between the predicted features and the true features. For a single sample, the squared difference between the predicted features and the true features of that sample is the mean squared error; for multiple samples, the average of the squared differences corresponding to the multiple samples is the mean squared error.

[0061] 8) Cosine similarity refers to the degree of similarity between predicted features and true features by calculating the cosine of the angle between them. The value of cosine similarity ranges from -1 to 1. The closer the value is to 1, the more similar the predicted features are to the true features, and the closer the value is to -1, the less similar the predicted features are to the true features.

[0062] It should be noted that feature alignment is typically achieved through artificial neural network models. However, when training these models, a single loss function is usually used to calculate the loss value, which affects the accuracy of the loss function. Furthermore, the aligned features are usually in vector form, resulting in the loss of information from the source modality data, further impacting the accuracy of the trained feature alignment model. In summary, these factors affect the accuracy of model training.

[0063] Based on this, embodiments of this application provide a multimodal data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of model training. The following describes exemplary applications of the electronic device for data processing (hereinafter referred to as the multimodal data processing device) provided in embodiments of this application. The data processing device provided in embodiments of this application can be implemented as various types of terminals such as robots, smartphones, smartwatches, laptops, tablets, desktop computers, smart home appliances, set-top boxes, smart vehicle devices, portable music players, personal digital assistants, dedicated messaging devices, intelligent voice interaction devices, portable gaming devices, and smart speakers. It can also be implemented as a server, or a combination of both. The following will describe exemplary applications when the multimodal data processing device is implemented as a server.

[0064] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the multimodal data processing system provided in the embodiments of this application; as shown... Figure 1 As shown, to support a multimodal data processing application, in the multimodal data processing system 100, terminals 200 (terminals 200-1 and 200-2 are shown as examples) are connected to server 400 via network 300. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. Additionally, the multimodal data processing system 100 also includes a database 500 for providing data support to server 400; and... Figure 1 The example shown illustrates a scenario where the database 500 is independent of the server 400. However, the database 500 can also be integrated into the server 400, and this embodiment does not limit this to any particular case.

[0065] Terminal 200 is used to receive rendering data sent by server 400 via network 300 and display the rendering data (exemplary graphical interfaces 210-1 for displaying a chair and 210-2 for displaying a vehicle are shown).

[0066] Server 400 is used to acquire source modality sample data and target modality sample data, and target modality sample features. It uses a first training model to extract features of the source modality sample data in the target modality feature space to obtain predicted target modality features. It uses P loss functions to calculate the feature loss function values ​​between the predicted target modality features and the target modality sample features. Based on the feature loss function values, it trains the first training model to obtain a feature alignment model. In response to an information prediction request, it obtains the source modality data to be aligned. It uses the feature alignment model to align the data to be aligned with the target modality features to obtain the aligned features to be processed. It uses an information prediction model to predict the information of the aligned features to be processed, obtaining a first information prediction result. This information prediction model is used to predict the information of the target modality data features. Based on the first information prediction result, it generates rendering data and sends the rendering data to terminal 200 via network 300.

[0067] In some embodiments, server 400 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0068] See Figure 2 , Figure 2 This is one of the embodiments provided in this application. Figure 1 A schematic diagram of the server structure in the diagram; such as Figure 2 As shown, server 400 includes at least one processor 410, memory 450, and at least one network interface 420. The various components in server 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0069] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0070] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0071] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0072] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0073] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0074] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB).

[0075] In some embodiments, the multimodal data processing apparatus provided in this application can be implemented in software. Figure 2 A multimodal data processing device 455 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a data acquisition module 4551, a feature alignment module 4552, a loss calculation module 4553, a model training module 4554, a model application module 4555, and a modality determination module 4556. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0076] In some embodiments, the multimodal data processing apparatus provided in this application can be implemented in hardware. As an example, the multimodal data processing apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the multimodal data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0077] In some embodiments, the terminal or server can implement the multimodal data processing method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as game APPs or virtual reality APPs; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0078] The following will describe the multimodal data processing method provided in this application embodiment, with reference to exemplary applications and implementations of the multimodal data processing device provided in the embodiments of this application. Furthermore, the multimodal data processing method provided in the embodiments of this application is applied to various data processing scenarios requiring feature alignment, such as cloud technology, artificial intelligence, smart transportation, games, virtual reality, and vehicle-mounted systems.

[0079] See Figure 3 , Figure 3 This is a flowchart illustrating the multimodal data processing method provided in the embodiments of this application. Figure 1 ,in, Figure 3 The main body executing each step is the multimodal data processing equipment; the following will combine... Figure 3 The steps shown are explained.

[0080] Step 101: Obtain source modal sample data and target modal sample features of target modal sample data.

[0081] In this embodiment, when the multimodal data processing device acquires a feature alignment dataset to train an artificial neural network model for feature alignment, it can obtain sample data of two modalities for the same object, namely source modal sample data and target modal sample data. It can also obtain the features of the target modal sample data, namely the target modal sample features. The same object can be, for example, a specified building, audio recording data, news information, a virtual object, or a virtual scene.

[0082] It should be noted that the source modality sample data and the target modality sample data can be a training sample in the form of a binary tuple, which can be any training sample included in the feature alignment dataset. In this case, the multimodal data processing device acquires the source modality sample data and the target modality sample features of the target modality sample data, including: acquiring the source modality sample data and the target modality sample data, and extracting features from the target modality sample data to obtain the target modality sample features. Furthermore, the source modality sample data and the target modality sample features can be a training sample in the form of a binary tuple, which can be any training sample in the feature alignment dataset.

[0083] It should also be noted that the source modality sample data and the target modality sample data are sample data describing the same object from different modalities, and the source modality sample data is sample data describing the object from the source modality, while the target modality sample data is sample data describing the object from the target modality. The source modality and the target modality are different modalities, and the modality can be any one of text, image, audio, video, and point cloud. Here, the target modality sample features are the features of the target modality sample data, and are the labels that align the features of the source modality sample data to the target modality.

[0084] Step 102: Use the first model to be trained to extract the features of the source modality sample data in the target modality feature space to obtain the target modality prediction features.

[0085] In this embodiment of the application, the artificial neural network model to be trained, used to align the source modality data to the target modality, is the first model to be trained. After obtaining the source modality sample data, the multimodal data processing device first uses the first model to train the source modality sample data to align the source modality sample data to the target modality. The obtained feature alignment result is the target modality predicted feature. Thus, the target modality predicted feature is the feature corresponding to the source modality sample data in the feature space of the target modality.

[0086] It should be noted that the first model to be trained is an artificial neural network model to be trained for aligning the data of the source modality to the features of the target modality; or, in other words, the first model to be trained is an artificial neural network model to be trained for extracting the features of the data of the source modality in the feature space of the target modality; the first model to be trained can be the original artificial neural network model that has been constructed, or it can be a pre-trained artificial neural network model, etc., and this application embodiment does not limit this. In addition, the first model to be trained includes a source modality sample data input module, a feature extraction module for the target modality feature space, and a feature output module; wherein, the source modality sample data input module is used to obtain the input source modality sample data and perform embedding representation on the source modality sample data; the feature extraction module is used to map the embedding representation result to the feature space of the target modality to obtain the target modality predicted features; the feature output module is used to output the target modality predicted features.

[0087] See Figure 4 , Figure 4 This is a flowchart illustrating the multimodal data processing method provided in the embodiments of this application. Figure 2 ,in, Figure 4 The main body executing each step is a multimodal data processing device; such as Figure 4 As shown in the embodiment of this application, step 102 can be implemented by steps 1021 to 1024; that is, the multimodal data processing device uses the first model to be trained to extract the features of the source modal sample data in the target modal feature space to obtain the target modal prediction features, including steps 1021 to 1024. Each step will be described below.

[0088] Step 1021: For the M feature channels corresponding to the target modality sample features, obtain the channel feature dimension of each feature channel.

[0089] In this embodiment of the application, the multimodal data processing device acquires the feature size of the target modal sample features, and based on the feature size of the target modal sample features, performs feature alignment of the source modal sample data to the target modality, so as to obtain a feature alignment result with the same feature size as the target modal sample features.

[0090] It should be noted that the feature size of the target modality sample feature corresponds to M feature channels, where M>1 and M is an integer; and the feature dimension corresponding to each feature channel is called the channel feature dimension, which can be in vector form (in which case, the feature size of the target modality sample feature is two-dimensional) or in matrix form (in which case, the feature size of the target modality sample feature is three-dimensional). This application embodiment does not limit this.

[0091] Step 1022: Using the initial feature extraction module of the first model to be trained, extract the features of the source modality sample data in the target modality feature space to obtain the initial alignment features.

[0092] In this embodiment of the application, the first model to be trained includes an initial feature extraction module, which is used to perform initial feature alignment of the source modality sample data to the target modality; thereby, the multimodal data processing device uses the initial feature extraction module to perform initial feature alignment of the source modality sample data, and the obtained initial feature alignment result is the initial alignment feature.

[0093] Step 1023: Using the scaling module of the first model to be trained, extract the channel prediction features of the initial alignment features based on the channel feature dimension in each of the M feature channels to obtain M channel prediction features.

[0094] In this embodiment, the first model to be trained further includes a scale adjustment module, used to adjust the scale of the initial alignment features to obtain channel prediction features with the same feature size as the target modality sample features. The initial feature extraction module and the scale adjustment module are two sub-modules of the feature extraction module in the first model to be trained.

[0095] It should be noted that when aligning the source modality sample data to the target modality, the channel prediction features are the alignment features corresponding to each obtained feature channel, and the feature dimension of the channel prediction features is the same as the channel feature dimension. Here, there is a one-to-one correspondence between the M feature channels and the M channel prediction features.

[0096] Step 1024: Determine the predicted features of the M channels as the predicted features of the target mode.

[0097] In this embodiment, the multimodal data processing device determines the M channel predicted features as the target modal predicted features. Thus, the target modal predicted features include the M channel predicted features, and the target modal predicted features are consistent with the target modal sample features in terms of feature size.

[0098] Understandably, when aligning source modality sample data to target modality features, the feature size of the predicted features of the target modality is used to ensure consistency between the predicted and sample features. This provides feasibility for data processing between the predicted and sample features, improving the efficiency and accuracy of such processing. Furthermore, since both the predicted and sample features include features corresponding to multiple feature channels, the information content of the predicted features is increased, thus improving the accuracy of feature alignment and consequently, the accuracy of model training.

[0099] Step 103: Use P loss functions to calculate the feature loss function value between the target modality prediction features and the target modality sample features.

[0100] In this embodiment, the multimodal data processing device uses P loss functions (P>1, where P is an integer) to calculate the loss function value between the predicted features of the target modality and the sample features of the target modality. The P loss functions include two or three of the following: error loss function, similarity loss function, and mutual exclusion loss function. The error loss function represents the absolute difference (or squared difference) between the predicted features and the true features, such as mean absolute error or mean squared error. The similarity loss function represents the similarity between the predicted features and the true features, such as cosine similarity. Here, when the error loss function is applied to calculate the loss function value between the predicted features of the target modality and the sample features of the target modality, the predicted features are the predicted features of the target modality, and the true features are the sample features of the target modality. The mutual exclusion loss function represents the mutual exclusion between the predicted features and irrelevant features, where irrelevant features refer to features unrelated to the true features, such as features that are different from the objects described by the true features. In other words, multimodal data processing equipment uses two or three of the following: error loss function, similarity loss function, and mutual exclusion loss function, to jointly calculate the loss function value between the predicted features of the target modality and the features of the target modality samples, and the calculated loss function value is called the feature loss function value.

[0101] It should be noted that the feature loss function value represents the degree of deviation between the predicted features of the target modality calculated based on P loss function values ​​and the features of the target modality samples. The feature loss function value is determined based on at least two differences between the predicted features of the target modality and the features of the target modality samples, and is positively correlated with at least two of these differences. Thus, the feature loss function value characterizes the feature alignment effect of the first model to be trained. The at least two differences between the predicted features of the target modality and the features of the target modality samples include at least two of the following: the absolute difference (or squared difference) between the predicted features of the target modality and the features of the target modality samples calculated using the error loss function; the similarity between the predicted features of the target modality and the features of the target modality samples calculated using the similarity loss function; and the mutual exclusion score between irrelevant features of the predicted features of the target modality and the features of the target modality samples calculated using the mutual exclusion loss function.

[0102] See Figure 5 , Figure 5 This is a schematic diagram of the process for obtaining feature loss function values ​​provided in an embodiment of this application, wherein, Figure 5 The main body executing each step is a multimodal data processing device; such as Figure 5As shown in the embodiment of this application, step 103 can be implemented by steps 1031A to 1034A; that is, the multimodal data processing device uses P loss functions to calculate the feature loss function value between the target modality prediction feature and the target modality sample feature, including steps 1031A to 1034A. Each step is described below.

[0103] In this embodiment of the application, the multimodal data processing device performs the following processing (steps 1031A to 1033A) for each of the M feature channels corresponding to the target modality prediction features and the target modality sample features.

[0104] Step 1031A: Obtain the channel prediction features corresponding to the feature channels from the target modality prediction features.

[0105] It should be noted that since the target modality prediction features correspond to M feature channels, the multimodal data processing device can obtain the corresponding features from the target modality prediction features for each of the M feature channels; here, the features obtained from the target modality prediction features that correspond to the feature channels are called channel prediction features.

[0106] Step 1032A: Obtain the channel sample features corresponding to the feature channels from the target modal sample features.

[0107] It should be noted that since the target modal sample features correspond to M feature channels, the multimodal data processing device can obtain the corresponding features from the target modal sample features for each of the M feature channels; here, the features obtained from the target modal sample features that correspond to the feature channels are called channel sample features.

[0108] In this embodiment of the application, steps 1031A and 1032A are not sequential in execution. Figure 5 The example shown is an exemplary execution order.

[0109] In this embodiment of the application, the multimodal data processing device performs the following processing (step 1033A) for each of the P loss functions.

[0110] Step 1033A: Calculate the sub-channel loss function value between the channel predicted features and the channel sample features using the loss function.

[0111] In this embodiment, the multimodal data processing device uses each loss function to calculate the loss function value between the channel predicted features and the channel sample features, and the calculated loss function value is called the sub-channel loss function value.

[0112] It should be noted that the loss function used to calculate the sub-channel loss function value is one of the following: error loss, similarity loss, and mutual exclusion loss.

[0113] In this embodiment of the application, when the loss function is an error loss function, step 1033A can be implemented through steps 1033AA1 and 1033AA2. The error loss can be, for example, mean square error loss, average deviation error loss, root mean square error loss, mean absolute error loss, etc. That is, the multimodal data processing device uses the loss function to calculate the sub-channel loss function value between the channel predicted features and the channel sample features, including steps 1033AA1 and 1033AA2. Each step is described below.

[0114] Step 1033AA1: Calculate the channel feature difference between the channel predicted features and the channel sample features.

[0115] In this embodiment, the multimodal data processing device can calculate the channel feature difference between the channel predicted feature and the channel sample feature by subtraction, or by calculating a ratio, etc., and this embodiment does not limit the specific method used. The channel feature difference represents the difference between the channel predicted feature and the channel sample feature.

[0116] Step 1033AA2: Determine the sub-channel loss function value based on the channel feature difference.

[0117] In this embodiment, the multimodal data processing device can directly determine the channel feature difference as the sub-channel loss function value, or it can determine the processing result of the channel feature difference as the sub-channel loss function value. This embodiment does not limit the specific determination of the sub-channel loss function value. Here, the multimodal data processing device can obtain the processing result of the channel feature difference by calculating the absolute value or square value of the channel feature difference.

[0118] It is understandable that by using the error loss function to calculate the sub-channel loss function value between the channel predicted features and the channel sample features, the process of numerically calculating the loss function value of the feature channel dimension is realized, which can numerically improve the consistency between the channel predicted features and the channel sample features.

[0119] In this embodiment of the application, when the loss function is a similarity loss function (such as a cosine similarity loss function), step 1033A can be implemented by steps 1033AB1 to 1033AB3; that is, the multimodal data processing device uses the loss function to calculate the sub-channel loss function value between the channel predicted features and the channel sample features, including steps 1033AB1 to 1033AB3. Each step is described below.

[0120] Step 1033AB1: Fuse the channel prediction features and channel sample features to obtain the first fused feature.

[0121] It should be noted that the multimodal data processing device can fuse the channel predicted features and channel sample features by multiplication, by concatenation, or by weighted summation, etc., and this application embodiment does not limit this. Here, the first fused feature is the fusion result of the channel predicted features and channel sample features.

[0122] Step 1033AB2: Fuse the absolute value of the channel prediction feature with the absolute value of the channel sample feature to obtain the second fused feature.

[0123] It should be noted that the method by which the multimodal data processing device fuses the absolute values ​​of the channel predicted features and the absolute values ​​of the channel sample features is similar to the method by which it fuses the channel predicted features and the channel sample features, and will not be described again in the embodiments of this application. Here, the second fused feature refers to the fusion result of the absolute values ​​of the channel predicted features and the absolute values ​​of the channel sample features.

[0124] Step 1033AB3: Obtain the sub-channel loss function values ​​that are positively correlated with the first fusion feature and negatively correlated with the second fusion feature.

[0125] It should be noted that the sub-channel loss function value obtained by the multimodal data processing device is positively correlated with the first fusion feature and negatively correlated with the second fusion feature. Here, the multimodal data processing device can use the ratio calculated with the first fusion feature as the numerator and the second fusion feature as the denominator as the sub-channel loss function value.

[0126] It is understandable that by using a similarity loss function to calculate the sub-channel loss function value between the channel predicted features and the channel sample features, the process of calculating the loss function value of the feature channel dimension from the perspective of similarity is realized, which can improve the consistency between the channel predicted features and the channel sample features from the perspective of similarity.

[0127] In this embodiment of the application, when the loss function is a mutually exclusive loss function, step 1033A can be implemented by steps 1033AC1 to 1033AC3; that is, the multimodal data processing device uses the loss function to calculate the sub-channel loss function value between the channel predicted features and the channel sample features, including steps 1033AC1 to 1033AC3. Each step is described below.

[0128] In this embodiment of the application, the multimodal data processing device performs the following processing (steps 1033AC1 to 1033AC3) on the k-th channel predicted feature and the corresponding k-th channel sample feature of the k-th object to be processed, based on the k-th channel predicted feature and the corresponding k-th channel sample feature of the k-th object to be processed.

[0129] It should be noted that 1≤k≤K, K>1, and K is an integer; and the predicted feature of the kth channel and the sample feature of the kth channel correspond to the same object to be processed (i.e., the kth object to be processed).

[0130] Step 1033AC1: Obtain the first overall fusion result of fusing the predicted feature of the k-th channel with the sample features of the K channels respectively.

[0131] In this embodiment of the application, the multimodal data processing device fuses each channel sample feature in the fusion of the k-th channel predicted feature and the K-th channel sample features, and the fusion method is the same as the fusion method for obtaining the first fusion feature; thus, the multimodal data processing device can obtain K fusion results of the k-th channel predicted feature and the channel sample features, and can obtain the first total fusion result by accumulating the obtained K fusion results.

[0132] Step 1033AC2: Obtain the second overall fusion result of fusing the sample features of the k-th channel with the predicted features of the K channels respectively.

[0133] In this embodiment, the multimodal data processing device fuses each channel prediction feature in the fusion of the k-th channel sample feature and the K-th channel prediction feature, and the fusion method is the same as the fusion method for obtaining the first fusion feature; thus, the multimodal data processing device can obtain the fusion results of K k-th channel sample features and channel prediction features, and can obtain the second total fusion result by accumulating the obtained K fusion results.

[0134] Step 1033AC3: Combine the k-th first fusion feature, the first total fusion result, and the second total fusion result to determine the sub-channel loss function value.

[0135] It should be noted that the multimodal data processing device calculates the sub-channel loss function value by combining the k-th first fusion feature, the first total fusion result, and the second total fusion result.

[0136] In this embodiment of the application, the multimodal data processing device determines the sub-channel loss function value by combining the k-th first fusion feature, the first total fusion result, and the second total fusion result. This includes: the multimodal data processing device acquiring a first mutually exclusive loss value that is positively correlated with the k-th first fusion feature and negatively correlated with the first total fusion result; acquiring a second mutually exclusive loss value that is positively correlated with the k-th first fusion feature and negatively correlated with the second total fusion result; and finally, combining the first mutually exclusive loss value and the second mutually exclusive loss value to determine the sub-channel loss function value.

[0137] It should be noted that the first mutual exclusion loss value is the mutual exclusion of each sample data in the dataset determined from the channel prediction feature dimension; the first mutual exclusion loss value can be obtained by calculating the ratio with the k-th first fusion feature as the numerator and the first total fusion result as the denominator. The second mutual exclusion loss value is the mutual exclusion of each sample data in the dataset determined from the channel sample feature dimension; the second mutual exclusion loss value can be obtained by calculating the ratio with the k-th first fusion feature as the numerator and the second total fusion result as the denominator. Here, the multimodal data processing device can add the negative of half of the first mutual exclusion loss value to the negative of half of the second mutual exclusion loss value to combine the first and second mutual exclusion loss values ​​to obtain the sub-channel loss function value.

[0138] Understandably, by using a mutually exclusive loss function to calculate the sub-channel loss function value between the channel predicted features and the channel sample features, the calculation process of obtaining the loss function value of the feature channel dimension from the mutual exclusivity of the sample set is realized, which can improve the consistency between the channel predicted features and the channel sample features from the mutual exclusivity of the sample set.

[0139] Step 1034A: Based on the P sub-channel loss function values ​​corresponding to the P loss functions, obtain the channel loss function values; based on the M channel loss function values ​​corresponding to the M feature channel loss functions, obtain the feature loss function values.

[0140] In this embodiment, the multimodal data processing device can calculate a sub-channel loss function value for each loss function, thus obtaining P sub-channel loss function values ​​for P loss functions. Here, the multimodal data processing device integrates the P sub-channel loss function values ​​into a single channel loss function value. This can be achieved by summing the P sub-channel loss function values, using a weighted sum of the P sub-channel loss function values, using the statistical results of the P sub-channel loss function values ​​(e.g., average, mode, maximum / minimum values), or a combination of the above, etc. This embodiment does not limit the specific implementation. Furthermore, the multimodal data processing device can calculate a channel loss function value for each feature channel, thus obtaining M channel loss function values ​​for M loss functions. Here, the multimodal data processing device integrates the M channel loss function values ​​into a feature loss function value. This can be achieved by summing the M channel loss function values, using a weighted sum of the M channel loss function values, using the statistical results of the M channel loss function values ​​(e.g., average, mode, maximum / minimum values), or a combination thereof. This application does not limit the specific implementation of this feature loss function value.

[0141] It is understandable that by using P loss functions to calculate the loss function value between the target modality prediction feature and the target modality sample feature in each of the M feature channels, the processing of calculating the loss function value in the feature channel dimension is realized, which improves the accuracy of the loss function value and thus improves the accuracy of the feature loss function value.

[0142] See Figure 6 , Figure 6 This is a flowchart illustrating the multimodal data processing method provided in the embodiments of this application. Figure 3 ,in, Figure 6 The main body executing each step is a multimodal data processing device; such as Figure 6 As shown in the embodiment of this application, step 103 can be implemented by steps 1031B to 1033B; that is, the multimodal data processing device uses P loss functions to calculate the feature loss function value between the target modality prediction feature and the target modality sample feature, including steps 1031B to 1033B. Each step will be described below.

[0143] Step 1031B: Perform J downsampling operations on the target modality prediction features and target modality sample features respectively to obtain the downsampling alignment features and downsampling sample features corresponding to each downsampling operation.

[0144] It should be noted that when the multimodal data processing device performs J downsampling operations on the target modality prediction features and the target modality sample features respectively, the downsampling size of the target modality prediction features and the target modality sample features is the same each time. Therefore, the downsampling aligned features and downsampling sample features obtained in each downsampling operation are the same in feature size. In addition, the downsampling aligned features are the downsampling target modality prediction features, and the downsampling sample features are the downsampling sample features, where J is a positive integer.

[0145] Step 1032B: Using P loss functions, calculate the downsampling loss function value between the downsampling alignment feature and the downsampling sample feature to obtain J downsampling loss function values ​​corresponding to J downsampling times.

[0146] In this embodiment, the multimodal data processing device calculates corresponding loss function values ​​using P loss functions for each downsampling of the downsampled aligned features and downsampled sample features. These calculated loss function values ​​are referred to as downsampling loss function values. The downsampling loss function value represents the difference between the downsampled aligned features and the downsampled sample features at the downsampled feature size, and is used to measure the difference between the target modality prediction features and the target modality sample features. Since each downsampling operation can calculate a downsampling loss function value, j downsampling loss function values ​​can be calculated for J downsampling operations.

[0147] It should be noted that the process by which the multimodal data processing device uses P loss functions to calculate the downsampling loss function value between the downsampling alignment feature and the downsampling sample feature is similar to the process described in steps 1031A to 1034A, and will not be repeated here in this embodiment; that is, the multimodal data processing device can use P loss functions to calculate the loss function value between the downsampling alignment feature and the downsampling sample feature in each of the M feature channels, so as to obtain the downsampling loss function value based on the obtained M loss function values.

[0148] Step 1033B: Based on J downsampling loss function values, obtain the feature loss function values.

[0149] In this embodiment, the multimodal data processing device can obtain the feature loss function value by integrating J downsampling loss function values; wherein, the integration method can be summation, mean calculation, maximum / minimum determination, weighted summation, etc., and this embodiment does not limit the method.

[0150] See also Figure 6In this embodiment of the application, step 1034B is included before step 1033B; that is, before the multimodal data processing device obtains the feature loss function value based on J downsampling loss function values, the multimodal data processing method further includes step 1034B, which will be described below.

[0151] Step 1034B: Using P loss functions, calculate the original scale loss function value between the target modality prediction features and the target modality sample features.

[0152] It should be noted that the process by which the multimodal data processing device calculates the original scale loss function value between the target modality predicted features and the target modality sample features using P loss functions is the process of obtaining M channel loss function values ​​described in steps 1031A to 1034A. By integrating the M channel loss function values, the original scale loss function value is obtained. Therefore, based on the M channel loss function values, the multimodal data processing device obtains the feature loss function value. This can be done by directly using the original scale loss function value as the feature loss function value, or by integrating the original scale loss function value with J downsampling loss function values ​​to obtain the feature loss function value.

[0153] Accordingly, see [link to relevant documentation] Figure 6 In this embodiment of the application, step 1033B can be implemented through step 1033B1; that is, the multimodal data processing device obtains the feature loss function value based on J downsampling loss function values, including step 1033B1, which will be described below.

[0154] Step 1033B1: Combine the J downsampling loss function values ​​and the original scale loss function values ​​to obtain the feature loss function values.

[0155] In the embodiments of this application, when the multimodal data processing device obtains the feature loss function value by integrating J downsampling loss function values, it can either integrate the J downsampling loss function values ​​with the original scale loss function value into the feature loss function value, or it can directly integrate the J downsampling loss function values ​​into the feature loss function value. The embodiments of this application do not limit this.

[0156] It is understandable that by downsampling the target modality prediction features and the target modality sample features, and then determining the feature loss function value between the target modality prediction features and the target modality sample features based on the downsampling alignment features and the downsampling sample features, the consistency between the target modality prediction features and the target modality sample features is improved from different feature scales, which can improve the accuracy of the feature loss function value.

[0157] Step 104: Train the first model to be trained based on the feature loss function value to obtain the feature alignment model.

[0158] In this embodiment, the multimodal data processing device performs backpropagation on the first training model based on the feature loss function value to adjust the model parameters in the first training model, thereby training the first training model. Furthermore, the training of the first training model can be iterative. When the iterative training ends, the first training model trained in the current iteration is the aligned model. Here, the multimodal data processing device determines that the iterative training ends when it meets the training termination condition; otherwise, iterative training continues.

[0159] It should be noted that the feature alignment model is used to align the data to be aligned in the source modality to the target modality. The training termination condition can be reaching an accuracy threshold, reaching an iteration count threshold, reaching an iteration duration threshold, or a combination of the above, etc. This application does not limit this.

[0160] In this embodiment of the application, when the feature alignment model is used to perform feature alignment on the source modality's data to be aligned, and the information prediction model is used to predict the information of the obtained aligned features, step 104, in which the multimodal data processing device trains the first training model based on the feature loss function value to obtain the feature alignment model, includes: the multimodal data processing device first uses the information prediction model to predict the information of the target modality's estimated features to obtain the second information prediction result; then uses the information prediction model to predict the target modality's sample features to obtain the information prediction label; next, the information prediction loss value is calculated by combining the difference between the second information prediction result and the information prediction label; finally, the first training model is trained by combining the feature loss function value and the information prediction loss value to obtain the feature alignment model.

[0161] Understandably, when training the first model to be trained, the first model is used to align the features of the source modality sample data to obtain the predicted features of the target modality. Then, P loss functions are used to calculate the feature loss function value between the predicted features of the target modality and the features of the target modality sample data. Finally, the first model to be trained is trained based on the feature loss function value to obtain the feature-aligned model. Since the feature loss function value calculated during the training of the first model to be trained is calculated using P loss functions, the diversity of loss functions used to calculate the loss function value is increased. Using multiple loss functions to calculate the loss function value can improve the accuracy of the feature loss function value. Furthermore, the P loss functions include two or three of the error loss function, similarity loss function, and mutual exclusion loss function, which can improve the accuracy of the feature loss function value from at least two dimensions of numerical value, similarity, and mutual exclusion. Therefore, when the model is trained based on the accurate feature loss function value, the accuracy of the model training can be improved.

[0162] See Figure 7 , Figure 7 This is a flowchart illustrating the multimodal data processing method provided in the embodiments of this application. Figure 4 ,in, Figure 7 The main body executing each step is a multimodal data processing device; such as Figure 7 As shown in the embodiment of this application, after step 103, steps 105 to 107 are also included; that is, after the multimodal data processing device trains the first model to be trained based on the feature loss function value and obtains the feature alignment model, the multimodal data processing method further includes steps 105 to 107. Each step will be described below.

[0163] Step 105: In response to the information prediction request, obtain the data to be aligned for the source modality.

[0164] In this embodiment, when a request is made to perform information prediction based on the data to be aligned from the source modality—for example, when generating virtual objects based on text-type data or segmenting entities based on point cloud-type data—the multimodal data processing device receives the information prediction request. In response to the information prediction request, the multimodal data processing device obtains the data on which the requested information prediction is based, which is the data to be aligned from the source modality.

[0165] Step 106: Using a feature alignment model, extract the features of the data to be aligned in the target modal feature space to obtain the alignment features to be processed.

[0166] It should be noted that since the data features used for information prediction are the data features of the target modality, the multimodal data processing device uses a feature alignment model to extract the features of the data to be aligned in the feature space of the target modality, so as to align the data to be aligned to the target modality. The obtained feature alignment result is the alignment feature to be processed. It is easy to see that the alignment feature to be processed is the data feature corresponding to the target modality.

[0167] Step 107: Use the information prediction model to predict the information of the alignment features to be processed, and obtain the first information prediction result.

[0168] In this embodiment of the application, the multimodal data processing device is equipped with an information prediction model, or can call an information prediction model from other devices. The information prediction model is used to predict the information of the data features of the target modality. Thus, the multimodal data processing device uses the information prediction model to predict the information of the alignment features to be processed, and the obtained information prediction result is the first information prediction result.

[0169] It should be noted that information prediction includes virtual object generation, entity segmentation, virtual reality enhancement, virtual reality detection, and information querying. Virtual object generation refers to generating virtual objects based on data features of the target modality; entity segmentation refers to segmenting entities based on data features of the target modality; virtual reality enhancement refers to virtually enhancing real-world information based on data features of the target modality; virtual reality detection refers to identifying objects to be enhanced based on data features of the target modality; and information querying refers to querying related information based on data features of the target modality.

[0170] It is understandable that the feature alignment model can accurately align the source modality data to the target modality features, thereby improving the accuracy of the aligned features to be processed; and thus, when making information predictions based on the aligned features to be processed, the accuracy of information predictions can be improved.

[0171] In this embodiment, before step 107, the method further includes determining the target modality and obtaining an information prediction model based on the target modality. That is, before the multimodal data processing device uses the information prediction model to predict the alignment features to be processed and obtains the first information prediction result, the multimodal data processing method further includes: the multimodal data processing device collecting the information prediction dataset of the second model to be trained; and in the information prediction dataset, counting the sample data volume corresponding to Q data modalities respectively; thereby determining the data modality with the largest sample data volume as the target modality; then obtaining the target dataset corresponding to the target modality from the information prediction dataset; and finally, training the second model to be trained based on the target dataset to obtain the information prediction model.

[0172] It should be noted that the second model to be trained is an artificial neural network model to be trained for information prediction of data features of the target modality, where Q>1 and Q is an integer. The Q data modalities include one or more of the following: text, image, audio, video, and point cloud. In this embodiment, the data modality can be simply referred to as a modality. Furthermore, the second model to be trained includes a sample data input module, an information prediction module, and a prediction output module. The sample data input module is used to acquire input information and embed it into a representation. The information prediction module is used to map the embedding representation result to a specified feature space and then perform information prediction based on the mapped features. The prediction output module is used to output the information prediction result.

[0173] In this embodiment of the application, the multimodal data processing device uses a feature alignment model to align the data to be aligned to the target modality. After obtaining the alignment features to be predicted, the multimodal data processing device further includes: the multimodal data processing device extracts features from the data to be processed in the target modality to obtain the data features to be processed; and combines the alignment features to be processed and the data features to be processed to perform data processing on the data to be aligned and the data to be processed.

[0174] It should be noted that the multimodal data processing device combines at least one feature corresponding to at least one piece of data to be processed and at least one alignment feature corresponding to at least one piece of data to be aligned to perform data processing on at least one piece of data to be processed and at least one piece of data to be aligned. There is a correlation between at least one piece of data to be processed and at least one piece of data to be aligned, such as both being graph nodes of the same heterogeneous graph.

[0175] It is understandable that feature alignment can integrate heterogeneous data, improve information processing efficiency, and increase the amount of information obtained from the data, thereby improving the accuracy of data processing.

[0176] The following describes an exemplary application of the embodiments of this application in a real-world application scenario. This exemplary application describes the process of aligning text and point cloud data to an image for feature mapping. It is readily apparent that the multimodal data processing method provided by the embodiments of this application is applicable to the process of aligning data from any source modality to a target modality for feature mapping. Here, the process of aligning text and point cloud data to an image for feature mapping is used as an example for illustration.

[0177] See Figure 8 , Figure 8 This is a schematic diagram illustrating the training of an exemplary feature alignment model provided in an embodiment of this application; as shown... Figure 8 As shown, input information 8-1 includes image sample 8-11 (referred to as target modality sample data), text sample 8-12 (referred to as source modality sample data), and point cloud sample 8-13 (referred to as source modality sample data). Here, we will explain using images as the target modality and text and point clouds as the source modalities. First, image encoder 8-21 is used to extract features from image sample 8-11, resulting in image features 8-311. Text encoder 8-22 (referred to as the first model to be trained) is used to align text sample 8-12 to the image features, resulting in text alignment features 8-321 (referred to as initial alignment features). Point cloud encoder 8-23 (referred to as the first model to be trained) is used to align point cloud sample 8-13 to the image features, resulting in point cloud alignment features 8-331 (denoted as...). The feature size is a specified feature size, referred to as the target modality prediction feature. Then, the size adjustment network 8-41 is used to adjust the size of image feature 8-311 to the specified feature size, resulting in image feature 8-312 (denoted as f). I 0 This is referred to as the target modality sample feature; and the size adjustment network 8-42 is used to adjust the size of the text alignment feature 8-321 to the specified feature size, resulting in the text alignment feature 8-322 (denoted as...). Target modality prediction features). Next, the multi-scale consistency module 8-5 is used to downsample the point cloud alignment feature 8-331, image feature 8-312, and text alignment feature 8-322 by J times, respectively, to obtain point cloud alignment features 8-332 at J+1 scales (denoted as J+1). Image features at scales J+1, 8-313 (denoted as J+1) ) and J+1 scale text alignment features 8-323 (denoted as Then, for each scale j (j∈[0,J]) in the J+1 scales, the channel consistency module 8-6 is used to calculate the channel consistency loss 8-71 (denoted as ) between the point cloud alignment feature 8-332 and the image feature 8-313. This is referred to as the downsampling loss function value or the original scale loss function value. Similarly, the channel consistency module 8-6 is used to calculate the channel consistency loss 8-72 between the text alignment feature 8-323 and the image feature 8-313. Finally, the point cloud encoder 8-23 is trained based on the obtained J+1 channel consistency losses 8-71, and the text encoder 8-22 is trained based on the J+1 channel consistency losses 8-72.

[0178] The following sections explain the feature extraction process, the size adjustment process, the calculation process of the channel consistency loss value, and the calculation process of the multi-scale consistency loss value.

[0179] When extracting features from the input information, the image encoder 8-21 and text encoder 8-22 used can be backbone networks, such as Transformer, ResNet, etc.; the point cloud encoder 8-23 used can be any network for point cloud feature extraction, such as PointNet, PointNet, PointBert, etc.

[0180] When resizing features, the resizing network can be implemented using a deep learning network for image semantic segmentation (RefineNet).

[0181] In summary, the process of obtaining image features 8-312, text alignment features 8-322 and point cloud alignment features 8-331 can be described by equations (1) to (3), which are shown below.

[0182]

[0183] Wherein, Refine1 represents the size adjustment process corresponding to size adjustment network 8-41, and Refine2 represents the size adjustment process corresponding to size adjustment network 8-42; E I E represents the feature extraction process corresponding to image encoder 8-21. Te This represents the feature alignment processing corresponding to text encoder 8-22, E P This indicates the feature alignment processing corresponding to point cloud encoder 8-23; I is image sample 8-11, Te is text sample 8-12, and P is point cloud sample 8-13.

[0184] When calculating the channel consistency loss value, point cloud alignment features and image features are used as examples for explanation.

[0185] Point cloud alignment features corresponding to scale j and image features Since the feature size is C×N, where C represents the number of feature channels and N represents the information dimension of each feature channel; and when j=0, C=M. Therefore, point cloud alignment features... It can be expressed as equation (4), image features It can be expressed as equation (5), as shown below.

[0186]

[0187]

[0188] It should be noted that the point cloud alignment features are calculated on the c-th channel (c∈[1,C]). and image features Various loss values, including the average error loss function value Loss. 71 Cosine similarity loss function value Loss 489:；e and mutual exclusion loss function value (All of these are referred to as sub-channel loss function values), as shown in equations (6) to (8).

[0189]

[0190] Wherein, input information 8-1 represents the k-th sample (k∈[1,K]) in the batch, K is the total number of samples in the batch, and τ is a hyperparameter; additionally, b∈[1,K] represents the b-th sample in the batch. This indicates the aforementioned channel characteristic difference. This refers to the first fusion feature mentioned above. This refers to the second fusion feature mentioned above. This represents the second mutually exclusive loss value mentioned above. This represents the first mutually exclusive loss value mentioned above.

[0191] Based on equations (6) to (8), the consistency loss value corresponding to scale j on the c-th feature channel is obtained. (referred to as the channel loss function value), as shown in equation (9).

[0192]

[0193] Based on equation (9), the channel consistency loss value at scale j is obtained. As shown in equation (10).

[0194]

[0195] Based on equation (10), the multi-scale consistency loss value L is obtained. P (referred to as the feature loss function value), as shown in equation (11).

[0196]

[0197] in, Figure 8 The point cloud encoder trained based on J+1 channel consistency loss (8-71) (8-23) is based on the multi-scale consistency loss value L. P Training point cloud encoder 8-23.

[0198] Similarly, based on the multi-scale consistency loss value L P The calculation process can calculate the multi-scale consistency loss value L between text alignment features and image features. Te (Referred to as the feature loss function value), based on the multi-scale consistency loss value L Te Train the text encoder 8-22 and the size adjustment network 8-42.

[0199] For example, see Figure 9 , Figure 9 This is an exemplary training diagram of feature alignment provided in an embodiment of this application; as shown... Figure 9As shown, the triplet 9-1 describing the same object includes text data 9-11, image information 9-12, and point cloud data 9-13. Here, the point cloud is the source modality. When the image is the target modality, the point cloud encoder 9-21 is used to align the point cloud data 9-13 to the image features. Thus, the point cloud encoder 9-21 is trained based on the difference between the point cloud alignment feature 9-31 output by the point cloud encoder 9-21 and the image features 9-32 output by the image encoder 9-22 for feature extraction of the image information 9-12. When the text is the target modality, the point cloud encoder 9-21 is used to align the point cloud data 9-13 to the text features. Thus, the point cloud encoder 9-21 is trained based on the difference between the point cloud alignment feature 9-31 output by the point cloud encoder 9-21 and the text features 9-33 output by the text encoder 9-23 for feature extraction of the text data 9-11.

[0200] See Figure 10 , Figure 10 This is another exemplary feature alignment diagram provided in the embodiments of this application; as shown Figure 10 As shown, to increase the number of samples, for the same object 10-1, the point cloud encoder 10-31 can be trained using image information sets 10-21 rendered from different perspectives. Here, the image-to-text model 10-32 is used to obtain the text data corresponding to each image information in the image information set 10-21, resulting in the text data set 10-22. In this context, point cloud is the source modality. When the image is the target modality, point cloud encoder 10-31 is used to align point cloud data 10-23 to the image features. Thus, based on the difference between the point cloud alignment feature 10-41 output by point cloud encoder 10-31 and the image features 10-42 output by image encoder 10-34 for feature extraction of image information in image information set 10-21, point cloud encoder 10-31 is trained. When text is the target modality, point cloud encoder 10-31 is used to align point cloud data 10-23 to the text features. Thus, based on the difference between the point cloud alignment feature 10-41 output by point cloud encoder 10-31 and the text features 10-43 output by text encoder 10-33 for feature extraction of text data in text data set 10-22, point cloud encoder 10-31 is trained.

[0201] It should be noted that after the multimodal data processing method provided in this application is used to complete the feature alignment of the source modality to the target modality, a generative model can be used to generate information from the feature-aligned features.

[0202] For example, see Figure 11 , Figure 11 This is an exemplary schematic diagram of information generation result provided in an embodiment of this application; as shown below. Figure 11As shown, information 11-1 is a chair generated based on features after feature alignment; information 11-2 is a vehicle generated based on features after feature alignment; and information 11-3 is a virtual character generated based on features after feature alignment.

[0203] It should also be noted that after the feature alignment of the source modality data to the target modality is completed using the multimodal data processing method provided in the embodiments of this application, the feature-aligned features can be used to process the graph nodes in the heterogeneous graph.

[0204] For example, see Figure 12 , Figure 12 This is an exemplary feature alignment application diagram provided in an embodiment of this application; as shown... Figure 12 As shown, graph structure 12-1 is a heterogeneous graph, including graph nodes of different modalities. Graph nodes 12-11 for the image modality and 12-12 for the point cloud modality are illustrated exemplarily. When processing graph structure 12-1, graph nodes 12-11 can be feature-aligned to the point cloud modality. Then, graph structure 12-1 is processed based on the feature-aligned point cloud features and the point cloud features of graph node 12-12.

[0205] It is understood that by setting the size of the aligned features to C×N in this embodiment, the information loss caused by feature compression can be reduced, thus improving the accuracy of feature alignment. Furthermore, this embodiment can flexibly determine the feature alignment direction, enabling feature alignment from various source modalities to various target modalities, thereby improving the applicability of feature alignment. In addition, this embodiment employs multi-scale calculation of the loss function value, and in each channel of each scale, uses multiple loss function values ​​for calculation, increasing the constraint on the loss function value and thus improving the training effect of the feature alignment model.

[0206] The following continues to describe an exemplary structure of the multimodal data processing device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the multimodal data processing device 455 in the memory 450 may include:

[0207] The data acquisition module 4551 is used to acquire source modal sample data and target modal sample data, wherein the source modal sample data and the target modal sample data are sample data describing the same object from different modalities;

[0208] The feature alignment module 4552 is used to extract the features of the source modality sample data in the target modality feature space using a first model to be trained, and to obtain the target modality prediction features. The first model to be trained is an artificial neural network model to be trained. The first model to be trained is used to extract the features of the source modality data in the target modality feature space.

[0209] The loss calculation module 4553 is used to calculate the feature loss function value between the target modality prediction feature and the target modality sample feature using P loss functions. The P loss functions include two or three of the following: error loss function, similarity loss function and mutual exclusion loss function, where P>1 and P is an integer.

[0210] The model training module 4554 is used to train the first model to be trained based on the feature loss function value to obtain the feature alignment model.

[0211] In this embodiment, the loss calculation module 4553 is further configured to perform the following processing for each of the M feature channels corresponding to the target modality prediction features and the target modality sample features, where M>1 and M is an integer: obtaining the channel prediction features corresponding to the feature channel from the target modality prediction features; obtaining the channel sample features corresponding to the feature channel from the target modality sample features; calculating the sub-channel loss function value between the channel prediction features and the channel sample features using the loss function; obtaining the channel loss function value based on the P sub-channel loss function values ​​corresponding to the P loss functions; and obtaining the feature loss function value based on the M channel loss function values ​​corresponding to the M feature channels.

[0212] In this embodiment of the application, when the loss function is an error loss function, the loss calculation module 4553 is further used to calculate the channel feature difference between the channel prediction feature and the channel sample feature; and to determine the sub-channel loss function value based on the channel feature difference.

[0213] In this embodiment of the application, when the loss function is a similarity loss function, the loss calculation module 4553 is further configured to fuse the channel prediction feature and the channel sample feature to obtain a first fused feature; fuse the absolute value of the channel prediction feature and the absolute value of the channel sample feature to obtain a second fused feature; and obtain the sub-channel loss function value that is positively correlated with the first fused feature and negatively correlated with the second fused feature.

[0214] In this embodiment of the application, when the loss function is a mutually exclusive loss function, the loss calculation module 4553 is further configured to perform the following processing on the k-th channel prediction feature and the corresponding k-th channel sample feature of the k-th object to be processed, based on the K channel prediction features corresponding to the K objects to be processed and the K channel sample features corresponding to the K objects to be processed, where 1≤k≤K, K>1, and K is an integer: obtain a first total fusion result of fusing the k-th channel prediction feature with the K channel sample features respectively; obtain a second total fusion result of fusing the k-th channel sample feature with the K channel prediction features respectively; and determine the sub-channel loss function value by combining the k-th first fusion feature, the first total fusion result, and the second total fusion result, wherein the k-th first fusion feature is the fusion result of the k-th channel prediction feature and the k-th channel sample feature.

[0215] In this embodiment of the application, the loss calculation module 4553 is further configured to obtain a first mutually exclusive loss value that is positively correlated with the kth first fusion feature and negatively correlated with the first total fusion result; obtain a second mutually exclusive loss value that is positively correlated with the kth first fusion feature and negatively correlated with the second total fusion result; and combine the first mutually exclusive loss value and the second mutually exclusive loss value to determine the sub-channel loss function value.

[0216] In this embodiment, the loss calculation module 4553 is further configured to perform J downsampling operations on the target modality prediction features and the target modality sample features respectively, to obtain the downsampling alignment features and downsampling sample features corresponding to each downsampling operation, where J is a positive integer; use P loss functions to calculate the downsampling loss function values ​​between the downsampling alignment features and the downsampling sample features, to obtain J downsampling loss function values ​​corresponding to the J downsampling operations; and obtain the feature loss function values ​​based on the J downsampling loss function values.

[0217] In this embodiment of the application, the loss calculation module 4553 is further configured to use P loss functions to calculate the original scale loss function value between the target modality prediction feature and the target modality sample feature.

[0218] In this embodiment of the application, the loss calculation module 4553 is further configured to combine J of the downsampling loss function values ​​and the original scale loss function values ​​to obtain the feature loss function value.

[0219] In this embodiment, the feature alignment module 4552 is further configured to: obtain the channel feature dimension of each of the M feature channels corresponding to the target modality sample features; extract the features of the source modality sample data in the target modality feature space using the initial feature extraction module of the first model to be trained, to obtain initial alignment features; extract the channel prediction features of the initial alignment features based on the channel feature dimension in each of the M feature channels using the scaling module of the first model to be trained, to obtain M channel prediction features; and determine the M channel prediction features as the target modality prediction features.

[0220] In this embodiment of the application, the multimodal data processing device 455 further includes a model application module 4555, which is used to obtain the source modality data to be aligned in response to an information prediction request; extract the features of the data to be aligned in the target modality feature space using the feature alignment model to obtain the alignment features to be processed; and perform information prediction on the alignment features to be processed using the information prediction model to obtain a first information prediction result, wherein the information prediction model is used to perform information prediction on the data features of the target modality.

[0221] In this embodiment, the multimodal data processing device 455 further includes a modality determination module 4556, used to collect an information prediction dataset of a second model to be trained, wherein the second model to be trained is an artificial neural network model to be trained for information prediction of data features of the target modality; in the information prediction dataset, the sample data volume corresponding to Q data modalities is counted, where Q>1 and Q is an integer, and the Q data modalities include one or more of the following: text, image, audio, video, and point cloud; the data modality with the largest sample data volume is determined as the target modality; a target dataset corresponding to the target modality is obtained from the information prediction dataset; the second model to be trained is trained based on the target dataset to obtain the information prediction model.

[0222] In this embodiment of the application, the model training module 4554 is further configured to: use an information prediction model to predict the target modality prediction features to obtain a second information prediction result; use the information prediction model to predict the target modality sample features to obtain an information prediction label; calculate an information prediction loss value between the second information prediction result and the information prediction label; and train the first model to be trained by combining the feature loss function value and the information prediction loss value to obtain the feature alignment model.

[0223] In this embodiment, the information prediction includes virtual object generation, entity segmentation, virtual reality enhancement, virtual reality detection, and information querying. Virtual object generation refers to generating virtual objects based on the data features of the target modality; entity segmentation refers to segmenting entities based on the data features of the target modality; virtual reality enhancement refers to virtually enhancing real-world information based on the data features of the target modality; virtual reality detection refers to determining the objects to be enhanced based on the data features of the target modality; and information querying refers to querying related information based on the data features of the target modality.

[0224] In this embodiment of the application, the multimodal data processing device 455 further includes a model application module 4555, which is used to extract features from the data to be processed of the target modality to obtain features of the data to be processed; and to perform data processing on the data to be aligned and the data to be processed by combining the alignment features and the features of the data to be processed.

[0225] This application provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. A processor of a multimodal data processing device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the multimodal data processing device to perform the multimodal data processing method described in this application.

[0226] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the multimodal data processing method provided in this application. For example, ... Figure 3 The multimodal data processing method is shown.

[0227] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0228] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0229] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0230] As an example, computer-executable instructions can be deployed to execute on a single electronic device (in which case, this single electronic device is a multimodal data processing device), or to execute on multiple electronic devices located at one location (in which case, multiple electronic devices located at one location are multimodal data processing devices), or to execute on multiple electronic devices distributed across multiple locations and interconnected via a communication network (in which case, multiple electronic devices distributed across multiple locations and interconnected via a communication network are multimodal data processing devices).

[0231] It is understood that in the embodiments of this application, data such as sample data and data to be aligned are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required. The collection, use and processing of related data must comply with relevant laws, regulations and standards, and conform to the principles of legality, legitimacy and necessity. It does not involve obtaining data types prohibited or restricted by laws and regulations, and will not hinder the normal operation of the target website.

[0232] In summary, in the embodiments of this application, when training the first training model for feature alignment, the first training model is first used to perform feature alignment on the source modality sample data to obtain the target modality predicted features; then, P loss functions are used to calculate the feature loss function value between the target modality predicted features and the target modality sample features of the target modality sample data; finally, the first training model is trained based on the feature loss function value to obtain the feature alignment model. Since the feature loss function value calculated during the training process of the first training model is calculated using P loss functions, the diversity of loss functions used to calculate the loss function value is increased. Using multiple loss functions to calculate the loss function value can improve the accuracy of the feature loss function value; and since the P loss functions include two or three of the error loss function, similarity loss function, and mutual exclusion loss function, the accuracy of the feature loss function value can be improved from at least two dimensions: numerical value, similarity, and mutual exclusion. Therefore, when the model is trained based on accurate feature loss function values, the accuracy of model training can be improved. Furthermore, by setting the feature size of the feature alignment result to M feature channels, the information loss caused by feature compression can be reduced, thereby improving the accuracy of feature alignment and model training. Additionally, by selecting the data modality with the largest number of samples as the target modality, the accuracy of the information prediction model can be improved.

[0233] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A multimodal data processing method, characterized in that, The method includes: Obtain source modal sample data and target modal sample data, wherein the source modal sample data and the target modal sample data are sample data describing the same object from different modalities; The first model to be trained is used to extract the features of the source modality sample data in the target modality feature space to obtain the target modality prediction features. The first model to be trained is an artificial neural network model to be trained. The first model to be trained is used to extract the features of the source modality data in the target modality feature space. P loss functions are used to calculate the feature loss function value between the predicted features of the target modality and the features of the target modality samples. The P loss functions include two or three of the following: error loss function, similarity loss function and mutual exclusion loss function, where P>1 and P is an integer. The first training model is trained based on the feature loss function value to obtain the feature alignment model.

2. The method according to claim 1, characterized in that, The step of using P loss functions to calculate the feature loss function value between the target modality prediction features and the target modality sample features includes: In the M feature channels corresponding to the target modality prediction features and the target modality sample features, the following processing is performed for each feature channel, where M>1 and M is an integer: From the target modality prediction features, obtain the channel prediction features corresponding to the feature channels; From the target modality sample features, obtain the channel sample features corresponding to the feature channels; The loss function is used to calculate the sub-channel loss function value between the channel prediction feature and the channel sample feature. Based on the P sub-channel loss function values ​​corresponding to the P loss functions, the channel loss function value is obtained. The feature loss function value is obtained based on the M channel loss function values ​​corresponding to the M feature channels.

3. The method according to claim 2, characterized in that, When the loss function is the error loss function, the step of calculating the sub-channel loss function value between the channel predicted feature and the channel sample feature using the loss function includes: Calculate the channel feature difference between the predicted channel features and the channel sample features; The sub-channel loss function value is determined based on the channel feature difference.

4. The method according to claim 2, characterized in that, When the loss function is the similarity loss function, the step of using the loss function to calculate the sub-channel loss function value between the channel predicted feature and the channel sample feature includes: The channel prediction features and the channel sample features are fused to obtain the first fused feature; The absolute values ​​of the channel prediction features and the absolute values ​​of the channel sample features are fused to obtain a second fused feature; Obtain the sub-channel loss function value that is positively correlated with the first fusion feature and negatively correlated with the second fusion feature.

5. The method according to claim 2, characterized in that, When the loss function is the mutually exclusive loss function, the step of calculating the sub-channel loss function value between the channel predicted feature and the channel sample feature using the loss function includes: Based on the K channel prediction features corresponding to the K objects to be processed, and the K channel sample features corresponding to the K objects to be processed, the following processing is performed on the k-th channel prediction feature and the k-th channel sample feature corresponding to the k-th object to be processed, where 1≤k≤K, K>1, and K is an integer: Obtain the first overall fusion result of fusing the predicted feature of the kth channel with the sample features of the K channels respectively; Obtain the second overall fusion result by fusing the feature of the kth channel sample with the predicted features of the K channels respectively; The sub-channel loss function value is determined by combining the k-th first fusion feature, the first total fusion result, and the second total fusion result. The k-th first fusion feature is the fusion result of the k-th channel prediction feature and the k-th channel sample feature.

6. The method according to claim 5, characterized in that, The step of combining the k-th first fusion feature, the first total fusion result, and the second total fusion result to determine the sub-channel loss function value includes: Obtain the first mutually exclusive loss value that is positively correlated with the kth first fusion feature and negatively correlated with the first total fusion result; Obtain a second mutually exclusive loss value that is positively correlated with the kth first fusion feature and negatively correlated with the second total fusion result; The sub-channel loss function value is determined by combining the first mutual exclusion loss value and the second mutual exclusion loss value.

7. The method according to claim 1, characterized in that, The step of using P loss functions to calculate the feature loss function value between the target modality prediction features and the target modality sample features includes: The target modality prediction features and the target modality sample features are downsampled J times respectively to obtain the downsampled alignment features and downsampled sample features corresponding to each downsampling, where J is a positive integer; Using P loss functions, calculate the downsampling loss function value between the downsampling alignment feature and the downsampling sample feature to obtain J downsampling loss function values ​​corresponding to J downsampling times; The feature loss function value is obtained based on J downsampling loss function values.

8. The method according to claim 7, characterized in that, Before obtaining the feature loss function value based on J downsampling loss function values, the method further includes: Using P loss functions, calculate the original-scale loss function value between the predicted features of the target modality and the sample features of the target modality; The process of obtaining the feature loss function value based on J downsampling loss function values ​​includes: The feature loss function value is obtained by combining J downsampling loss function values ​​and the original scale loss function value.

9. The method according to any one of claims 1 to 8, characterized in that, The step of extracting features from the source modality sample data in the target modality feature space using a first model to be trained, to obtain the target modality prediction features, includes: For the M feature channels corresponding to the target modality sample features, obtain the channel feature dimension of each feature channel; The initial feature extraction module of the first model to be trained is used to extract the features of the source modality sample data in the target modality feature space to obtain the initial alignment features; Using the scaling module of the first model to be trained, channel prediction features of the initial alignment features are extracted based on the channel feature dimension in each of the M feature channels to obtain M channel prediction features; The M predicted features of the channels are determined as the predicted features of the target mode.

10. The method according to any one of claims 1 to 8, characterized in that, After training the first model to be trained based on the feature loss function value to obtain the feature alignment model, the method further includes: In response to the information prediction request, the data to be aligned for the source mode is obtained; Using the aforementioned feature alignment model, the features of the data to be aligned in the target modality feature space are extracted to obtain the alignment features to be processed; An information prediction model is used to predict the information of the alignment features to be processed, and a first information prediction result is obtained. The information prediction model is used to predict the information of the data features of the target modality.

11. The method according to claim 10, characterized in that, Before using an information prediction model to predict the information of the alignment features to be processed and obtaining the first information prediction result, the method further includes: Collect the information prediction dataset of the second model to be trained, wherein the second model to be trained is an artificial neural network model to be trained for information prediction of the data features of the target modality; In the information prediction dataset, the sample data volume corresponding to each of the Q data modalities is counted, where Q>1 and Q is an integer. The Q data modalities include one or more of the following: text, image, audio, video, and point cloud. The data mode with the largest amount of sample data is determined as the target mode; Obtain the target dataset corresponding to the target modality from the information prediction dataset; The second training model is trained based on the target dataset to obtain the information prediction model.

12. The method according to claim 10, characterized in that, The step of training the first model to be trained based on the feature loss function value to obtain the feature alignment model includes: The target modality prediction features are predicted using an information prediction model to obtain a second information prediction result. The information prediction model is used to predict the features of the target modality sample to obtain the information prediction label; Calculate the information prediction loss value between the second information prediction result and the information prediction label; The first training model is trained by combining the feature loss function value and the information prediction loss value to obtain the feature alignment model.

13. The method according to claim 10, characterized in that, The information prediction includes virtual object generation, entity segmentation, virtual reality enhancement, virtual reality detection, and information query. The virtual object generation refers to generating virtual objects based on the data features of the target modality; the entity segmentation refers to segmenting entities based on the data features of the target modality; the virtual reality enhancement refers to virtually enhancing real-world information based on the data features of the target modality; the virtual reality detection refers to determining the objects to be enhanced by virtual reality based on the data features of the target modality; and the information query refers to querying related information based on the data features of the target modality.

14. The method according to claim 10, characterized in that, After extracting the features of the data to be aligned in the target modality feature space using the feature alignment model to obtain the alignment features to be processed, the method further includes: Feature extraction is performed on the data to be processed for the target mode to obtain the features of the data to be processed; The data to be aligned and the data to be processed are combined and then processed.

15. A multimodal data processing device, characterized in that, The multimodal data processing device includes: The data acquisition module is used to acquire source modal sample data and target modal sample data, wherein the source modal sample data and the target modal sample data are sample data describing the same object from different modalities; The feature alignment module is used to extract the features of the source modality sample data in the target modality feature space using a first model to be trained, so as to obtain the target modality prediction features. The first model to be trained is an artificial neural network model to be trained. The first model to be trained is used to extract the features of the source modality data in the target modality feature space. The loss calculation module is used to calculate the feature loss function value between the target modality prediction feature and the target modality sample feature using P loss functions. The P loss functions include two or three of the following: error loss function, similarity loss function and mutual exclusion loss function, where P>1 and P is an integer. The model training module is used to train the first model to be trained based on the feature loss function value to obtain the feature alignment model.

16. An electronic device for multimodal data processing, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the multimodal data processing method according to any one of claims 1 to 14.

17. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the multimodal data processing method according to any one of claims 1 to 14 is implemented.

18. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the multimodal data processing method according to any one of claims 1 to 14 is implemented.

Images