Feature alignment model training method and apparatus, multi-modal data processing method and apparatus, electronic device, computer readable storage medium, and computer program product

Abstract

A feature alignment model training method and apparatus, a multi-modal data processing method and apparatus, an electronic device, a computer readable storage medium, and a computer program product, applied to various data processing scenarios for feature alignment such as cloud technology, artificial intelligence, smart transportation, games, virtual reality, and vehicle-mounted systems. The feature alignment model training method comprises: acquiring source modal sample data and a target modal sample feature of target modal sample data (101), wherein the source modal sample data and the target modal sample data are sample data describing the same object from different modalities; using a first model to be trained to extract a feature of the source modal sample data in a target modal feature space to obtain a target modal estimated feature (102); using P loss functions to calculate a feature loss function value between the target modal estimated feature and the target modal sample feature (103), wherein the P loss functions comprise two or three of the following loss functions: an error loss function, a similarity loss function, and a mutually exclusive loss function, and P is an integer greater than 1; and on the basis of the feature loss function value, training the first model to be trained to obtain a feature alignment model (104).

Description

Feature alignment model training method, multi-modal data processing method, device, electronic equipment, computer readable storage medium and computer program product

[0001] Cross-reference to Related Applications

[0002] The present application is based on and claims priority to Chinese Patent Application No. 202410658700.3, filed on May 24, 2024, the entire contents of which are hereby incorporated by reference into the present application. TECHNICAL FIELD

[0003] The present application relates to data processing technology in the field of artificial intelligence, and relates to but is not limited to a feature alignment model training method, a multi-modal data processing method, a device, an electronic equipment, a computer readable storage medium and a computer program product. BACKGROUND

[0004] Feature alignment refers to obtaining data features of a target modality based on data of a source modality. Through feature alignment, the flexibility of multi-modal data processing can be improved. Generally, feature alignment is usually implemented through an artificial neural network model. However, when training an artificial neural network model for feature alignment, a single loss function is usually used to calculate a loss function value, thereby affecting the accuracy of the loss function value and further affecting the accuracy of model training. SUMMARY

[0005] The embodiments of the present application provide a feature alignment model training method, a multi-modal data processing method, a device, an electronic equipment, a computer readable storage medium and a computer program product, which can improve the accuracy of the calculated feature loss function value, and further improve the accuracy of feature alignment model training.

[0006] The technical solutions of the embodiments of the present application are implemented as follows:

[0007] The embodiments of the present application provide a feature alignment model training method, which is executed by an electronic equipment. The method comprises: obtaining source modality sample data and target modality sample features of target modality sample data; the source modality sample data and the target modality sample data are sample data describing the same object from different modalities; extracting features of the source modality sample data in a target modality feature space by using a first to-be-trained model to obtain target modality estimated features; calculating feature loss function values between the target modality estimated features and the target modality sample features by using P loss functions; the P loss functions include two or three of the following loss functions: an error loss function, a similarity loss function and a mutual exclusion loss function, and P is an integer greater than 1; training the first to-be-trained model based on the feature loss function values to obtain a feature alignment model.

[0008] An embodiment of the present application provides a multi-modal data processing method, the method is executed by an electronic device, and the method comprises the following steps: in response to an information prediction request, determining to-be-aligned data of a source modality; using a feature alignment model, extracting features of the to-be-aligned data in a target modality feature space to obtain to-be-processed aligned features; the feature alignment model is trained by using the feature alignment model training method described above; using an information prediction model to perform information prediction on the to-be-processed aligned features to obtain a first information prediction result, and the information prediction model is used for performing information prediction on data features of a target modality.

[0009] An embodiment of the present application provides a feature alignment model training device, the feature alignment model training device comprises the following modules: a data acquisition module configured to acquire source modality sample data and target modality sample features of target modality sample data; the source modality sample data and the target modality sample data are sample data for describing a same object from different modalities; a feature alignment module configured to extract features of the source modality sample data in a target modality feature space by using a first to-be-trained model to obtain target modality estimated features; a loss calculation module configured to calculate feature loss function values between the target modality estimated features and the target modality sample features by using P loss functions; the P loss functions comprise two or three of the following loss functions: an error loss function, a similarity loss function and a mutual exclusion loss function, and P is an integer greater than 1; and a model training module configured to train the first to-be-trained model based on the feature loss function values to obtain a feature alignment model.

[0010] An embodiment of the present application provides a multi-modal data processing device, the multi-modal data processing device comprises the following modules: a data determination module configured to determine to-be-aligned data of a source modality in response to an information prediction request; a model calling module configured to extract features of the to-be-aligned data in a target modality feature space by using a feature alignment model to obtain to-be-processed aligned features; the feature alignment model is trained by using the feature alignment model training method described above; and the model calling module is further configured to perform information prediction on the to-be-processed aligned features by using an information prediction model to obtain a first information prediction result, and the information prediction model is used for performing information prediction on data features of a target modality.

[0011] An embodiment of the present application provides an electronic device, the electronic device comprises a memory configured to store computer executable instructions or computer programs, and a processor configured to execute the computer executable instructions or computer programs stored in the memory to implement the feature alignment model training method or the multi-modal data processing method provided in the embodiments of the present application.

[0012] The embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions or a computer program, and the computer executable instructions or the computer program are used to implement the feature alignment model training method or the multi-modal data processing method provided by the embodiment of the present application when executed by a processor.

[0013] The embodiment of the present application provides a computer program product, which comprises computer executable instructions or a computer program, and the computer executable instructions or the computer program are used to implement the feature alignment model training method or the multi-modal data processing method provided by the embodiment of the present application when executed by a processor.

[0014] The embodiment of the present application has at least the following beneficial effects: when the first to-be-trained model for feature alignment is trained, the first to-be-trained model is used to perform feature alignment on source modal sample data to obtain target modal estimated features; P loss functions are used to calculate feature loss function values between the target modal estimated features and target modal sample features of the target modal sample data; finally, the first to-be-trained model is trained based on the feature loss function values to obtain a feature alignment model. Since the calculated feature loss function values are calculated by using P loss functions in the training process of the first to-be-trained model, the diversity of the loss function used to calculate the feature loss function values is improved by calculating the feature loss function values by using multiple loss functions, and the accuracy of the feature loss function values is also improved by calculating the feature loss function values by using multiple loss functions. In addition, the P loss functions include two or three of the error loss function, the similarity loss function and the mutual exclusion loss function, so that the accuracy of the calculated feature loss function values is improved from at least two dimensions of the numerical value, the similarity and the mutual exclusion in the loss value calculation process, and then the accuracy of the model training is improved when the model is trained based on the accurate feature loss function values. BRIEF DESCRIPTION OF DRAWINGS

[0015] FIG. 1 is an architecture schematic diagram of a multi-modal data processing system provided by the embodiment of the present application;

[0016] FIG. 2 is a structural schematic diagram of a server provided by the embodiment of the present application;

[0017] FIG. 3 is a flow schematic diagram one of a feature alignment model training method provided by the embodiment of the present application;

[0018] FIG. 4 is a flow schematic diagram two of a feature alignment model training method provided by the embodiment of the present application;

[0019] FIG. 5A is a flow schematic diagram of obtaining a feature loss function value provided by the embodiment of the present application;

[0020] FIG. 5B is a flow diagram of calculating a sub-channel loss function value when the loss function is an error loss function according to an embodiment of the present application;

[0021] FIG. 5C is a flow diagram of calculating a sub-channel loss function value when the loss function is a similarity loss function according to an embodiment of the present application;

[0022] FIG. 5D is a flow diagram of calculating a sub-channel loss function value when the loss function is a mutual exclusion loss function according to an embodiment of the present application;

[0023] FIG. 6 is a flow diagram of a feature alignment model training method according to an embodiment of the present application;

[0024] FIG. 7 is a flow diagram of a multi-modal data processing method according to an embodiment of the present application;

[0025] FIG. 8 is a training diagram of an exemplary feature alignment model according to an embodiment of the present application;

[0026] FIG. 9 is a training diagram of an exemplary feature alignment according to an embodiment of the present application;

[0027] FIG. 10 is another exemplary feature alignment diagram according to an embodiment of the present application;

[0028] FIG. 11 is an exemplary information generation result diagram according to an embodiment of the present application;

[0029] FIG. 12 is an exemplary feature alignment application diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purposes, technical solutions and advantages of the present application clearer, the following will further describe the present application in conjunction with the accompanying drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without creative labor shall fall within the scope of protection of the present application.

[0031] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0032] In the following description, the terms "first\second" are used to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first\second" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] In the embodiments of this application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0034] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by one of ordinary skill in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0035] The relevant data collection and processing in the embodiments of the present application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and within the scope of authorization of the laws and regulations and the personal information subject, carry out subsequent data use and processing.

[0036] Before the embodiments of the present application are further described in detail, the terms and terms involved in the embodiments of the present application are explained, and the terms and terms involved in the embodiments of the present application are applicable to the following explanations.

[0037] 1) Artificial Intelligence (AI) is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. That is, artificial intelligence is a comprehensive technology of computer science, which is used to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines to enable machines to have perception, reasoning and decision-making functions. It should be noted that artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-training model technology, operation / interaction system, mechatronics, etc. Among them, the pre-training model is also called a large model or a basic model; the pre-training model can be widely used in downstream tasks of artificial intelligence in various directions after fine-tuning. Artificial intelligence software technology includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. In the embodiments of the present application, feature alignment can be achieved through artificial intelligence technology.

[0038] 2) Machine Learning (ML), is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis and algorithmic complexity theory, etc. It is used to study the computer simulation or implementation of human learning behavior to acquire new knowledge or skills; reorganize the existing knowledge structure to continuously improve its performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Machine learning is applied to various fields of artificial intelligence. Machine learning / deep learning usually includes artificial neural networks, belief networks, reinforcement learning, transfer learning and inductive learning, etc. Large models are the latest development of machine learning / deep learning, which integrates the above technologies. In the embodiments of the present application, feature alignment can be realized by combining machine learning / deep learning.

[0039] 3) Artificial neural network, is a mathematical model that simulates the structure and function of biological neural network. The exemplary structure of artificial neural network in the embodiments of the present application includes Graph Convolutional Network (GCN, a neural network for processing graph-structured data), Deep Neural Networks (DNN), Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN), Neural State Machine (NSM) and Phase-Functioned Neural Network (PFNN), etc. In the embodiments of the present application, feature alignment can be realized by artificial neural network model. The first to-be-trained model, the feature alignment model, the second to-be-trained model and the information prediction model provided in the embodiments of the present application can all be artificial neural network models.

[0040] 4) Feature alignment, refers to the process of obtaining the data features of the target modality based on the data of the source modality, that is, feature alignment refers to converting the data of the source modality to the features in the feature space of the target modality. For example, when the data of the source modality is image data and the target modality is point cloud, feature alignment refers to obtaining the features corresponding to the point cloud space through image data. The feature space of the target modality can be a shared feature space, therefore, feature alignment can also refer to converting the data of different modalities (such as image, text and point cloud, etc.) into a shared feature space, so that the data from different modalities have comparability and consistency at the feature level, thereby realizing effective fusion and interaction between different modalities.

[0041] 5) Feature channel refers to a dimension used to represent data features in the field of artificial intelligence, usually referring to filters or convolution kernels in convolution layers. Each feature channel corresponds to different features learned by an artificial neural network model, such as edge features, texture features, and color features, etc. The number of feature channels is positively correlated with the feature representation ability, so the number of feature channels can be determined based on the complexity of the task.

[0042] 6) Mean absolute error, also known as L1 loss function, refers to the prediction error of an artificial neural network model by calculating the absolute difference between the estimated feature and the true feature. For a single sample, the absolute difference between the estimated feature and the true feature of the sample is the mean absolute error; for multiple samples, the average of multiple absolute differences corresponding to multiple samples is the mean absolute error.

[0043] 7) Mean squared error, also known as L2 loss function, refers to the prediction error of an artificial neural network model by calculating the squared difference between the estimated feature and the true feature. For a single sample, the squared difference between the estimated feature and the true feature of the sample is the mean squared error; for multiple samples, the average of multiple squared differences corresponding to multiple samples is the mean squared error.

[0044] 8) Cosine similarity refers to the similarity between the estimated feature and the true feature by calculating the cosine value of the included angle between them. The cosine similarity value ranges from -1 to 1, and the closer the cosine similarity value is to 1, the more similar the estimated feature and the true feature are, and the closer the cosine similarity value is to -1, the less similar the estimated feature and the true feature are.

[0045] It should be noted that feature alignment is usually achieved through an artificial neural network model. However, when training an artificial neural network model for feature alignment, a single loss function is usually used to calculate the loss function value, which affects the accuracy of the loss function value. In addition, when training an artificial neural network model for feature alignment, the aligned features are usually vector-form features, which lose the information of the source modal data, and thus affect the accuracy of the trained feature alignment model. In summary, the feature alignment method in the related art affects the accuracy of model training.

[0046] Based on this, the embodiment of the present application provides a feature alignment model training method, a multi-modal data processing method, an apparatus, an electronic device, a computer readable storage medium and a computer program product, which can improve the accuracy of the calculated feature loss function value, and further improve the accuracy of feature alignment model training. The following describes an exemplary application of an electronic device (hereinafter referred to as a feature alignment model training device and a multi-modal data processing device) provided by the embodiment of the present application for performing feature alignment model training and multi-modal data processing. It should be noted that the electronic device for performing feature alignment model training and the electronic device for performing multi-modal data processing can be the same electronic device, or can be two different electronic devices. If the electronic device for performing feature alignment model training and the electronic device for performing multi-modal data processing are two different electronic devices, the structures of the two electronic devices can be completely the same. The embodiment of the present application takes the electronic device for performing feature alignment model training and the electronic device for performing multi-modal data processing as an example to illustrate. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as robots, smart phones, smart watches, notebook computers, tablet computers, desktop computers, smart home appliances, set-top boxes, smart car devices, portable music players, personal digital assistants, dedicated message devices, smart voice interaction devices, portable game devices and smart speakers. It can also be implemented as a server, and can also be implemented as a combination of the two. The following will illustrate an exemplary application when the electronic device is implemented as a server.

[0047] Referring to FIG. 1, FIG. 1 is an architecture schematic diagram of a multi-modal data processing system provided by the embodiment of the present application; as shown in FIG. 1, in order to support a multi-modal data processing application, a feature alignment model can be pre-trained, and a multi-modal data processing method is realized through the feature alignment model. In the multi-modal data processing system 100, the terminal 200 (exemplarily shows the terminal 200-1 and the terminal 200-2) is connected to the server 400 through the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two. In addition, the multi-modal data processing system 100 also includes a database 500 for providing data support to the server 400; and in FIG. 1, a case where the database 500 is independent of the server 400 is shown, in addition, the database 500 can also be integrated in the server 400, which is not limited by the embodiment of the present application.

[0048] The terminal 200 is configured to receive rendering data transmitted by the server 400 through the network 300, and display the rendering data, which is source modality sample data and target modality sample data (exemplarily, a graphical interface 210-1 displaying a chair and a graphical interface 210-2 displaying a vehicle). The server 400 is configured to receive rendering data transmitted by the terminal 200 through the network 300, so as to obtain source modality sample data and target modality sample features of the target modality sample data; extract features of the source modality sample data in a target modality feature space by using a first to-be-trained model, to obtain target modality estimated features; then, calculate a feature loss function value between the target modality estimated features and the target modality sample features by using P loss functions; finally, train the first to-be-trained model based on the feature loss function value, to obtain a feature alignment model. After the feature alignment model is trained, the server 400 can also receive an information prediction request transmitted by a terminal. The terminal transmitting the information prediction request can be the same terminal as the terminal transmitting the rendering data, or can be a different terminal. After receiving the information prediction request, the server 400 responds to the information prediction request, obtains source modality data to be aligned; then, performs feature alignment on the data to be aligned to the target modality by using the feature alignment model, to obtain processed aligned features; then, performs information prediction on the processed aligned features by using an information prediction model, to obtain a first information prediction result, the information prediction model being configured to perform information prediction on data features of the target modality; finally, generates new rendering data based on the first information prediction result, and transmits the new rendering data to the terminal 200 through the network 300.

[0049] In some embodiments, the server 400 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the embodiments of the present application.

[0050] FIG. 2 is a structural schematic diagram of a server according to an embodiment of the present application. As shown in FIG. 2, the server 400 includes at least one processor 410, a memory 450, and at least one network interface 420. The various components in the server 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is configured to realize the connection and communication between the components. The bus system 440 includes not only a data bus, but also a power bus, a control bus, and a status signal bus. However, for the purpose of clarity and conciseness, all the buses are marked as the bus system 440 in FIG. 2.

[0051] The processor 410 can be an integrated circuit chip that has a processing capability of signals, such as a general processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general processor can be a microprocessor or any conventional processor, etc. The memory 450 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memories, hard disk drives, optical disk drives, etc. The memory 450 can optionally include one or more storage devices that are physically located away from the processor 410. The memory 450 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0052] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0053] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks; the network communication module 452 is used to communicate with other electronic devices via one or more (wired or wireless) network interfaces 420, and exemplary network interfaces 420 include Bluetooth, wireless compatibility authentication (Wi-Fi), universal serial bus (USB), etc.

[0054] In some embodiments, the multi-modal data processing apparatus provided by the embodiments of the present application can be realized in a software manner, and FIG. 2 shows a feature alignment model training apparatus 455 stored in the memory 450, which can be software in the form of programs and plug-ins, etc., including the following software modules: a data acquisition module 4551, a feature alignment module 4552, a loss calculation module 4553, and a model training module 4554. These modules are logical, and thus can be combined or further split according to the functions implemented. In other embodiments, the multi-modal data processing apparatus (not shown in FIG. 2) stored in the memory 450 can also be software in the form of programs and plug-ins, etc., including the following software modules: a data determination module and a model calling module. The functions of each module will be described below.

[0055] In some embodiments, the feature alignment model training apparatus and the multi-modal data processing apparatus provided by the embodiments of the present application can be implemented in a hardware manner. For example, the feature alignment model training apparatus and the multi-modal data processing apparatus provided by the embodiments of the present application can be a hardware decoding processor programmed to execute the feature alignment model training method and the multi-modal data processing method provided by the embodiments of the present application. For example, the hardware decoding processor can be one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs), or other electronic elements.

[0056] In some embodiments, the terminal or server can implement the feature alignment model training method and the multi-modal data processing method provided by the embodiments of the present application by running various computer-executable instructions or computer programs. For example, the computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. The computer program can be a native program or a software module in an operating system, and the computer program can be a Native Application (APP) that needs to be installed in an operating system to run, such as a game APP or a virtual reality APP. Alternatively, the computer program can be a small program that can be embedded into any APP, i.e., a program that only needs to be downloaded into a browser environment to run. In summary, the above computer-executable instructions can be any form of instructions, and the above computer programs can be any form of application programs, modules, or plug-ins.

[0057] In the following, the feature alignment model training method and the multi-modal data processing method provided by the embodiments of the present application will be described in conjunction with exemplary applications and implementations of electronic devices provided by the embodiments of the present application. In addition, the feature alignment model training method and the multi-modal data processing method provided by the embodiments of the present application can be applied to various data processing scenarios for feature alignment in cloud technology, artificial intelligence, intelligent transportation, games, virtual reality, and vehicle-mounted devices.

[0058] Referring to FIG. 3, FIG. 3 is a flowchart of a feature alignment model training method provided by an embodiment of the present application. In FIG. 3, the execution subject of each step is an electronic device, which can be a server. The steps shown in FIG. 3 will be described in the following.

[0059] Step 101, obtaining source modality sample data and target modality sample feature of target modality sample data.

[0060] In the embodiment of the present application, when the electronic device obtains the feature alignment data set and trains the artificial neural network model for feature alignment using the feature alignment data set, sample data of two modalities can be obtained for the same object, i.e., source modality sample data and target modality sample data, and the feature of the target modality sample data, i.e., the target modality sample feature, can also be obtained. The same object may, for example, be a specified building, recorded data, news information, a virtual object, or a virtual scene, etc.

[0061] In the embodiment of the present application, feature alignment refers to converting data of different modalities (such as images, texts, or point clouds, etc.) into a shared feature space, so that the data from different modalities have comparability and consistency at the feature level, thereby realizing effective fusion and interaction between different modalities of data.

[0062] The features of different modalities, i.e., the feature vectors of different modalities, can include image features, text features, and point cloud features. Image features are features extracted from images, which can reflect the feature vectors of image content (such as objects or scenes, etc.). For example, semantic features of images can be extracted through a convolutional neural network (CNN), which can represent information such as the shape, color, and texture of objects in the image. Text features are features extracted from text descriptions, which can reflect the feature vectors of text semantics. For example, semantic features of texts can be extracted through a natural language processing (NLP) model, which can represent information such as the attributes of objects and the description of scenes in the text description. Point cloud features are features extracted from point cloud data, which can reflect the geometric and semantic features of three-dimensional objects or scenes represented by the point cloud. For example, features of point clouds can be extracted through a point cloud processing network (such as PointNet), which can represent information such as the shape and structure of the point cloud.

[0063] The goal of feature alignment is to align these feature vectors of different modalities into a shared feature space, so that these feature vectors of different modalities have comparability and consistency in the space. For example, for an image describing an object and a corresponding text description, after feature alignment, the image features and text features should be able to match and correspond to each other in the feature space, thereby realizing the semantic association between the image and the text.

[0064] The feature alignment technology can be applied to various different scenarios, such as a multi-modal interaction scenario, a cross-modal retrieval scenario, multi-modal content generation, etc. Each scenario is illustrated below.

[0065] In the multi-modal interaction scenario, feature alignment is the basis for realizing the interaction and fusion of different modal data. For example, in the task of controlling image generation or editing through text, it is necessary to align the text features with the image features, so that the image generation model can accurately generate or edit the image content according to the text description.

[0066] In the cross-modal retrieval scenario, for the cross-modal retrieval task, feature alignment can make the data of different modalities comparable in the feature space, thereby realizing the function of retrieving one kind of modal data based on another kind of modal data. For example, retrieving images or point cloud data related to a text description through the text description, or retrieving a text description related to an image through the image.

[0067] In the multi-modal content generation scenario, for the multi-modal content generation task, feature alignment can provide richer semantic guidance for content generation. For example, in the text-to-image or text-to-three-dimensional model generation task, through the aligned text features and image or point cloud features, content that conforms to user needs and semantic descriptions can be generated.

[0068] For the feature alignment technology, first, feature vectors need to be extracted from data of different modalities. For images, convolutional neural networks can be used to extract features; for text, natural language processing models can be used to extract features; for point clouds, point cloud processing networks can be used to extract features. After the features are extracted, they need to be aligned in a shared feature space through certain methods. For example, through contrastive learning, a feature mapping function can be learned to map features of different modalities to a shared feature space, so that features of the same semantics are closer in the space, and features of different semantics are farther apart.

[0069] The precision and robustness of feature alignment are important indicators for measuring the alignment effect. High-precision alignment can make the data of different modalities more accurately match and correspond in the feature space, thereby improving the effect of multi-modal interaction and fusion. Robustness refers to the stability of the alignment algorithm in the face of different data distributions, noise interference, etc. A robust alignment algorithm can more stably realize feature alignment. Feature alignment is a key link in multi-modal learning and interaction. Feature alignment enables different modal data to be effectively fused and interacted at the feature level, thereby providing stronger technical support for multi-modal applications.

[0070] The source modality sample data and the target modality sample data can be a training sample in a two-tuple form, and the source modality sample data and the target modality sample data are any one of training samples included in the feature alignment data set. At this time, the electronic device obtains the source modality sample data and the target modality sample feature of the target modality sample data, which can be: first, obtaining the source modality sample data and the target modality sample data, and then performing feature extraction on the target modality sample data to obtain the target modality sample feature, so as to obtain the source modality sample data and the target modality sample feature. In addition, the source modality sample data and the target modality sample feature can be a training sample in a two-tuple form, and the training sample in the two-tuple form can be any one of training samples in the feature alignment data set.

[0071] The source modality sample data and the target modality sample data can be sample data describing the same object from different modalities, the source modality sample data is sample data describing the object from the source modality, and the target modality sample data is sample data describing the object from the target modality. The source modality and the target modality are different modalities, and the modality here can be any one of text, image, audio, video and point cloud. Here, the target modality sample feature is a feature of the target modality sample data, and the target modality sample feature is a label for feature alignment of the source modality sample data to the target modality.

[0072] In step 102, a first to-be-trained model is used to extract features of the source modality sample data in the target modality feature space, to obtain target modality estimated features.

[0073] In the embodiment of the present application, the to-be-trained artificial neural network model for feature alignment of data of the source modality to the target modality is the first to-be-trained model. After obtaining the source modality sample data, the electronic device can first use the first to-be-trained model to process the source modality sample data, so as to perform feature alignment of the source modality sample data to the target modality. The obtained feature alignment result is the target modality estimated feature, that is, the target modality estimated feature is the feature corresponding to the source modality sample data in the feature space of the target modality.

[0074] It should be noted that the first to-be-trained model is a to-be-trained artificial neural network model for feature alignment of data of the source modality to the target modality; or in other words, the first to-be-trained model is a to-be-trained artificial neural network model for extracting features of data of the source modality in the feature space of the target modality. The first to-be-trained model can be a constructed original artificial neural network model, or a pre-trained artificial neural network model, etc., and the embodiment of the present application does not limit this.

[0075] In the embodiments of the present application, the first to-be-trained model comprises a source modality sample data input module, a feature extraction module of a target modality feature space, and a feature output module; the source modality sample data input module is configured to obtain input source modality sample data and perform embedding representation on the source modality sample data; the feature extraction module is configured to map the embedding representation result to the feature space of the target modality to obtain target modality estimated features; and the feature output module is configured to output the target modality estimated features.

[0076] Referring to FIG. 4, FIG. 4 is a flowchart of a feature alignment model training method according to an embodiment of the present application. In FIG. 4, the execution subject of each step is an electronic device. As shown in FIG. 4, in the embodiments of the present application, step 102 can be implemented by steps 1021 to 1024; that is, the electronic device uses the first to-be-trained model to extract the features of the source modality sample data in the feature space of the target modality to obtain target modality estimated features, including steps 1021 to 1024, which are described below.

[0077] In step 1021, for M feature channels corresponding to the target modality sample features, the channel feature dimension of each feature channel is obtained.

[0078] In the embodiments of the present application, the electronic device obtains the feature size of the target modality sample features, and performs feature alignment on the source modality sample data to the target modality based on the feature size of the target modality sample features to obtain a feature alignment result with the same feature size as the target modality sample features.

[0079] It should be noted that the feature size of the target modality sample features corresponds to M feature channels, where M > 1 and M is an integer. The feature dimension corresponding to each feature channel is referred to as the channel feature dimension, which can be in the form of a vector or a matrix. In the case of a vector form of the channel feature dimension, the feature size of the target modality sample features can be two-dimensional; in the case of a matrix form of the channel feature dimension, the feature size of the target modality sample features can be three-dimensional.

[0080] Here, the channel feature dimension refers to the dimension of the feature vector corresponding to each feature channel in the feature map. In a convolutional neural network, the dimension of the feature map is usually represented as [height, width, channel number]. Among them, the channel number represents the component in the depth direction of the feature map. Each channel represents a specific feature, such as an edge, texture, or color, etc. For example, a feature map with a dimension of [32, 32, 64] indicates that the height of the feature map is 32, the width is 32, and the channel number is 64. The 64 channels here each have a 32x32 feature matrix, representing the spatial distribution of the features extracted by the channel.

[0081] For different modal sample features, if the target modal sample feature is an image modal sample feature, the channel feature dimension of the image modal sample feature is usually related to the number of convolution kernels. For example, after using 64 convolution kernels to perform convolution operation on an image, the output feature map will have 64 channels, each channel corresponding to a feature extracted by a convolution kernel, which can be an edge, texture or color of the image, etc. If the target modal sample feature is a text modal sample feature, in natural language processing, text features are usually represented by word embeddings or context embeddings. Although the text modal sample feature does not have a clear "channel" concept like the image modal sample feature, the text modal sample feature can be regarded as a feature vector of different dimensions, each dimension representing a certain semantic or grammatical feature of the text. If the target modal sample feature is a point cloud modal sample feature, the point cloud modal sample feature extraction is usually realized by a point cloud processing network, and the channel feature dimension of the point cloud modal sample feature can be represented as the dimension of the feature vector of each point, such as the position, normal or color of the point, etc. Although the channel feature dimension is different in different modalities, the core idea is the same, that is, each channel or dimension represents a specific feature. In the image modal, the channel feature dimension corresponds to the feature extracted by the convolution kernel; in the text modal, the channel feature dimension corresponds to the dimension of the word embedding or context embedding; in the point cloud modal, the channel feature dimension corresponds to the feature vector of each point.

[0082] Step 1022, using the initial feature extraction module of the first to-be-trained model, extracting the features of the source modal sample data in the target modal feature space to obtain initial alignment features.

[0083] In the embodiments of the present application, the first to-be-trained model includes an initial feature extraction module, and the initial feature extraction module is used for initial feature alignment of the source modal sample data to the target modal. The electronic device performs initial feature alignment on the source modal sample data using the initial feature extraction module, and the obtained initial feature alignment result is the initial alignment feature.

[0084] Step 1023, using the scale adjustment module of the first to-be-trained model, extracting the channel estimated feature of the initial alignment feature based on the channel feature dimension on each feature channel of the M feature channels to obtain M channel estimated features.

[0085] In the embodiments of the present application, the first to-be-trained model further includes a scale adjustment module, and the scale adjustment module is used for scale adjustment of the initial alignment feature to obtain a channel estimated feature with the same feature size as the target modal sample feature. The initial feature extraction module and the scale adjustment module are two sub-modules of the feature extraction module in the first to-be-trained model.

[0086] It should be noted that when the source modality sample data is aligned to the target modality, the channel estimation feature is the alignment feature corresponding to each obtained feature channel, and the feature dimension of the channel estimation feature is the channel feature dimension. Here, the M feature channels correspond to the M channel estimation features one by one, that is, each feature channel corresponds to a channel estimation feature.

[0087] In step 1024, the M channel estimation features are determined as target modality estimation features.

[0088] In the embodiment of the present application, the electronic device determines the M channel estimation features as the target modality estimation features, that is, the target modality estimation features include the M channel estimation features, and the target modality estimation features are consistent with the target modality sample features in feature size.

[0089] It can be understood that when the source modality sample data is aligned to the target modality, the feature alignment can be performed based on the feature size of the target modality estimation feature, which can make the target modality estimation feature consistent with the target modality sample feature in feature size, thereby providing feasibility for data processing between the target modality estimation feature and the target modality sample feature, and improving the efficiency and accuracy of data processing between the target modality estimation feature and the target modality sample feature. In addition, since the target modality estimation feature and the target modality sample feature include a plurality of feature channels respectively corresponding to features in feature size, the plurality of feature channels respectively corresponding to the features improve the information amount of the target modality estimation feature, which can improve the accuracy of feature alignment, and further improve the accuracy of model training.

[0090] In step 103, P loss functions are used to calculate the feature loss function value between the target modality estimation feature and the target modality sample feature.

[0091] In the embodiment of the present application, when calculating the loss function value between the target modality estimation feature and the target modality sample feature, the electronic device can use P loss functions to achieve it, P>1, and P is an integer; and the P loss functions include two or three of the following loss functions: error loss function, similarity loss function and mutual exclusion loss function.

[0092] The error loss function represents the absolute difference (or squared difference) between the estimated feature and the true feature, that is, the error loss function is used to measure the difference between the estimated feature (output of the model) and the true feature (target value), and the error loss function quantifies this difference by calculating the absolute difference or squared difference between the two, such as mean absolute error, mean squared error, etc., wherein the mean absolute error can be represented by the following formula (1):

[0093] where yi is a real feature; is an estimated feature; N is the number of samples. MAE measures the average absolute error between the estimated feature and the real feature, and all errors are treated equally.

[0094] The mean square error MSE can be expressed as the following formula (2):

[0095] The mean square error MSE measures the average square error between the estimated feature and the real feature, and the larger error is punished more severely because the square of the error will amplify the larger error.

[0096] The error loss function can be applied to regression tasks and classification tasks, etc., where the regression task is used to predict continuous values such as price prediction, temperature prediction, etc.; although the classification task usually uses cross-entropy loss, but in some cases, the error loss function can also be used, for example, in multi-label classification tasks.

[0097] The similarity loss function represents the similarity between the estimated feature and the real feature, such as cosine similarity, contrastive loss; here, when the error loss function is applied to calculate the loss function value between the target modality estimated feature and the target modality sample feature, the estimated feature is the target modality estimated feature, and the real feature is the target modality sample feature. The similarity loss function quantifies the closeness of the estimated feature and the real feature by calculating the similarity between the two feature vectors. Among them, the cosine similarity Cosine Loss can be expressed as the following formula (3):

[0098] Where f1 is the real feature, f2 is the estimated feature; · represents the dot product, and ||·|| represents the norm of the vector. The cosine similarity loss measures the angle between two vectors, and the smaller the value, the more similar.

[0099] The contrastive loss Contrastive Loss can be expressed as the following formula (4): Contrastive Loss=y·d 2 +(1-y)·max (0,m-d) 2 ) (4);

[0100] Where y is the label (1 for similar and 0 for not similar), d is the Euclidean distance between two feature vectors, and m is a threshold. The contrastive loss pulls similar samples closer and pushes dissimilar samples further apart.

[0101] The similarity loss function can be applied to feature learning, metric learning, image retrieval, and face recognition. Among them, feature learning is used to learn the representation of feature vectors, so that similar samples are closer in the feature space, and dissimilar samples are farther apart; metric learning is used to learn a metric space, so that similar samples are closer in the space; image retrieval is used to measure the similarity between a query image and images in the database, and find the most similar image; face recognition is used to measure whether two face images belong to the same person.

[0102] The mutual exclusivity loss function represents the mutual exclusivity between the estimated feature and the irrelevant feature, and the irrelevant feature refers to a feature that is irrelevant to the true feature, such as a feature that is different from the object described by the true feature. The goal of the mutual exclusivity loss function is to ensure that the estimated feature and the irrelevant feature have clear discrimination, avoiding ambiguity. The mutual exclusivity loss function Mutual Exclusivity Loss can be represented by the following formula (5):

[0103] where C is the number of categories, and are the predicted probabilities of different categories by the model. The mutual exclusivity loss function ensures that the output of the model is mutually exclusive between multiple categories by penalizing the sum of the predicted probabilities of multiple categories being greater than 1.

[0104] The mutual exclusivity loss function can be applied to multi-label classification tasks and multi-object detection tasks. Multi-label classification tasks can ensure that the output of the model has clear discrimination between multiple labels, avoiding multiple labels being predicted as high probability at the same time; multi-object detection tasks can ensure that the model has clear discrimination between multiple objects, avoiding multiple objects being predicted as the same category.

[0105] In the embodiments of the present application, the electronic device adopts two or three of the error loss function, the similarity loss function, and the mutual exclusivity loss function to jointly calculate the loss function value between the target modality estimated feature and the target modality sample feature, and the calculated loss function value is referred to as the feature loss function value.

[0106] It should be noted that the feature loss function value represents the degree of deviation between the target modality estimated feature and the target modality sample feature calculated based on P loss function values; the feature loss function value is determined based on at least two differences between the target modality estimated feature and the target modality sample feature, and is positively correlated with the at least two differences, so that the feature loss function value can be used to represent the feature alignment effect of the first to-be-trained model. The at least two differences between the target modality estimated feature and the target modality sample feature include at least two of the following differences: an absolute difference (or a squared difference) between the target modality estimated feature and the target modality sample feature calculated by using an error loss function, a similarity between the target modality estimated feature and the target modality sample feature calculated by using a similarity loss function, and a mutual exclusivity score between irrelevant features of the target modality estimated feature and the target modality sample feature calculated by using a mutual exclusivity loss function.

[0107] In some embodiments, in step 103, the P loss functions are used to calculate the feature loss function value between the target modality estimated feature and the target modality sample feature, which can first determine a channel loss function value under each feature channel in the M feature channels corresponding to the target modality estimated feature based on the target modality estimated feature and the target modality sample feature; then, based on the M channel loss function values in the M feature channels, the feature loss function value is determined; M is an integer greater than 1.

[0108] Referring to FIG. 5A, FIG. 5A is a flowchart of obtaining a feature loss function value according to an embodiment of the present application, wherein the execution subject of each step in FIG. 5A is an electronic device; as shown in FIG. 5A, in the embodiment of the present application, step 103 can be implemented by steps 1031 to 1032; that is, the electronic device uses P loss functions to calculate the feature loss function value between the target modality estimated feature and the target modality sample feature, which includes steps 1031 to 1032, which will be described below.

[0109] In step 1031, a channel loss function value under each feature channel in the M feature channels corresponding to the target modality estimated feature is determined based on the target modality estimated feature and the target modality sample feature.

[0110] In the embodiment of the present application, the M feature channels corresponding to the target modality estimation feature can also be the M feature channels corresponding to the target modality sample feature. The electronic device can perform the following processing for each feature channel among the M feature channels corresponding to the target modality estimation feature and the target modality sample feature: first, obtain the channel estimation feature under the feature channel from the target modality estimation feature; then, obtain the channel sample feature under the feature channel from the target modality sample feature; then, calculate the sub-channel loss function value between the channel estimation feature and the channel sample feature by using each loss function to obtain P sub-channel loss function values; and finally, determine the channel loss function value under the feature channel based on the P sub-channel loss function values.

[0111] Here, when obtaining the channel estimation feature corresponding to the feature channel from the target modality estimation feature, since the target modality estimation feature corresponds to the M feature channels, the electronic device can obtain the corresponding feature from the target modality estimation feature for each feature channel among the M feature channels. The feature obtained from the target modality estimation feature and corresponding to the feature channel can be referred to as the channel estimation feature.

[0112] When obtaining the channel sample feature corresponding to the feature channel from the target modality sample feature, since the target modality sample feature corresponds to the M feature channels, the electronic device can obtain the corresponding feature from the target modality sample feature for each feature channel among the M feature channels. The feature obtained from the target modality sample feature and corresponding to the feature channel can be referred to as the channel sample feature.

[0113] It should be noted that the step of obtaining the channel estimation feature corresponding to the feature channel from the target modality estimation feature and the step of obtaining the channel sample feature corresponding to the feature channel from the target modality sample feature do not have a specific execution sequence.

[0114] In the embodiment of the present application, the electronic device can calculate the sub-channel loss function value between the channel estimation feature and the channel sample feature by using each loss function for each loss function among the P loss functions, thereby obtaining P sub-channel loss function values. The electronic device calculates the loss function value between the channel estimation feature and the channel sample feature by using each loss function, and the calculated loss function value is referred to as the sub-channel loss function value. It should be noted that the loss function used to calculate the sub-channel loss function value is one of the error loss, the similarity loss, and the mutual exclusion loss.

[0115] After obtaining the P sub-channel loss function values, the electronic device determines the channel loss function value under the feature channel based on the P sub-channel loss function values.

[0116] Step 1032, determining the feature loss function value based on the M channel loss function values under the M feature channels; M is an integer greater than 1.

[0117] In the embodiment of the present application, when the loss function is an error loss function, referring to FIG. 5B, in step 1031, when calculating the sub-channel loss function value between the channel estimated feature and the channel sample feature by using each loss function, it can be implemented by step 10311 and step 10312, wherein the error loss is, for example, mean square error loss, mean deviation error loss, root mean square error loss or mean absolute error loss, etc. That is, the electronic device calculates the sub-channel loss function value between the channel estimated feature and the channel sample feature by using the loss function, including step 10311 and step 10312, which will be described below.

[0118] Step 10311, calculating the channel feature difference between the channel estimated feature and the channel sample feature.

[0119] In the embodiment of the present application, the electronic device can calculate the channel feature difference between the channel estimated feature and the channel sample feature by using the subtracting method, or calculate the channel feature difference between the channel estimated feature and the channel sample feature by using the ratio calculation method, etc., which is not limited in the embodiment of the present application. Wherein, the channel feature difference represents the difference between the channel estimated feature and the channel sample feature.

[0120] Step 10312, determining the sub-channel loss function value based on the channel feature difference.

[0121] In the embodiment of the present application, the electronic device can directly determine the channel feature difference as the sub-channel loss function value, or determine the processing result of the channel feature difference as the sub-channel loss function value, which is not limited in the embodiment of the present application. Here, the electronic device can obtain the processing result of the channel feature difference by calculating the absolute value or square value of the channel feature difference.

[0122] It can be understood that, by using the error loss function to calculate the sub-channel loss function value between the channel estimated feature and the channel sample feature, the process of calculating the loss function value of the feature channel dimension from the numerical value is realized, which can improve the consistency of the channel estimated feature and the channel sample feature from the numerical value.

[0123] In the embodiment of the present application, when the loss function is a similarity loss function (for example, a cosine similarity loss function), referring to FIG. 5C, in step 1031, when each loss function is used to calculate the sub-channel loss function value between the channel estimated feature and the channel sample feature, steps 10313 to 10315 can be implemented; that is, the electronic device uses the loss function to calculate the sub-channel loss function value between the channel estimated feature and the channel sample feature, which includes steps 10313 to 10315, and each step is described below.

[0124] Step 10313, fusing the channel estimated feature and the channel sample feature to obtain a first fused feature.

[0125] It should be noted that the electronic device can fuse the channel estimated feature and the channel sample feature in a multiplication manner, or in a splicing manner, or in a weighted summation manner, and the present application does not limit this. Here, the first fused feature is the fusion result of the channel estimated feature and the channel sample feature.

[0126] Step 10314, fusing the absolute value of the channel estimated feature and the absolute value of the channel sample feature to obtain a second fused feature.

[0127] It should be noted that the manner in which the electronic device fuses the absolute value of the channel estimated feature and the absolute value of the channel sample feature is similar to the manner in which the electronic device fuses the channel estimated feature and the channel sample feature, and the present application does not repeat the description here. Here, the second fused feature refers to the fusion result of the absolute value of the channel estimated feature and the absolute value of the channel sample feature.

[0128] Step 10315, obtaining a sub-channel loss function value that is positively correlated with the first fused feature and negatively correlated with the second fused feature.

[0129] It should be noted that the sub-channel loss function value obtained by the electronic device is positively correlated with the first fused feature and negatively correlated with the second fused feature. Here, the electronic device can calculate the ratio of the first fused feature as the numerator and the second fused feature as the denominator as the sub-channel loss function value.

[0130] It can be understood that using the similarity loss function to calculate the sub-channel loss function value between the channel estimated feature and the channel sample feature realizes the process of calculating the loss function value of the feature channel dimension from the similarity, which can improve the consistency of the channel estimated feature and the channel sample feature from the similarity.

[0131] In the embodiment of the present application, when the loss function is the mutually exclusive loss function, referring to FIG. 5D, in step 1031, when each loss function is used to calculate the sub-channel loss function value between the channel predicted feature and the channel sample feature, steps 10316 to 10318 can be implemented; that is, the electronic device uses the loss function to calculate the sub-channel loss function value between the channel predicted feature and the channel sample feature, which includes steps 10316 to 10318, and each step is described below.

[0132] In the embodiment of the present application, the electronic device performs the following processing (steps 10316 to 10318) on the kth channel predicted feature corresponding to the kth to-be-processed object and the kth channel sample feature corresponding thereto based on the K channel predicted features corresponding to the K to-be-processed objects and the K channel sample features corresponding to the K to-be-processed objects.

[0133] It should be noted that 1≤k≤K, K>1, and K is an integer; and the kth channel predicted feature and the kth channel sample feature correspond to the same to-be-processed object (i.e., the kth to-be-processed object).

[0134] Step 10316, obtaining a first total fusion result after the kth channel predicted feature is fused with K channel sample features respectively.

[0135] In the embodiment of the present application, the electronic device fuses each channel sample feature in the fusion of the kth channel predicted feature and the K channel sample features, and the fusion manner is the same as that of obtaining the first fusion feature; thus, the electronic device can obtain the fusion results of the K kth channel predicted features and channel sample features, and the first total fusion result can be obtained by accumulating the K fusion results.

[0136] Step 10317, obtaining a second total fusion result after the kth channel sample feature is fused with K channel predicted features respectively.

[0137] In the embodiment of the present application, the electronic device fuses each channel predicted feature in the fusion of the kth channel sample feature and the K channel predicted features, and the fusion manner is the same as that of obtaining the first fusion feature; thus, the electronic device can obtain the fusion results of the K kth channel sample features and channel predicted features, and the second total fusion result can be obtained by accumulating the K fusion results.

[0138] Step 10318, determining the sub-channel loss function value by combining the kth first fusion feature, the first total fusion result, and the second total fusion result.

[0139] It should be noted that the electronic device combines the kth first fusion feature, the first total fusion result and the second total fusion result to calculate the sub-channel loss function value.

[0140] In the embodiment of the application, the electronic device combines the kth first fusion feature, the first total fusion result and the second total fusion result to determine the sub-channel loss function value, including: the electronic device obtains a first mutual exclusion loss value positively correlated with the kth first fusion feature and negatively correlated with the first total fusion result; and obtains a second mutual exclusion loss value positively correlated with the kth first fusion feature and negatively correlated with the second total fusion result; finally, combines the first mutual exclusion loss value and the second mutual exclusion loss value to determine the sub-channel loss function value.

[0141] It should be noted that the first mutual exclusion loss value is the mutual exclusivity of each sample data in the data set determined from the channel prediction feature dimension; the first mutual exclusion loss value can be obtained by calculating the ratio of the kth first fusion feature as the numerator and the first total fusion result as the denominator. The second mutual exclusion loss value is the mutual exclusivity of each sample data in the data set determined from the channel sample feature dimension; the second mutual exclusion loss value can be obtained by calculating the ratio of the kth first fusion feature as the numerator and the second total fusion result as the denominator. Here, the electronic device can add the opposite of half of the first mutual exclusion loss value to the opposite of half of the second mutual exclusion loss value to realize the combination of the first mutual exclusion loss value and the second mutual exclusion loss value to obtain the sub-channel loss function value.

[0142] It can be understood that the mutual exclusion loss function is used to calculate the sub-channel loss function value between the channel prediction feature and the channel sample feature, which realizes the calculation process of obtaining the loss function value of the feature channel dimension from the mutual exclusivity of the sample set, and can improve the consistency of the channel prediction feature and the channel sample feature from the mutual exclusivity of the sample set.

[0143] In the embodiment of the application, the channel loss function value can be obtained based on the P sub-channel loss function values corresponding to the P loss functions, and the feature loss function value can be obtained based on the M channel loss function values corresponding to the M feature channel loss functions.

[0144] In the embodiment of the present application, the electronic device can calculate a sub-channel loss function value for each loss function, so that P sub-channel loss function values can be obtained for P loss functions. Here, the electronic device can integrate the P sub-channel loss function values into a channel loss function value, which can be the cumulative sum of the P sub-channel loss function values, or the weighted sum of the P sub-channel loss function values, or the statistical result (such as the average, mode, and extreme value) of the P sub-channel loss function values, or a combination of the above, and the present application does not limit the embodiment. In addition, the electronic device can calculate a channel loss function value for each feature channel, so that M channel loss function values can be obtained for M loss functions. Here, the electronic device can integrate the M channel loss function values into a feature loss function value, which can be the cumulative sum of the M channel loss function values, or the weighted sum of the M channel loss function values, or the statistical result (such as the average, mode, and extreme value) of the M channel loss function values, or a combination of the above, and the present application does not limit the embodiment.

[0145] It can be understood that using P loss functions to calculate the loss function value between the target modality estimated feature and the target modality sample feature in each feature channel of the M feature channels realizes the processing of calculating the loss function value in the feature channel dimension, improves the accuracy of the loss function value, and further improves the accuracy of the feature loss function value.

[0146] Referring to FIG. 6, FIG. 6 is a flowchart of a feature alignment model training method according to an embodiment of the present application, wherein the execution subject of each step in FIG. 6 is an electronic device; as shown in FIG. 6, in the embodiment of the present application, step 103 can be implemented by steps 1031B to 1033B; that is, the electronic device uses P loss functions to calculate the feature loss function value between the target modality estimated feature and the target modality sample feature, including steps 1031B to 1033B, which will be described below.

[0147] Step 1031B, performing J times of downsampling on the target modality estimated feature and the target modality sample feature respectively to obtain a downsampled alignment feature and a downsampled sample feature corresponding to each time of downsampling.

[0148] Here, the target modality estimated feature can be downsampled J times to obtain a downsampled alignment feature after each time of downsampling; and the target modality sample feature can be downsampled J times to obtain a downsampled sample feature after each time of downsampling.

[0149] It should be noted that the electronic device has the same size of the target modality estimated feature and the target modality sample feature in each downsampling, and thus the size of the downsampling alignment feature and the downsampling sample feature obtained in each downsampling is the same. In addition, the downsampling alignment feature is the target modality estimated feature after downsampling, the downsampling sample feature is the downsampling sample feature after downsampling, and J is a positive integer.

[0150] In step 1032B, the electronic device calculates the downsampling loss function value between the downsampling alignment feature and the downsampling sample feature after each downsampling, and obtains J downsampling loss function values corresponding to J times of downsampling.

[0151] In the embodiment of the present application, the electronic device calculates the corresponding loss function value by using P loss functions for the downsampling alignment feature and the downsampling sample feature obtained in each downsampling, and the calculated loss function value is referred to as a downsampling loss function value. The downsampling loss function value represents the difference between the downsampling alignment feature and the downsampling sample feature in the feature size after downsampling, and is used to measure the difference between the target modality estimated feature and the target modality sample feature. Since one downsampling loss function value can be calculated in each downsampling, J downsampling loss function values can be calculated for J times of downsampling.

[0152] It should be noted that the process of calculating the downsampling loss function value between the downsampling alignment feature and the downsampling sample feature by using P loss functions by the electronic device is similar to the process described above for calculating the feature loss function value, and the embodiment of the present application will not be repeated here. That is, the electronic device can calculate the loss function value between the downsampling alignment feature and the downsampling sample feature by using P loss functions in each feature channel of the M feature channels, so as to obtain the downsampling loss function value based on the obtained M loss function values.

[0153] In step 1033B, the electronic device determines the feature loss function value based on the J downsampling loss function values.

[0154] In the embodiment of the present application, the electronic device can obtain the feature loss function value by integrating the J downsampling loss function values. The integration method can be accumulation, calculation of mean value, determination of extreme value, weighted summation, etc., which is not limited in the embodiment of the present application.

[0155] Continuing to refer to FIG. 6, in the embodiment of the present application, step 1034B is further included before step 1033B. That is, before the electronic device determines the feature loss function value based on the J downsampling loss function values, the multi-modal data processing method further includes step 1034B, which will be described below.

[0156] Step 1034B, calculate the original scale loss function value between the target modality estimated feature and the target modality sample feature by using P loss functions.

[0157] It should be noted that the process of the electronic device calculating the original scale loss function value between the target modality estimated feature and the target modality sample feature by using P loss functions is the process of obtaining M channel loss function values described in the process of calculating the feature loss function value. The original scale loss function value is obtained by integrating the M channel loss function values. Therefore, the electronic device obtains the feature loss function value based on the M channel loss function values, which can be directly using the original scale loss function value as the feature loss function value, or integrating the original scale loss function value and the J down-sampling loss function values into the feature loss function value.

[0158] Correspondingly, continuing to refer to FIG. 6, in the embodiment of the present application, step 1033B can also be implemented by step 1033B1; that is, the electronic device obtains the feature loss function value based on the J down-sampling loss function values, including step 1033B1, which will be described below.

[0159] Step 1033B1, determine the feature loss function value by combining the J down-sampling loss function values and the original scale loss function value.

[0160] In the embodiment of the present application, when the electronic device obtains the feature loss function value by integrating the J down-sampling loss function values, it can integrate the J down-sampling loss function values and the original scale loss function value into the feature loss function value, or directly integrate the J down-sampling loss function values into the feature loss function value, which is not limited in the embodiment of the present application.

[0161] It can be understood that by down-sampling the target modality estimated feature and the target modality sample feature, and then determining the feature loss function value between the target modality estimated feature and the target modality sample feature based on the down-sampling loss function value between the down-sampling aligned feature and the down-sampling sample feature, the consistency of the target modality estimated feature and the target modality sample feature can be improved from different feature scales, thereby further improving the accuracy of the feature loss function value.

[0162] Step 104, train the first to-be-trained model based on the feature loss function value to obtain a feature alignment model.

[0163] In the embodiment of the present application, the electronic device performs back propagation in the first to-be-trained model based on the feature loss function value to adjust the model parameters in the first to-be-trained model, so as to realize the training of the first to-be-trained model; in addition, the training of the first to-be-trained model can be iteratively performed, and when the iterative training ends, the first to-be-trained model trained in the current iteration is the feature alignment model. Here, when the electronic device determines that the iterative training meets the training end condition, it is determined that the iterative training ends; otherwise, the iterative training is continued.

[0164] It should be noted that the feature alignment model is used to align the features of the to-be-aligned data of the source modality to the target modality. The training end condition can be that the accuracy index threshold is reached, or the iteration number threshold is reached, or the iteration time threshold is reached, or the above combinations, and the present application does not limit the same.

[0165] In the embodiment of the present application, when the feature alignment model is used to align the features of the to-be-aligned data of the source modality, and the information prediction model is used to predict the information of the obtained to-be-processed alignment features, the electronic device trains the first to-be-trained model based on the feature loss function value in step 104 to obtain the feature alignment model, which can be realized in the following way: the electronic device first uses the information prediction model to predict the information of the target modality estimated features to obtain a second information prediction result; then uses the information prediction model to predict the information prediction label of the target modality sample features; then, the difference between the second information prediction result and the information prediction label is combined to calculate the information prediction loss value; finally, the first to-be-trained model is trained based on the feature loss function value and the information prediction loss value to obtain the feature alignment model.

[0166] It can be understood that when the first to-be-trained model is trained, the first to-be-trained model is first used to align the features of the source modality sample data to obtain the target modality estimated features; then P loss functions are used to calculate the feature loss function value between the target modality estimated features and the target modality sample features of the target modality sample data; finally, the first to-be-trained model is trained based on the feature loss function value to obtain the feature alignment model. Since the calculated feature loss function value is calculated by using P loss functions in the training process of the first to-be-trained model, the diversity of the loss function used to calculate the loss function value is improved, and the use of multiple loss functions to calculate the loss function value can improve the accuracy of the feature loss function value. In addition, the P loss functions include two or three of the error loss function, the similarity loss function and the mutual exclusion loss function, so that the accuracy of the feature loss function value can be improved from at least two dimensions of the value, the similarity and the mutual exclusion, and then when the model is trained based on the accurate feature loss function value, the accuracy of the model training can be improved.

[0167] In some embodiments, after the first to-be-trained model is trained based on the feature loss function value to obtain the feature alignment model, the feature alignment model can also be used to implement multi-modal data processing. The multi-modal data processing method of the present application embodiment using the feature alignment model to implement multi-modal data processing is described below. Referring to FIG. 7, FIG. 7 is a flowchart of a multi-modal data processing method according to an embodiment of the present application. In FIG. 7, the execution subject of each step is an electronic device, which can be the same electronic device as the electronic device used to implement the feature alignment model training method. As shown in FIG. 7, in the present application embodiment, after step 104, steps 105 to 107 are further included; that is, after the first to-be-trained model is trained based on the feature loss function value to obtain the feature alignment model, the multi-modal data processing method further includes steps 105 to 107, which are described below.

[0168] Step 105: In response to an information prediction request, obtain to-be-aligned data of a source modality.

[0169] In the present application embodiment, when information prediction based on to-be-aligned data of a source modality is requested, for example, when virtual object generation is performed based on text type data, or when entity segmentation is performed based on point cloud type data, the electronic device receives an information prediction request. At this time, the electronic device can obtain data on which information prediction requested by the information prediction request is based, that is, to-be-aligned data of a source modality, in response to the information prediction request.

[0170] Step 106: Use the feature alignment model to extract features of the to-be-aligned data in the target modality feature space to obtain to-be-processed aligned features.

[0171] The feature alignment model here is the feature alignment model trained by the feature alignment model training method described above.

[0172] It should be noted that since the data features used for information prediction are data features of a target modality, the electronic device uses the feature alignment model to extract features of the to-be-aligned data in the target modality feature space to perform feature alignment of the to-be-aligned data to the target modality, and the obtained feature alignment result is the to-be-processed aligned features. Here, the to-be-processed aligned features are data features corresponding to the target modality.

[0173] Step 107: Use an information prediction model to perform information prediction on the to-be-processed aligned features to obtain a first information prediction result.

[0174] In the embodiments of the present application, an information prediction model is deployed in the electronic device or the electronic device can call the information prediction model from other devices, wherein the information prediction model is used to perform information prediction on data features of a target modality, so that the electronic device performs information prediction on the to-be-processed alignment features by using the information prediction model, and the obtained information prediction result is the first information prediction result.

[0175] It should be noted that the information prediction includes virtual object generation, entity segmentation, virtual reality enhancement, virtual reality detection, and information query, etc. The virtual object generation refers to generating a virtual object based on data features of a target modality; the entity segmentation refers to segmenting an entity based on data features of a target modality; the virtual reality enhancement refers to virtually enhancing real information based on data features of a target modality; the virtual reality detection refers to determining an object to be virtually enhanced based on data features of a target modality; and the information query refers to querying associated information based on data features of a target modality.

[0176] It can be understood that, since the feature alignment model can accurately realize the feature alignment of the data of the source modality to the target modality, the accuracy of the to-be-processed alignment features is improved, and thus the accuracy of the information prediction can be improved when the information prediction is performed based on the to-be-processed alignment features.

[0177] In the embodiments of the present application, the process of determining the target modality and obtaining the information prediction model based on the target modality is further included before step 107; that is, before the electronic device performs information prediction on the to-be-processed alignment features by using the information prediction model to obtain the first information prediction result, the multi-modal data processing method further includes: the electronic device collects an information prediction data set of a second to-be-trained model; and in the information prediction data set, the sample data amount of the information prediction data corresponding to each of the Q data modalities is counted; so as to determine the data modality with the largest sample data amount as the target modality; then, the target data set corresponding to the target modality is obtained from the information prediction data set; finally, the second to-be-trained model is trained based on the target data set to obtain the information prediction model.

[0178] It should be noted that the second to-be-trained model is an artificial neural network model to be trained and used to perform information prediction on data features of a target modality, Q>1, and Q is an integer, and the Q data modalities include one or more of the following: text, image, audio, video, and point cloud. In the embodiments of the present application, the data modality can be referred to as modality. In addition, the second to-be-trained model includes a sample data input module, an information prediction module, and a prediction output module; wherein the sample data input module is used to obtain input information and perform embedding representation on the information; the information prediction module is used to map the embedding representation result to a specified feature space and perform information prediction based on the mapped features; and the prediction output module is used to output the information prediction result.

[0179] In the embodiment of the present application, the electronic device aligns the to-be-aligned data to the target modality by using the feature alignment model to obtain the to-be-predicted alignment feature, and then the multi-modal data processing method further comprises: the electronic device extracts features of the to-be-processed data of the target modality to obtain to-be-processed data features; and combines the to-be-processed alignment features and the to-be-processed data features to process the to-be-aligned data and the to-be-processed data.

[0180] It should be noted that the electronic device combines at least one to-be-processed data feature corresponding to at least one to-be-processed data and at least one to-be-processed alignment feature corresponding to at least one to-be-aligned data to process at least one to-be-processed data and at least one to-be-aligned data. There is an association between the at least one to-be-processed data and the at least one to-be-aligned data, such as the at least one to-be-processed data and the at least one to-be-aligned data being graph nodes of the same heterogeneous graph.

[0181] It can be understood that the integration of heterogeneous data can be achieved by feature alignment, thereby improving the processing efficiency of information, and the amount of information obtained from the data can be improved by feature alignment, thereby improving the accuracy of data processing.

[0182] In the following, an exemplary application of the embodiment of the present application in an actual application scenario will be described. The exemplary application describes the process of aligning the features of data in text and point cloud modalities to image. It should be noted that the multi-modal data processing method provided by the embodiment of the present application is applicable to the process of aligning the features of data in any source modality to the target modality, and here, the process of aligning the features of data in text and point cloud modalities to image is taken as an example for description.

[0183] Referring to FIG. 8, FIG. 8 is a schematic diagram of training an exemplary feature alignment model provided by the embodiment of the present application, as shown in FIG. 8, the input information 8-1 includes image samples 8-11 (i.e. the above-mentioned target modality sample data), text samples 8-12 (i.e. text modality sample data, corresponding to the above-mentioned source modality sample data) and point cloud samples 8-13 (i.e. point cloud modality sample data, corresponding to the above-mentioned source modality sample data). Here, image is taken as the target modality, and text and point cloud are taken as the source modality for description. First, the image encoder 8-21 is used to extract features of the image samples 8-11 to obtain image features 8-311; the text encoder 8-22 (i.e. the above-mentioned first to-be-trained model) is used to align the features of the text samples 8-12 to the image to obtain text alignment features 8-321 (i.e. the above-mentioned initial alignment features); then, the point cloud encoder 8-23 (i.e. the above-mentioned first to-be-trained model) is used to align the features of the point cloud samples 8-13 to the image to obtain point cloud alignment features 8-331 (denoted as The feature size is a specified feature size, and the point cloud alignment feature is the target modality prediction feature described above. Then, the size of the image feature 8-311 is adjusted to the specified feature size by using the size adjustment network 8-41 to obtain the image feature 8-312 (denoted as The text alignment feature is the target modality sample feature described above; and the size of the text alignment feature 8-321 is adjusted to the specified feature size by using the size adjustment network 8-42 to obtain the text alignment feature 8-322 (denoted as The point cloud alignment feature is the target modality prediction feature described above. Then, the size of the image feature 8-311 is adjusted to the specified feature size by using the size adjustment network 8-41 to obtain the image feature 8-312 (denoted as ) and J+1 scales of the text alignment feature 8-323 (denoted as ). Then, for each scale j (j ∈ [0, J]) of the J+1 scales, the channel consistency loss 8-71 between the point cloud alignment feature 8-332 and the image feature 8-313 is calculated by using the channel consistency module 8-6 (denoted as The point cloud alignment feature is the target modality prediction feature described above. Then, the size of the image feature 8-311 is adjusted to the specified feature size by using the size adjustment network 8-41 to obtain the image feature 8-312 (denoted as The point cloud alignment feature is the target modality prediction feature described above. Then, the size of the image feature 8-311 is adjusted to the specified feature size by using the size adjustment network 8-41 to obtain the image feature 8-312 (denoted as

[0184] The feature extraction process, the size adjustment process, the calculation process of the channel consistency loss value, and the calculation process of the multi-scale consistency loss value are described below.

[0185] In the feature extraction of the input information, the image encoder 8-21 and the text encoder 8-22 used can be a backbone network, for example, the backbone network can be a transformer network or a residual network (Resnet), etc.; and the point cloud encoder 8-23 used is any network for point cloud feature extraction, for example, the point cloud encoder 8-23 used can be a point cloud-based three-dimensional target detection network (Pointnet) or a point cloud-based deep network (Pointbert), etc.

[0186] In the size adjustment of the feature, the size adjustment network can be implemented by using a deep learning network for image semantic segmentation (RefineNet).

[0187] In summary, the processes of obtaining the image feature 8-312, the text alignment feature 8-322 and the point cloud alignment feature 8-331 can be described by formulas (6) to (8) as follows.

[0188] wherein Refine1 represents the size adjustment processing corresponding to the size adjustment network 8-41, and Refine2 represents the size adjustment processing corresponding to the size adjustment network 8-42; E I represents the feature extraction processing corresponding to the image encoder 8-21, E Te represents the feature alignment processing corresponding to the text encoder 8-22, E P represents the feature alignment processing corresponding to the point cloud encoder 8-23; I is the image sample 8-11, Te is the text sample 8-12, and P is the point cloud sample 8-13.

[0189] In calculating the channel consistency loss value, taking the point cloud alignment feature and the image feature as an example for illustration.

[0190] For the point cloud alignment feature corresponding to the scale j and the image feature Since the feature sizes are both CxN, C represents the number of feature channels, and N represents the information dimension of each feature channel; and when j = 0, C = M. Therefore, the point cloud alignment feature can be represented by the following formula (9), and the image feature can be represented by the following formula (10) as follows.

[0191] It should be noted that on the c-th (c ∈ [1, C]) channel, various loss values between the point cloud alignment feature and the image feature are calculated, including the average error loss function value Loss L1 , the cosine similarity loss function value Loss cosine and the mutual exclusion loss function value (all referred to as sub-channel loss function values), as shown in formulas (11) to (13).

[0192] wherein the input information 8-1 is the k-th (k ∈ [1, K]) sample in the batch sample, K is the total number of samples in the batch sample, and τ is a hyperparameter; in addition, b ∈ [1, K] represents the b-th sample in the batch sample. In addition, represents the channel feature difference described above, represents the first fusion feature described above, represents the second fusion feature described above, denotes the second mutual exclusive loss value described above, denotes the first mutual exclusive loss value described above.

[0193] Based on formula (11) to formula (13), the consistency loss value on the c-th feature channel corresponding to the scale j can be obtained (referred to as the channel loss function value), as shown in formula (14).

[0194] Based on formula (14), the channel consistency loss value of scale j can be obtained as shown in formula (15).

[0195] Based on formula (15), the multi-scale consistency loss value L P (referred to as the feature loss function value), as shown in formula (16).

[0196] In FIG. 8, the point cloud encoder 8-23 is trained based on J+1 channel consistency losses 8-71, that is, based on the multi-scale consistency loss value L P The point cloud encoder 8-23 is trained.

[0197] Similarly, based on the calculation process of the multi-scale consistency loss value L P , the multi-scale consistency loss value L Te (referred to as the feature loss function value) between the text alignment features and the image features can be calculated, and the text encoder 8-22 and the size adjustment network 8-42 are trained based on the multi-scale consistency loss value L Te .

[0198] FIG. 9 is an exemplary feature alignment training schematic provided by an embodiment of the present application. As shown in FIG. 9, the triplet 9-1 describing the same object includes text data 9-11, image information 9-12, and point cloud data 9-13. Here, the point cloud is the source modality, and when the image is the target modality, the point cloud encoder 9-21 is used to perform feature alignment of the point cloud data 9-13 to the image, so as to train the point cloud encoder 9-21 based on the difference between the point cloud alignment features 9-31 output by the point cloud encoder 9-21 and the image features 9-32 output by the image encoder 9-22 for feature extraction of the image information 9-12. When the text is the target modality, the point cloud encoder 9-21 is used to perform feature alignment of the point cloud data 9-13 to the text, so as to train the point cloud encoder 9-21 based on the difference between the point cloud alignment features 9-31 output by the point cloud encoder 9-21 and the text features 9-33 output by the text encoder 9-23 for feature extraction of the text data 9-11.

[0199] FIG. 10 is another exemplary feature alignment schematic diagram provided by the embodiment of the present application. As shown in FIG. 10, in order to improve the sample quantity, for the same object 10-1, the image information set 10-21 rendered under different viewing angles can be used for training of the point cloud encoder 10-31. Here, the graph-to-text model 10-32 can be used to obtain the text data corresponding to each image information in the image information set 10-21, to obtain the text data set 10-22. When the point cloud is the source modality and the image is the target modality, the point cloud encoder 10-31 is used to perform feature alignment of the point cloud data 10-23 to the image, so as to train the point cloud encoder 10-31 based on the difference between the point cloud alignment feature 10-41 output by the point cloud encoder 10-31 and the image feature 10-42 output by the image encoder 10-34 for feature extraction of the image information in the image information set 10-21. When the text is the target modality, the point cloud encoder 10-31 is used to perform feature alignment of the point cloud data 10-23 to the text, so as to train the point cloud encoder 10-31 based on the difference between the point cloud alignment feature 10-41 output by the point cloud encoder 10-31 and the text feature 10-43 output by the text encoder 10-33 for feature extraction of the text data in the text data set 10-22.

[0200] It should be noted that after the feature alignment of the data of the source modality to the target modality is completed by using the multi-modal data processing method provided by the embodiment of the present application, the generated model can be used to generate information based on the feature after the feature alignment. Exemplarily, referring to FIG. 11, FIG. 11 is an exemplary information generation result schematic diagram provided by the embodiment of the present application. The information 11-1 is a chair generated based on the feature after the feature alignment. The information 11-2 is a vehicle generated based on the feature after the feature alignment. The information 11-3 is a virtual role generated based on the feature after the feature alignment.

[0201] It should be further noted that after the feature alignment of the data of the source modality to the target modality is completed by using the multi-modal data processing method provided by the embodiment of the present application, the feature after the feature alignment can also be used to perform data processing on the graph nodes in the heterogeneous graph. Exemplarily, referring to FIG. 12, FIG. 12 is an exemplary feature alignment application schematic diagram provided by the embodiment of the present application. The graph structure 12-1 is a heterogeneous graph, including graph nodes of different modalities, exemplarily including the image modality graph node 12-11 and the point cloud modality graph node 12-12. When the graph structure 12-1 is processed, the feature alignment of the graph node 12-11 to the point cloud modality can be performed, and then the graph structure 12-1 is processed based on the point cloud feature after the feature alignment and the point cloud feature of the graph node 12-12.

[0202] It can be understood that the embodiments of the present application set the size of the aligned features to CxN, which can reduce the information loss caused by feature compression and improve the accuracy of feature alignment. In addition, the embodiments of the present application can flexibly determine the feature alignment direction, thereby realizing feature alignment of various source modalities to various target modalities and improving the applicability of feature alignment. Furthermore, the embodiments of the present application use multiple scales to calculate the loss function value, and in each channel of each scale, multiple loss function values are used to calculate the loss function value, thereby improving the constraint of the loss function value, and thus the training effect of the feature alignment model can be improved.

[0203] The following continues to illustrate an exemplary structure of the feature alignment model training apparatus 455 implemented as a software module. In some embodiments, as shown in FIG. 2, the software module in the feature alignment model training apparatus 455 stored in the memory 450 can include: a data acquisition module 4551 configured to acquire source modality sample data and target modality sample features of target modality sample data of the target modality sample data; the source modality sample data and the target modality sample data are sample data describing the same object from different modalities; a feature alignment module 4552 configured to extract features of the source modality sample data in the target modality feature space using a first to-be-trained model to obtain target modality estimated features; a loss calculation module 4553 configured to calculate a feature loss function value between the target modality estimated features and the target modality sample features using P loss functions; the P loss functions include two or three of the following loss functions: error loss function, similarity loss function and mutual exclusion loss function, and P is an integer greater than 1; a model training module 4554 configured to train the first to-be-trained model based on the feature loss function value to obtain a feature alignment model.

[0204] In some embodiments, the first to-be-trained model is a to-be-trained artificial neural network model; and the first to-be-trained model is used to extract features of source modality data in the target modality feature space.

[0205] In some embodiments, the loss calculation module 4553 is further configured to, in the M feature channels corresponding to the target modality estimated features, determine a channel loss function value under each feature channel based on the target modality estimated features and the target modality sample features using the P loss functions; and determine the feature loss function value based on the M channel loss function values in the M feature channels; and M is an integer greater than 1.

[0206] In some embodiments, the loss calculation module 4553 is further configured to, for each feature channel, obtain a channel predicted feature under the feature channel from the target modality predicted feature, obtain a channel sample feature under the feature channel from the target modality sample feature, calculate a sub-channel loss function value between the channel predicted feature and the channel sample feature using each loss function, and obtain P sub-channel loss function values, and determine a channel loss function value under the feature channel based on the P sub-channel loss function values.

[0207] In some embodiments, the loss calculation module 4553 is further configured to, in response to the loss function being an error loss function, calculate a channel feature difference between the channel predicted feature and the channel sample feature, and determine the sub-channel loss function value based on the channel feature difference.

[0208] In some embodiments, the loss calculation module 4553 is further configured to, in response to the loss function being a similarity loss function, fuse the channel predicted feature and the channel sample feature to obtain a first fused feature, fuse an absolute value of the channel predicted feature and an absolute value of the channel sample feature to obtain a second fused feature, and obtain the sub-channel loss function value which is positively correlated with the first fused feature and negatively correlated with the second fused feature.

[0209] In some embodiments, the loss calculation module 4553 is further configured to, in response to the loss function being an exclusive loss function, based on K channel predicted features corresponding to K to-be-processed objects and K channel sample features corresponding to the K to-be-processed objects, perform the following processing on a kth channel predicted feature corresponding to a kth to-be-processed object and a kth channel sample feature corresponding to the kth to-be-processed object: obtain a first total fusion result obtained by fusing the kth channel predicted feature with K channel sample features; obtain a second total fusion result obtained by fusing the kth channel sample feature with K channel predicted features; and determine the sub-channel loss function value by combining the kth first fused feature, the first total fusion result and the second total fusion result; the kth first fused feature is a fusion result of the kth channel predicted feature and the kth channel sample feature; 1≤k≤K, and K is an integer greater than 1.

[0210] In some embodiments, the loss calculation module 4553 is further configured to obtain a first exclusive loss value which is positively correlated with the kth first fused feature and negatively correlated with the first total fusion result, obtain a second exclusive loss value which is positively correlated with the kth first fused feature and negatively correlated with the second total fusion result, and determine the sub-channel loss function value by combining the first exclusive loss value and the second exclusive loss value.

[0211] In some embodiments, the loss calculation module 4553 is further configured to down-sample the target modality predicted feature J times to obtain a down-sampled alignment feature after each down-sampling, down-sample the target modality sample feature J times to obtain a down-sampled sample feature after each down-sampling, J is a positive integer, use P loss functions to calculate a down-sampling loss function value between the down-sampled alignment feature and the down-sampled sample feature after each down-sampling, and obtain J down-sampling loss function values corresponding to the J times of down-sampling, and determine the feature loss function value based on the J down-sampling loss function values.

[0212] In some embodiments, the loss calculation module 4553 is further configured to obtain a loss function value of an original scale between the target modality predicted feature and the target modality sample feature, and determine the feature loss function value by combining the J down-sampling loss function values and the loss function value of the original scale.

[0213] In some embodiments, the loss calculation module 4553 is further configured to use P loss functions to calculate a loss function value of an original scale between the target modality predicted feature and the target modality sample feature.

[0214] In some embodiments, the feature alignment module 4552 is further configured to, for M feature channels corresponding to the target modality sample feature, obtain a channel feature dimension of each feature channel, use an initial feature extraction module of the first to-be-trained model to extract a feature of the source modality sample data in the target modality feature space to obtain an initial alignment feature, use a scale adjustment module of the first to-be-trained model to extract a channel predicted feature of the initial alignment feature based on the channel feature dimension on each feature channel to obtain M channel predicted features, and determine the M channel predicted features as the target modality predicted feature.

[0215] In some embodiments, the multi-modal data processing apparatus 455 further includes a model application module 4555 configured to, in response to an information prediction request, configure to-be-aligned data of a source modality, use the feature alignment model to extract a feature of the to-be-aligned data in a target modality feature space to obtain to-be-processed alignment features, and use an information prediction model to perform information prediction on the to-be-processed alignment features to obtain a first information prediction result, the information prediction model being used for information prediction on data features of a target modality.

[0216] In some embodiments, the multi-modal data processing apparatus 455 further comprises a modality determination module 4556 configured to collect information prediction data sets of a second to-be-trained model; the second to-be-trained model is an artificial neural network model to be trained for information prediction of data features of the target modality; in the information prediction data sets, the sample data amount of information prediction data corresponding to Q data modalities is counted respectively, Q is an integer greater than 1; the data modality with the largest sample data amount is determined as the target modality; the target data set of the target modality is obtained from the information prediction data sets; and the second to-be-trained model is trained based on the target data set to obtain the information prediction model.

[0217] In some embodiments, the model training module 4554 is further configured to perform information prediction on the target modality estimated features by using the information prediction model to obtain a second information prediction result; perform information prediction on the target modality sample features by using the information prediction model to obtain an information prediction label; determine an information prediction loss value between the second information prediction result and the information prediction label; and train the first to-be-trained model by combining the feature loss function value and the information prediction loss value to obtain the feature alignment model.

[0218] In some embodiments, the information prediction includes virtual object generation, entity segmentation, virtual reality enhancement, virtual reality detection, or information query; the virtual object generation refers to generating a virtual object based on data features of a target modality; the entity segmentation refers to segmenting an entity based on data features of the target modality; the virtual reality enhancement refers to virtually enhancing real information based on data features of the target modality; the virtual reality detection refers to determining an object to be virtually enhanced based on data features of the target modality; and the information query refers to querying associated information based on data features of the target modality.

[0219] In some embodiments, the model application module 4555 is further configured to obtain to-be-processed data of a target modality; perform feature extraction on the to-be-processed data to obtain to-be-processed data features; and perform data processing on the to-be-aligned data and the to-be-processed data by combining the to-be-processed alignment features and the to-be-processed data features.

[0220] The following continues to illustrate an exemplary structure of the multi-modal data processing apparatus provided by the embodiments of the present application implemented as a software module. In some embodiments, the software module in the multi-modal data processing apparatus stored in the memory can include: a data determination module configured to determine to-be-aligned data of a source modality in response to an information prediction request; a model calling module configured to extract features of the to-be-aligned data in a target modality feature space by using a feature alignment model, to obtain to-be-processed alignment features; the feature alignment model is trained by using the feature alignment model training method described above; the model calling module is further configured to perform information prediction on the to-be-processed alignment features by using an information prediction model, to obtain a first information prediction result, and the information prediction model is used to perform information prediction on data features of the target modality.

[0221] The embodiments of the present application provide a computer program product, which includes computer executable instructions or computer programs stored in a computer readable storage medium. A processor of an electronic device reads the computer executable instructions or computer programs from the computer readable storage medium, and the processor executes the computer executable instructions or computer programs, so that the electronic device performs the feature alignment model training method or the multi-modal data processing method provided by the embodiments of the present application.

[0222] The embodiments of the present application provide a computer readable storage medium, which stores computer executable instructions or computer programs. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the feature alignment model training method or the multi-modal data processing method provided by the embodiments of the present application, for example, the feature alignment model training method shown in FIG. 3. In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or various devices including one or any combination of the above memories.

[0223] In some embodiments, the computer-executable instructions can take the form of programs, software, software modules, scripts, or code, written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and they can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. By way of example, computer- executable instructions can, but need not, correspond directly to files in a file system. Computer- executable instructions can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. Computer-executable instructions can be deployed to be executed on one electronic device (in which case the one electronic device is the feature alignment model training device or the multi-modal data processing device), or on multiple electronic devices that are located close to one another (in which case the multiple electronic devices that are located close to one another are the feature alignment model training device or the multi-modal data processing device), or on multiple electronic devices that are distributed across multiple locations and interconnected through a communication network (in which case the multiple electronic devices that are distributed across multiple locations and interconnected through a communication network are the feature alignment model training device or the multi-modal data processing device).

[0224] It can be understood that, in the embodiments of the present application, related data such as sample data and data to be aligned are involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards, meet the principles of legality, legitimacy and necessity, do not involve obtaining data types prohibited or restricted by laws and regulations, and do not hinder the normal operation of target websites.

[0225] In summary, in the training of the first to-be-trained model for feature alignment, the first to-be-trained model is used to perform feature alignment on the source modality sample data to obtain target modality estimated features; P loss functions are used to calculate the feature loss function values between the target modality estimated features and the target modality sample features of the target modality sample data; finally, the first to-be-trained model is trained based on the feature loss function values to obtain a feature alignment model. Since the feature loss function values calculated in the training process of the first to-be-trained model are calculated by using P loss functions, the diversity of the loss functions used to calculate the loss function values is improved, and the use of multiple loss functions to calculate the loss function values can improve the accuracy of the feature loss function values; and the P loss functions include two or three of the error loss function, the similarity loss function and the mutual exclusion loss function, so that the accuracy of the feature loss function values can be improved from at least two dimensions of the value, the similarity and the mutual exclusion; and then, when the model is trained based on the accurate feature loss function values, the accuracy of the model training can be improved. In addition, by setting the feature size of the feature alignment result to M feature channels, the information loss caused by feature compression can be reduced, and the accuracy of feature alignment and model training can be improved. Furthermore, by selecting the data modality with the largest number of samples as the target modality, the accuracy of the information prediction model can be improved.

[0226] The above merely describes the embodiments of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement and improvement within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. A feature alignment model training method, the method being performed by an electronic device, the method comprising: obtaining source modality sample data and target modality sample features of target modality sample data; the source modality sample data and the target modality sample data are sample data describing the same object from different modalities; extracting, by using a first to-be-trained model, features of the source modality sample data in a target modality feature space, to obtain target modality estimated features; calculating, by using P loss functions, a feature loss function value between the target modality estimated features and the target modality sample features; the P loss functions include two or three of the following loss functions: an error loss function, a similarity loss function, and a mutual exclusion loss function, and P is an integer greater than 1; training the first to-be-trained model based on the feature loss function value to obtain a feature alignment model.

2. The method of claim 1, wherein, the first to-be-trained model is a to-be-trained artificial neural network model; the first to-be-trained model is used to extract features of source modality data in the target modality feature space.

3. The method of claim 1 or 2, wherein, the calculating, by using the P loss functions, of the feature loss function value between the target modality estimated features and the target modality sample features comprises: in M feature channels corresponding to the target modality estimated features, determining, by using the P loss functions, a channel loss function value under each feature channel based on the target modality estimated features and the target modality sample features; determining the feature loss function value based on M channel loss function values under the M feature channels; M is an integer greater than 1.

4. The method according to any one of claims 1 to 3, wherein, the determining, by using the P loss functions, of the channel loss function value under each feature channel based on the target modality estimated features and the target modality sample features in the M feature channels corresponding to the target modality estimated features and the target modality sample features comprises: for each feature channel, obtaining a channel estimated feature under the feature channel from the target modality estimated features; obtaining a channel sample feature under the feature channel from the target modality sample features; calculating, by using each loss function, a sub-channel loss function value between the channel estimated feature and the channel sample feature, to obtain P sub-channel loss function values; determining the channel loss function value under the feature channel based on the P sub-channel loss function values.

5. The method according to any one of claims 1 to 4, wherein, the calculating, by using each loss function, of the sub-channel loss function value between the channel estimated feature and the channel sample feature comprises: in response to the loss function being an error loss function, calculating a channel feature difference between the channel estimated feature and the channel sample feature; determining the sub-channel loss function value based on the channel feature difference.

6. The method according to any one of claims 1 to 5, wherein, the calculating, by using each loss function, of the sub-channel loss function value between the channel estimated feature and the channel sample feature comprises: in response to the loss function being a similarity loss function, fusing the channel estimated feature and the channel sample feature to obtain a first fused feature; fusing an absolute value of the channel estimated feature and an absolute value of the channel sample feature to obtain a second fused feature; The sub-channel loss function value is obtained based on the first fusion feature being positively correlated with the second fusion feature being negatively correlated.

7. The method according to any one of claims 1 to 6, wherein, The sub-channel loss function value between the channel predicted feature and the channel sample feature is calculated by using each loss function, including: In response to the loss function being an exclusive loss function, based on K channel predicted features corresponding to K to-be-processed objects and K channel sample features corresponding to K to-be-processed objects, the following processing is performed on the kth channel predicted feature corresponding to the kth to-be-processed object and the kth channel sample feature corresponding to the kth to-be-processed object: A first total fusion result is obtained by fusing the kth channel predicted feature with K channel sample features respectively. A second total fusion result is obtained by fusing the kth channel sample feature with K channel predicted features respectively. The sub-channel loss function value is determined in combination with the kth first fusion feature, the first total fusion result and the second total fusion result; the kth first fusion feature is a fusion result of the kth channel predicted feature and the kth channel sample feature; 1≤k≤K, K is an integer greater than 1.

8. The method according to any one of claims 1 to 7, wherein, The sub-channel loss function value is determined based on the kth first fusion feature, the first total fusion result and the second total fusion result, including: A first exclusive loss value is obtained based on the kth first fusion feature being positively correlated with the first total fusion result being negatively correlated. A second exclusive loss value is obtained based on the kth first fusion feature being positively correlated with the second total fusion result being negatively correlated. The sub-channel loss function value is determined in combination with the first exclusive loss value and the second exclusive loss value.

9. The method according to any one of claims 1 to 8, wherein, The feature loss function value between the target modality predicted feature and the target modality sample feature is calculated by using P loss functions, including: The target modality predicted feature is down-sampled J times to obtain a down-sampled alignment feature after each down-sampling; The target modality sample feature is down-sampled J times to obtain a down-sampled sample feature after each down-sampling; J is a positive integer; P loss functions are used to calculate a down-sampling loss function value between the down-sampled alignment feature and the down-sampled sample feature after each down-sampling, to obtain J down-sampling loss function values corresponding to J times of down-sampling; The feature loss function value is determined based on the J down-sampling loss function values.

10. The method according to any one of claims 1 to 9, wherein, The feature loss function value is determined based on the J down-sampling loss function values, including: An original scale loss function value between the target modality predicted feature and the target modality sample feature is obtained; The feature loss function value is determined in combination with the J down-sampling loss function values and the original scale loss function value.

11. The method according to any one of claims 1 to 10, wherein, The original scale loss function value between the target modality predicted feature and the target modality sample feature is obtained, including: P loss functions are used to calculate an original scale loss function value between the target modality predicted feature and the target modality sample feature.

12. The method according to any one of claims 1 to 11, wherein, The target modality predicted feature is obtained by extracting the feature of the source modality sample data in the target modality feature space by using the first to-be-trained model, including: For the M feature channels corresponding to the target modality sample features, a channel feature dimension of each feature channel is obtained; An initial feature extraction module of the first to-be-trained model is used to extract features of the source modality sample data in the target modality feature space, to obtain initial alignment features; A scale adjustment module of the first to-be-trained model is used to extract, on each feature channel, a channel predicted feature of the initial alignment features based on the channel feature dimension, to obtain M channel predicted features; The M channel predicted features are determined as the target modality predicted features.

13. The method according to any one of claims 1 to 12, wherein, After training the first to-be-trained model based on the feature loss function value to obtain a feature alignment model, the method further comprises: In response to an information prediction request, determining to-be-aligned data of a source modality; Using the feature alignment model to extract features of the to-be-aligned data in a target modality feature space to obtain to-be-processed alignment features; Using an information prediction model to perform information prediction on the to-be-processed alignment features to obtain a first information prediction result, the information prediction model being used for information prediction on data features of a target modality.

14. The method of any one of claims 1 to 13, wherein, Before using the information prediction model to perform information prediction on the to-be-processed alignment features to obtain a first information prediction result, the method further comprises: Collecting an information prediction data set of a second to-be-trained model; the second to-be-trained model being a to-be-trained artificial neural network model used for information prediction on data features of the target modality; In the information prediction data set, the sample data amount of information prediction data corresponding to Q data modalities is counted, Q being an integer greater than 1; The data modality with the largest sample data amount is determined as the target modality; From the information prediction data set, a target data set of the target modality is obtained; Training the second to-be-trained model based on the target data set to obtain the information prediction model.

15. The method according to any one of claims 1 to 14, wherein, The training of the first to-be-trained model based on the feature loss function value to obtain a feature alignment model comprises: Using an information prediction model to perform information prediction on the target modality predicted features to obtain a second information prediction result; Using the information prediction model to perform information prediction on the target modality sample features to obtain an information prediction label; Determining an information prediction loss value between the second information prediction result and the information prediction label; Training the first to-be-trained model in combination with the feature loss function value and the information prediction loss value to obtain the feature alignment model.

16. The method of any one of claims 1 to 15, wherein, The information prediction comprises virtual object generation, entity segmentation, virtual reality enhancement, virtual reality detection, or information query; The virtual object generation refers to generating a virtual object based on data features of a target modality; the entity segmentation refers to segmenting an entity based on data features of the target modality; the virtual reality enhancement refers to virtually enhancing real information based on data features of the target modality; the virtual reality detection refers to determining an object to be virtually enhanced based on data features of the target modality; and the information query refers to querying associated information based on data features of the target modality.

17. The method of any one of claims 1 to 16, wherein, After the feature alignment model is used to extract the features of the data to be aligned in the feature space of the target modality, the method further includes: obtaining to-be-processed data of a target modality; performing feature extraction on the to-be-processed data to obtain to-be-processed data features; performing data processing on the data to be aligned and the to-be-processed data in combination with the to-be-processed alignment features and the to-be-processed data features.

18. A multi-modal data processing method, the method being performed by an electronic device, and the method comprising: in response to an information prediction request, determining to-be-aligned data of a source modality; using a feature alignment model to extract features of the to-be-aligned data in a feature space of a target modality to obtain to-be-processed alignment features; the feature alignment model is trained using the feature alignment model training method of any one of claims 1 to 17; using an information prediction model to perform information prediction on the to-be-processed alignment features to obtain a first information prediction result, the information prediction model being configured to perform information prediction on data features of a target modality.

19. A feature alignment model training apparatus, the feature alignment model training apparatus comprising: a data obtaining module configured to obtain source modality sample data and target modality sample features of target modality sample data; the source modality sample data and the target modality sample data are sample data describing the same object from different modalities; a feature alignment module configured to use a first to-be-trained model to extract features of the source modality sample data in a feature space of a target modality to obtain target modality estimated features; a loss calculation module configured to use P loss functions to calculate a feature loss function value between the target modality estimated features and the target modality sample features; the P loss functions include two or three of the following loss functions: an error loss function, a similarity loss function, and a mutual exclusion loss function, and P is an integer greater than 1; a model training module configured to train the first to-be-trained model based on the feature loss function value to obtain a feature alignment model.

20. A multi-modal data processing apparatus, the multi-modal data processing apparatus comprising: a data determination module configured to determine to-be-aligned data of a source modality in response to an information prediction request; a model calling module configured to use a feature alignment model to extract features of the to-be-aligned data in a feature space of a target modality to obtain to-be-processed alignment features; the feature alignment model is trained using the feature alignment model training method of any one of claims 1 to 17; the model calling module is further configured to use an information prediction model to perform information prediction on the to-be-processed alignment features to obtain a first information prediction result, the information prediction model being configured to perform information prediction on data features of a target modality.

21. An electronic device, the electronic device comprising: a memory configured to store computer executable instructions or computer programs; A processor, configured to implement the feature alignment model training method according to any one of claims 1 to 17, or the multi-modal data processing method according to claim 18, when executing computer executable instructions or computer programs stored in the memory. 22.A computer readable storage medium storing computer executable instructions or computer programs, the computer executable instructions or computer programs, when executed by a processor, implement the feature alignment model training method according to any one of claims 1 to 17, or the multi-modal data processing method according to claim 18. 23.A computer program product comprising computer executable instructions or computer programs, the computer executable instructions or computer programs, when executed by a processor, implement the feature alignment model training method according to any one of claims 1 to 17, or the multi-modal data processing method according to claim 18.

Images