Method, electronic device and computer program product for training a model

Abstract

Embodiments of the present disclosure provide a method, an electronic device and a computer program product for training a model. The method can include determining image features, audio features and text features of a reference object based on reference image information, reference audio information and reference text information associated with the reference object, respectively. The method can also include constructing the image features, the audio features and the text features into a feature tensor. Furthermore, the method can further include decomposing the feature tensor into a first feature vector, a second feature vector and a third feature vector corresponding to the image features, the audio features and the text features, respectively, to determine a loss function value of the model. The method can also include updating parameters of the model based on the loss function value. Embodiments of the present disclosure achieve the association and denoising of cross-modal information by constructing a feature tensor in the model training process, thereby improving user experience.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of data processing, and more specifically, to methods, electronic devices, and computer program products for training models. Background Technology

[0002] The intelligent and digital reconstruction of features such as speaking faces is crucial for research in many fields. Therefore, reconstructing three-dimensional or higher-dimensional visual representations based on image or video information, along with corresponding audio and text information, has wide-ranging applications. Importantly, humans acquire information and learn knowledge through seeing, listening, and reading; that is, the primary way humans learn is by exploring multimodal resources, including images, audio, and text information. Extending this to machine learning or artificial intelligence, machines with human-like intelligence should also be trained based on multimodal resources. However, currently, effective cross-modal model training schemes are lacking. Summary of the Invention

[0003] The embodiments disclosed herein provide a model training scheme.

[0004] In a first aspect of this disclosure, a method for training a model is provided. The method may include determining image features, audio features, and text features of a reference object based on reference image information, reference audio information, and reference text information associated with the reference object, respectively. The method may further include constructing a feature tensor from the image features, audio features, and text features. Furthermore, the method may further include decomposing the feature tensor into a first feature vector, a second feature vector, and a third feature vector corresponding to the image features, audio features, and text features, respectively, to determine a loss function value for the model. The method may also include updating the parameters of the model based on the loss function value.

[0005] In a second aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory coupled to the processor, the memory having instructions stored therein, the instructions causing the electronic device to perform actions when executed by the processor, the actions comprising: determining image features, audio features, and text features of the reference object based on reference image information, reference audio information, and reference text information associated with the reference object, respectively; constructing the image features, the audio features, and the text features into a feature tensor; decomposing the feature tensor into a first feature vector, a second feature vector, and a third feature vector corresponding to the image features, the audio features, and the text features, respectively, to determine a loss function value of the model; and updating the parameters of the model based on the loss function value.

[0006] In a third aspect of this disclosure, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform any step of the method according to the first aspect.

[0007] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0008] The above and other objects, features, and advantages of this disclosure will become more apparent from the accompanying drawings, in which the same or similar reference numerals generally represent the same or similar parts. In the drawings:

[0009] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure can be implemented is shown;

[0010] Figure 2 A schematic diagram of a detailed example environment for training and applying a model according to embodiments of the present disclosure is shown;

[0011] Figure 3 A flowchart illustrating a process for training a model according to an embodiment of the present disclosure is shown;

[0012] Figure 4 A schematic diagram of an example environment for tensor construction and decomposition according to embodiments of the present disclosure is shown;

[0013] Figure 5 A flowchart illustrating the process of applying a trained cross-modal coding model according to embodiments of the present disclosure is shown; and

[0014] Figure 6 A block diagram of a computing device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0015] The principles of this disclosure will now be described with reference to several exemplary embodiments shown in the accompanying drawings.

[0016] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "a set of example embodiments". The term "another embodiment" means "another set of embodiments". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] As mentioned above, traditional 3D or higher-dimensional reconstruction techniques are typically based solely on 2D image information. To more accurately and faithfully reconstruct high-dimensional visual representations, multi-modal 3D reconstruction techniques have been proposed. For example, 3D or higher-dimensional visual representations can be reconstructed using 2D video information (i.e., image information), along with corresponding audio and text information—three modalities of reconstruction information. It should be understood that, in addition to 2D video information, audio information can be used to more precisely control the mouth movements of a person or object, and text information can be used to more subtly express the mood and facial expressions of a person or object.

[0018] Cross-modal high-dimensional visual representation reconstruction technology has a wide range of applications. For telepresence applications in AR or VR, faithful reproduction of head posture, mouth shape, facial expressions, and other features is required. Especially in educational settings, reconstructed virtual visual representations of students can protect their privacy. Furthermore, it can help people with disabilities conceal their physical limitations, allowing them to play and run freely in the virtual world like everyone else.

[0019] However, traditional research on cross-modal high-dimensional visual representation reconstruction techniques typically considers only two modalities. Even in some explorations that have used resources from three modalities, this information is simply directly spliced ​​and applied. During traditional model training, the information resources from the three modalities are not fully learned and contain unwanted noise. Therefore, current cross-modal high-dimensional visual representation reconstruction techniques still suffer from low efficiency and low accuracy.

[0020] In view of this, embodiments of the present disclosure propose a model training scheme. In this scheme, image, audio, and text information from the training dataset are input into corresponding encoders to obtain corresponding image, audio, and text features. Furthermore, to improve training performance, the obtained image, audio, and text features can be constructed as feature tensors, i.e., three-dimensional matrices. Then, known tensor decomposition algorithms can be used to decompose the feature tensors into first, second, and third feature vectors corresponding to the image, audio, and text features, as well as noise information. This approach achieves the association and denoising of cross-modal information, thereby optimizing the model training process. The model trained in this way can reconstruct more realistic facial features, actions, and other visual identifiers, thus improving the user experience.

[0021] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings. Figure 1 A schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, the example environment 100 includes multiple modal information resources from the user, such as image information 110, audio information 120, and text information 130. In some embodiments, image information 110 may be video information composed of multiple frames of images, and in subsequent processing, image information 110 may be a sub-image of at least one region in one or more frames of images. Audio information 120 may be at least one segment of audio from an audio resource. Text information 130 may be at least one character, word, or phrase from a text resource.

[0022] like Figure 1 As shown, system 100 may include computing device 140. Computing device 140 may be configured to receive image information 110, audio information 120, and text information 130 associated with user operations, and to determine or reconstruct a high-dimensional representation 150 of the user through computation.

[0023] In some embodiments, the computing device 140 may include, but is not limited to, a personal computer, a server computer, a handheld or laptop device, a mobile device (such as a mobile phone, a personal digital assistant PDA, a media player, etc.), a consumer electronics product, a minicomputer, a mainframe computer, cloud computing resources, etc. It should be understood that, based on considerations such as cost, the computing device 140 may or may not have sufficient computing resources for model training.

[0024] In some embodiments, the high-dimensional representation 150 may be a virtual three-dimensional or higher-dimensional image or animation used to reproduce a user's facial expressions, movements, etc.

[0025] It should be understood that the architecture and functionality in example environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Embodiments of this disclosure can also be applied to other environments with different structures and / or functionalities.

[0026] To describe the training process of the image processing model in more detail, the following will refer to... Figure 2 This describes the training and application process of the model. Figure 2 A schematic diagram of a detailed example environment 200 for training and applying a model according to embodiments of the present disclosure is shown. Figure 2 As shown, the example environment 200 can generally include a model training system 260 and a model application system 270. As an example, the model training system 260 and / or the model application system 270 can be configured as follows: Figure 1 The example is implemented in the computing device 140 shown. It should be understood that the description of the structure and function of the example environment 200 is for illustrative purposes only and is not intended to limit the scope of the subject matter described herein. The subject matter described herein may be implemented in different structures and / or functions.

[0027] As previously described, the process of reconstructing a high-dimensional representation from low-dimensional multimodal information resources can be divided into two stages: a model training stage and a model application stage. As an example, in the model training stage, the model training system 260 can use the training dataset 250 to train a cross-modal coding model 240 to perform the corresponding function. In the model application stage, the model application system 270 can receive the trained cross-modal coding model 240. Thus, the cross-modal coding model 240 loaded into the computing device 140 of the model application system 270 can reconstruct a high-dimensional representation 230 based on the input arbitrary multimodal information 210.

[0028] In other embodiments, model 240 can be constructed as a learning network. In some embodiments, the learning network may include multiple networks, each of which may be a multi-layer neural network composed of a large number of neurons. Through a training process, the corresponding parameters of the neurons in each network can be determined. The parameters of the neurons in these networks are collectively referred to as the parameters of the cross-modal coding model 240.

[0029] The training process of the cross-modal coding model 240 can be performed iteratively until at least some of the parameters of the cross-modal coding model 240 converge or until a predetermined number of iterations is reached, thereby obtaining the final model parameters.

[0030] The technical solutions described above are for illustrative purposes only and are not intended to limit the invention. It should be understood that the networks can be arranged in other ways and with different connections. To more clearly explain the principles of the above solutions, the following will refer to... Figure 3Let's describe the model training process in more detail.

[0031] Figure 3 A flowchart of a process 300 for training a model according to an embodiment of the present disclosure is shown. In some embodiments, process 300 may be performed at... Figure 1 Implemented in computing device 140 or other computing devices. See now for reference. Figure 3 and combined Figure 1 The process of training a model according to an embodiment of this disclosure is described in 300. For ease of understanding, the specific examples mentioned in the following description are exemplary and are not intended to limit the scope of protection of this disclosure.

[0032] At 302, computing device 140 can determine the image features, audio features, and text features of the reference object based on reference image information, reference audio information, and reference text information associated with the reference object, respectively. In some embodiments, the reference object may be a human face, i.e., the reference image information, reference audio information, and reference text information are all derived from a training dataset related to human facial movements. Alternatively or additionally, the reference object may also be the body language, actions, etc., of a person or other object. In some embodiments, the reference image information may be a sub-image of at least one region in one or more frames of images, the reference audio information may be at least one segment of audio from an audio resource, and the reference text information may be at least one character, word, or phrase from a text resource.

[0033] In some embodiments, to determine the aforementioned image features, audio features, and text features, the computing device 140 may use a pre-set video encoder to determine image features based on reference image information, a pre-set audio encoder to determine audio features based on reference audio information, and a pre-set text encoder to determine text features based on reference text information. As an example, a fast R-CNN may be used as the video encoder, wave2vec2.0 as the audio encoder, and BERT as the text encoder to determine the image features, audio features, and text features, respectively.

[0034] At 304, computing device 140 can construct a feature tensor from the determined image features, audio features, and text features. In some embodiments, computing device 140 can set the determined image features, audio features, and text features along a first coordinate, a second coordinate, and a third coordinate to form a three-dimensional space. As an example, if the image features correspond to X sub-images, the audio features correspond to Y sub-audio sounds, and the text features correspond to Z characters or words, then a three-dimensional feature tensor of X×Y×Z can be constructed.

[0035] It should be understood that a position in this three-dimensional space corresponds to a combination of an image feature in the image features, a corresponding audio feature in the audio features, and a corresponding text feature in the text features. As an example, in the coordinate system of the above three-dimensional space, the coordinates (1,1,1) can correspond to the combination of the first image feature, the first audio feature, and the first text feature.

[0036] Furthermore, to construct the feature tensor, the computing device 140 can also determine the values ​​of the corresponding coordinate positions based on the pre-labeled association information of the above combinations, thus forming a part of the feature tensor. As an example, during model training, one or two features of a specific combination of image features, audio features, and text features can be replaced, resulting in mismatches between image features, audio features, and text features. Therefore, various matching or mismatching cases can be assigned values, i.e., association information. For example, a case where image features, audio features, and text features match can be defined as 1; a case where image features and audio features match, but text features do not match can be defined as 2; a case where image features and text features match, but audio features do not match can be defined as 3; a case where audio features and text features match, but image features do not match can be defined as 4; and a case where image features and audio features match, but text features do not match can be defined as 5. In this way, corresponding association information can be filled at all coordinates of the feature tensor, thus considering the association information between each modality in the subsequent training process.

[0037] At 306, computing device 140 can decompose the constructed feature tensor into a first feature vector, a second feature vector, and a third feature vector, respectively, corresponding to the determined image features, audio features, and text features, in order to determine the loss function value of the model.

[0038] In some embodiments, the CANDECOMP / PARAFAC decomposition (i.e., CP decomposition) algorithm can be specifically used to decompose the feature tensor into three eigenvectors and noise. This yields a first eigenvector, a second eigenvector, and a third eigenvector. Each of the first, second, and third eigenvectors contains noise-removed correlation information of the feature tensor. It should be understood that this disclosure is applicable to other decomposition algorithms for decomposing a tensor into vectors of a specified bibliography.

[0039] In some embodiments, the loss function value can be determined by the computing device 140 based on the first feature vector, the second feature vector, the third feature vector, and the corresponding image features, audio features, and text features. As an example, the absolute values ​​of the differences between the first feature vector and the image features, the second feature vector and the audio features, and the third feature vector and the text features can be determined respectively, and these absolute values ​​can be summed.

[0040] At 308, computing device 140 can update the model parameters based on the determined loss function value. Similar to traditional model training methods, computing device 140 can adjust the model parameters based on the determined loss function value until the loss function value is minimized, thereby training a convergent model.

[0041] To illustrate the embodiments of this disclosure in more detail, reference is now made to... Figure 4 Describe in detail the process of constructing and decomposing the feature tensor. Figure 4 A schematic diagram of an example environment 400 for tensor construction and decomposition according to embodiments of the present disclosure is shown.

[0042] like Figure 4 As shown, after image information 410, audio information 420, and text information 430 are input into a computing device used to train the model, the video encoder 412 in the computing device processes the image information 410 to obtain the corresponding feature representation 414. Similarly, and in parallel, the audio encoder 422 in the computing device processes the audio information 420 to obtain the corresponding feature representation 424. The text encoder 432 in the computing device processes the text information 430 to obtain the corresponding feature representation 434. As an example, the video encoder 412 could be R_CNN, the audio encoder 422 could be wave2vec 2.0, and the text encoder 432 could be BERT.

[0043] Tensor construction decomposition unit 440 first constructs feature representations 414, 424, and 434 into feature tensors. As an example, tensor construction decomposition unit 440 can construct image features corresponding to X sub-images, audio features corresponding to Y sub-audio, and text features corresponding to Z characters or words into a three-dimensional feature tensor of size X×Y×Z.

[0044] Subsequently, the tensor decomposition unit 440 can use any tensor decomposition algorithm to decompose the aforementioned three-dimensional feature tensor into a first eigenvector 416, a second eigenvector 426, a third eigenvector 436, and noise. In this way, a denoised vector representation can be obtained.

[0045] Through the above processing, the first feature vector 416, the second feature vector 426, and the third feature vector 436 can be obtained. Therefore, the loss function value of the model can be determined more accurately based on the first feature vector 416, the second feature vector 426, the third feature vector 436, and the feature representations 414, 424, and 434, thereby optimizing the model training process.

[0046] It should be understood that after training the cross-modal coding model 240, it is possible to... Figure 1 The model is applied as shown. Figure 5 A flowchart of a process 500 for applying a trained cross-modal coding model 240 according to embodiments of the present disclosure is shown. In some embodiments, process 500 may be performed at... Figure 1 Implemented in computing device 140 or other computing devices. See now for reference. Figure 5 and combined Figure 1 The process 500 describes an application model according to an embodiment of this disclosure. For ease of understanding, the specific examples mentioned in the following description are exemplary and are not intended to limit the scope of protection of this disclosure.

[0047] At 502, computing device 140 can train cross-modal coding model 240 as described in the above embodiments. In some embodiments, for example, when computing device 140 has insufficient computing power, the cross-modal coding model 240 can be trained without using computing device 140, and a model trained by other computing devices with stronger computing power can be directly loaded onto computing device 140.

[0048] At 504, computing device 140 can input the user's image information 110, audio information 120, and text information 130 into a trained cross-modal coding model 240 to determine the user's high-dimensional representation. As an example, a three-dimensional, four-dimensional, or higher-dimensional virtual representation of the user's face can be obtained. It should be understood that the specific process of determining high-dimensional virtual representations based on cross-modal coding models has been mentioned in many research articles (such as Guy Gafni and others. "Dynamic Neural RadianceFields for Monocular 4D Facial Avatar Reconstruction". in:CoRR abs / 2012.03065(2020).arXiv:2012.03065.), and therefore will not be elaborated upon here.

[0049] Through the above embodiments, this disclosure provides a novel framework for reconstructing high-dimensional virtual representations, which integrates three modalities during model training. Furthermore, this disclosure improves the cross-modal pre-training framework using a tensor decomposition algorithm, thereby allowing model training to consider more information (such as inter-modal correlations) and eliminate noise. Thus, the model training method of this disclosure improves training efficiency and accuracy, and the trained model can reconstruct more realistic facial features, actions, and other visual identifiers, optimizing the user experience.

[0050] Figure 6A block diagram of a computing device 600 capable of implementing various embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0051] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0052] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0053] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as processes 300 and 500. For example, in some embodiments, processes 300 and 500 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of processes 300 and 500 described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute processes 300, 500 by any other suitable means (e.g., by means of firmware).

[0054] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0055] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0056] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0057] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0058] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0059] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0060] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0061] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a model, comprising: Using a video encoder, an audio encoder, and a text encoder respectively, the image features, audio features, and text features of the reference object are determined based on reference image information, reference audio information, and reference text information associated with the reference object. The image features, audio features, and text features are constructed into a feature tensor. Constructing the feature tensor includes: setting the image features, audio features, and text features along a first coordinate, a second coordinate, and a third coordinate to form a three-dimensional space, where a given position in the three-dimensional space corresponds to a combination of a corresponding image feature in the image features, a corresponding audio feature in the audio features, and a corresponding text feature in the text features; and determining the value of the given position based on pre-labeled association information of the combination to form a part of the feature tensor. The feature tensor is decomposed into a first feature vector, a second feature vector, and a third feature vector, respectively, corresponding to the image features, the audio features, and the text features, to determine the loss function value of the model, wherein the loss function value is calculated at least in part based on a combination of the following: a first absolute value of the difference between the first feature vector and one or more of the image features, a second absolute value of the difference between the second feature vector and one or more of the audio features, and a third absolute value of the difference between the third feature vector and one or more of the text features; and The parameters of the model are updated based on the loss function value. The method is performed by at least one processing device, which includes a processor coupled to a memory.

2. The method according to claim 1, wherein the first feature vector, the second feature vector, and the third feature vector all contain the noise-removed association information of the feature tensor.

3. The method according to claim 1, wherein the feature tensor corresponds to the first feature vector, the second feature vector, the third feature vector, and noise.

4. The method according to claim 1, wherein the reference object is a human face.

5. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, the instructions which, when executed by the processor, cause the electronic device to perform actions, the actions including: Using a video encoder, an audio encoder, and a text encoder respectively, the image features, audio features, and text features of the reference object are determined based on reference image information, reference audio information, and reference text information associated with the reference object. The image features, audio features, and text features are constructed into a feature tensor. Constructing the feature tensor includes: setting the image features, audio features, and text features along a first coordinate, a second coordinate, and a third coordinate to form a three-dimensional space, where a given position in the three-dimensional space corresponds to a combination of a corresponding image feature in the image features, a corresponding audio feature in the audio features, and a corresponding text feature in the text features; and determining the value of the given position based on pre-labeled association information of the combination to form a part of the feature tensor. The feature tensor is decomposed into a first feature vector, a second feature vector, and a third feature vector, respectively, corresponding to the image features, the audio features, and the text features, to determine the model's loss function value, wherein the loss function value is calculated at least in part based on a combination of the following: a first absolute value of the difference between the first feature vector and one or more of the image features, a second absolute value of the difference between the second feature vector and one or more of the audio features, and a third absolute value of the difference between the third feature vector and one or more of the text features; and The parameters of the model are updated based on the loss function value.

6. The electronic device of claim 5, wherein the first feature vector, the second feature vector, and the third feature vector each contain the noise-removed association information of the feature tensor.

7. The electronic device of claim 5, wherein the feature tensor corresponds to the first feature vector, the second feature vector, the third feature vector, and noise.

8. The electronic device according to claim 5, wherein the reference object is a human face.

9. A computer program product tangibly stored on a computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 4.

Images