Model training method and apparatus, and computing device cluster

Abstract

The present invention provides a model training method and apparatus, and a computing device cluster. The method comprises: determining a first triplet sample corresponding to each modality among a plurality of modalities, wherein the first triplet sample is obtained by encoding, in a triplet manner, a sample corresponding to the modality, a triplet consists of a first data point, a second data point, and the relationship between the first data point and the second data point, and the first triplet sample is a plurality of instances of the triplet arranged in sequence; and training a model on the basis of the first triplet sample corresponding to each modality, to obtain a trained model. Unified data encoding is performed on data of a plurality of modalities by means of triplets, so that data of different modalities is unified into a common representation space, and the model can make full use of information of the data, thereby improving the generalization and precision of the model.

Description

Model training method, device and computing device cluster

[0001] The present application claims priority to Chinese Patent Application No. 202410799249.7, filed on June 19, 2024, entitled "A Model Training Method and Device", and Chinese Patent Application No. 202411037116.2, filed on July 30, 2024, entitled "A Model Training Method, Device and Computing Device Cluster", the contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and particularly relates to a model training method, device and computing device cluster. BACKGROUND

[0003] In the field of artificial intelligence, data encoding is needed to lay the foundation for subsequent model training, analysis and application. Data encoding refers to converting raw data into a format that can be understood, stored and processed by a computer.

[0004] Currently, data encoding can be achieved by converting data into tensors. Tensors, as a representation of multi-dimensional arrays, play a crucial role in deep learning frameworks as they can efficiently represent and compute weights, inputs and outputs in models.

[0005] However, the data encoding method of converting data into tensors may simplify or ignore the complexity and diversity of data, resulting in a model that cannot fully utilize the information of the data. SUMMARY

[0006] The present application provides a model training method, device and computing device cluster, which uniformly encodes data of multiple modalities through triplets, thereby unifying data of different modalities into a common representation space and enabling the model to fully utilize the information of the data and improve the generalization and precision of the model.

[0007] The first aspect of the present application provides a model training method, the method comprising: determining a first triplet sample corresponding to each modality in a plurality of modalities, the first triplet sample being obtained by encoding a sample corresponding to the modality in a triplet manner; the triplet consisting of a first data point, a second data point and a relationship between the first data point and the second data point; the first triplet sample being a plurality of instances of the triplet arranged in sequence; training a model based on the first triplet sample corresponding to each modality to obtain a trained model.

[0008] In the scheme, the data of various modalities is uniformly encoded through triplets, so that the data of different modalities is unified into a common representation space, so that the model can fully utilize the information of the data and improve the generalization and precision of the model.

[0009] In a possible implementation, the model is trained based on the first triplet samples corresponding to each modality, including:

[0010] For each modality, the target element in the first triplet sample corresponding to the modality is changed, and the changed first triplet sample is determined; and the model parameters of the model are updated based on the changed first triplet sample and the target element.

[0011] In the scheme, the model combines the data of various modalities to express elements, so that the element expression has rich knowledge content.

[0012] In a possible implementation, the changing manner is masking or adding noise.

[0013] In a possible implementation, the model parameters of the model are updated based on the changed first triplet sample and the target element, including:

[0014] The vectorization result is determined, the vectorization result indicating the first vectorization representation of each instance in the changed first triplet sample, the first vectorization representation being determined by the second vectorization representation of each element in the instance; the vectorization result is input to the model to obtain the output of the model, the output of the model being the fourth vectorization representation of each instance in the first triplet sample corresponding to the modality; the representation error of the target element is determined based on the output of the model and the third vectorization representation of the target element; and the model parameters of the model are updated based on the representation error of the target element.

[0015] In a possible implementation, the first vectorization representation of the instance is the result of fusion processing of the second vectorization representation of each element in the instance.

[0016] The representation error of the target element is determined based on the output of the model and the third vectorization representation of the target element, including:

[0017] The fourth vectorization representation of the instance in which the target element is located in the output of the model is determined; the second vectorization representation of other elements in the instance in which the target element is located is determined; the fifth vectorization representation of the target element is obtained through inverse operation of fusion operation based on the fourth vectorization representation and the second vectorization representation of the other elements; and the representation error of the target element is determined based on the third vectorization representation and the fifth vectorization representation of the target element.

[0018] In a possible implementation, the method further includes:

[0019] determine a second triple sample corresponding to the target modality in the plurality of modalities, the second triple sample being obtained by encoding samples corresponding to the target modality in a triple manner; the target modality being at least one modality in the plurality of modalities; and training the trained model based on the second triple sample corresponding to the target modality to obtain the downstream task model.

[0020] In this scheme, the rich knowledge and context information learned by pre-training are used to quickly adapt to new tasks.

[0021] In a possible implementation, the plurality of modalities includes any of the following:

[0022] structured data, images, graph data, and sequence data.

[0023] In a possible implementation, for structured data, field values of adjacent fields in the structured data are located in the same instance of the triple, and the relationship is the position of the first data point or the second data point in the structured data.

[0024] For images, adjacent pixel values in the same row and / or the same column of the image are located in the same instance of the triple, and the relationship is the position of the first data point or the second data point in the structured data.

[0025] For graph data, two connected nodes in the graph data are located in the same instance of the triple, and the relationship is the position of the first data point or the second data point in the structured data, or the weight of the edge between the two connected nodes.

[0026] For sequence data, two adjacent objects in the sequence data are located in the same instance of the triple.

[0027] The model training device provided in the embodiment of the present application includes a plurality of modules, each module is used to execute each step in the model training method provided in the first aspect of the present application, and the division of the modules is not limited herein. For the specific functions performed by each module of the model training device and the beneficial effects achieved, please refer to the functions of each step of the model training method provided in the first aspect of the present application, which will not be repeated here.

[0028] Exemplarily, the model training device includes:

[0029] The encoding module is configured to determine a first triple sample corresponding to each modality in the plurality of modalities, the first triple sample being obtained by encoding samples corresponding to the modality in a triple manner; the triple being composed of a first data point, a second data point, and a relationship between the first data point and the second data point; and the first triple sample being a plurality of instances of the triple arranged in sequence.

[0030] The training module is configured to train the model based on the first triple sample corresponding to each modality to obtain a trained model.

[0031] In a possible implementation, the training module is configured to change the target element in the first triple sample corresponding to each modality to obtain a changed first triple sample; and update the model parameter of the model based on the changed first triple sample and the target element.

[0032] In a possible implementation, the changing manner is masking or adding noise.

[0033] In a possible implementation, the training module is configured to determine a vectorization result, the vectorization result indicating a first vectorization representation of each instance in the changed first triple sample, the first vectorization representation being determined by a second vectorization representation of each element in the instance; input the vectorization result to the model to obtain an output of the model, the output of the model being a fourth vectorization representation of each instance in the first triple sample corresponding to the modality; determine a representation error of the target element based on the output of the model and a third vectorization representation of the target element; and update the model parameter of the model based on the representation error of the target element.

[0034] In a possible implementation, the first vectorization representation of the instance is a result of fusion processing of the second vectorization representation of each element in the instance.

[0035] The training module is configured to determine a fourth vectorization representation of the instance in which the target element is located in the output of the model; determine a second vectorization representation of another element in the instance in which the target element is located; obtain a fifth vectorization representation of the target element by inverse operation of fusion operation based on the fourth vectorization representation and the second vectorization representation of the another element; and determine a representation error of the target element based on the third vectorization representation and the fifth vectorization representation of the target element.

[0036] In a possible implementation, the apparatus further includes:

[0037] The fine-tuning module is configured to determine a second triple sample corresponding to a target modality in the plurality of modalities, the second triple sample being obtained by encoding the sample corresponding to the target modality in a triple manner; the target modality being at least one modality in the plurality of modalities; and train the trained model based on the second triple sample corresponding to the target modality to obtain a downstream task model.

[0038] In a possible implementation, the plurality of modalities includes any of the following:

[0039] Structured data, image, graph data, and sequence data.

[0040] In a possible implementation, for structured data, field values of adjacent fields in the structured data are located in the same instance of the triple, and the relationship is a position of the first data point or the second data point in the structured data.

[0041] For images, adjacent pixel values in the same row of the image, and / or, adjacent pixel values in the same column of the image are located in the same instance of the triple, and the relationship is a position of the first data point or the second data point in the structured data.

[0042] For graph data, two connected nodes in the graph data are located in the same instance of the triple, and the relationship is a position of the first data point or the second data point in the structured data, or a weight of an edge between the two connected nodes.

[0043] For sequence data, two adjacent objects in the sequence data are located in the same instance of the triple.

[0044] A third aspect of the present application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device performs the method provided in any possible design of the first aspect.

[0045] A fourth aspect of the present application provides a computer program product including instructions, when the instructions are executed by a computing device cluster, the computing device cluster performs the method provided in the first aspect or any possible design of the first aspect.

[0046] A fifth aspect of the present application provides a computer-readable storage medium including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method provided in the first aspect or any possible design of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required to be used in the embodiments will be simply introduced as follows.

[0048] FIG. 1 is a schematic diagram of an artificial intelligence framework provided by an embodiment of the present application;

[0049] FIG. 2 is a schematic diagram of a model training scenario provided in the related art;

[0050] FIG. 3 is an architectural schematic diagram of a model management system provided by an embodiment of the present application;

[0051] FIG. 4 is a flow schematic diagram of a model training method provided by an embodiment of the present application;

[0052] FIG. 5 is a schematic diagram of a triple provided by an embodiment of the present application;

[0053] FIG. 6 is a schematic diagram of graph data provided by an embodiment of the present application;

[0054] FIG. 7 is a schematic diagram of triple sample vectorization provided by an embodiment of the present application;

[0055] FIG. 8a is a schematic diagram of error calculation one provided by an embodiment of the present application;

[0056] FIG. 8b is a schematic diagram of error calculation two provided by an embodiment of the present application;

[0057] FIG. 8c is a schematic diagram of error calculation three provided by an embodiment of the present application;

[0058] FIG. 9a is a schematic diagram of architecture of a model training system one provided by an embodiment of the present application;

[0059] FIG. 9b is a schematic diagram of architecture of a model training system two provided by an embodiment of the present application;

[0060] FIG. 10 is a schematic diagram of a flow of a model application method provided by an embodiment of the present application;

[0061] FIG. 11 is a schematic diagram of a model training apparatus provided by an embodiment of the present application;

[0062] FIG. 12 is a schematic diagram of a structure of a computing device provided by an embodiment of the present application;

[0063] FIG. 13 is a schematic diagram of a structure of a computing device cluster provided by an embodiment of the present application;

[0064] FIG. 14 is a schematic diagram of a connection of computing devices in a computer cluster provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] Hereinafter, some terms in the present embodiment are explained. It should be noted that the explanations are for the convenience of understanding by those skilled in the art, and do not limit the scope of protection required by the present application.

[0066] Artificial Intelligence (AI): is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence, and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc.

[0067] Pre-training / trained: refers to a pre-trained model or the process of pre-training a model. Generally, pre-training is done on a large dataset to facilitate subsequent fine-tuning.

[0068] Fine-tuning: refers to the process of applying a pre-trained model to your own dataset and adapting the parameters to your dataset. Generally, the real task dataset is small, and fine-tuning the pre-trained model can achieve good results.

[0069] Data encoding: In AI, data encoding refers to the process of converting raw data into a format that computers can understand, store, and process. This is a crucial step in AI and machine learning projects, as it ensures data consistency and accuracy and lays the foundation for subsequent model training and analysis.

[0070] Triplet Set (ST): a data structure commonly used in mathematics and computer science, consisting of three elements, usually represented as (a, b, c), where each element can be a number, character, object, or any other type of data.

[0071] Structured data (Tabular, T): structured data refers to data stored and represented in a computer through data structures, such as tables, charts, and tables in relational databases.

[0072] Graph data (Graph, G): a data structure used to represent complex relationships between entities (nodes or vertices). Graph data consists of nodes and edges connecting these nodes, making it ideal for representing connections between entities such as networks, social relationships, and transportation systems.

[0073] Time series data (Time Series, TS): a collection of data recorded in chronological order, with each data point associated with a specific timestamp. This data type is commonly used in finance, meteorology, industrial automation, healthcare, and transportation monitoring.

[0074] Visual data (Vision, V): information obtained through visual means, usually including images, videos, and other visual forms of data. This data type is crucial in computer vision, image processing, medical imaging, security monitoring, and autonomous driving.

[0075] Inverse operation: refers to the inverse operation of a mathematical operation, which can restore the result to its original value.

[0076] Figure 1 shows a schematic diagram of an artificial intelligence subject framework, which describes the overall workflow of an artificial intelligence system, and is applicable to general artificial intelligence field requirements.

[0077] The above-mentioned artificial intelligence subject framework is described below from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0078] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom".

[0079] The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of artificial intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0080] (1) Infrastructure:

[0081] The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. Hardware acceleration chips); the underlying platform includes distributed computing framework and network related platform guarantee and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication acquire data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for calculation.

[0082] (2) Data

[0083] The data above the infrastructure layer is used to represent the data source in the field of artificial intelligence. Data involves graphics, images, speech, text, and also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, humidity, etc.

[0084] (3) Data processing

[0085] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0086] Among them, machine learning and deep learning can symbolize and formalize intelligent information modeling, extraction, data preprocessing, training, etc.

[0087] Reasoning refers to simulating human intelligent reasoning methods in a computer or intelligent system, using formalized information to perform machine thinking and solve problems according to a reasoning control strategy, and the typical function is search and matching.

[0088] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0089] (4) General capabilities

[0090] After the data mentioned above is processed, some general capabilities can be formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0091] (5) Intelligent products and industry applications

[0092] Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields, which are the packaging of the overall solution of artificial intelligence, and realize the application of intelligent information decision-making products. Its application fields mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent medical treatment, intelligent security, autonomous driving, safe city, intelligent terminal, etc.

[0093] At present, many scenarios of artificial intelligence follow the following paradigm: as shown in FIG. 2, first, through data, a pre-trained model, and then fine-tune it towards multiple, for example, N downstream tasks, to obtain respective downstream task models for different downstream tasks.

[0094] In order to solve the problems proposed in the background art, the embodiments of the present application provide a model training method, which is mainly explained in three aspects.

[0095] 1. Triplet (used to represent 2 data points and the relationship between the 2 data points) as a general standardized format, uniformly encodes multiple modal data (such as structured table data, visual data, graph data, and time series data, etc.), so that different modal data is unified into a common representation space. This encoding method can break down the barriers between different modal data, so that the model can more effectively extract information, thereby improving the generalization and precision of the model.

[0096] 2. Cross-modal pre-training: based on the result of uniformly encoding multiple modal data by the triplet, pre-training the model to obtain a pre-trained model, so that the model can learn relevant information of different modalities sufficiently, and provide rich knowledge and context information for subsequent downstream tasks.

[0097] 3. Downstream task fine-tuning: when facing specific downstream tasks such as classification, regression, image recognition, etc., based on the result of encoding the data of the downstream task based on the triplets, fine-tune the pre-trained model to obtain a downstream task model. Since rich cross-modal knowledge has been learned in the pre-training stage, the pre-trained model can adapt to new tasks faster and achieve better performance in the fine-tuning stage.

[0098] It should be noted that the pre-trained model obtained by the cross-modal pre-training provided by the embodiments of the present application is not limited to a specific task, but can be widely applied to different fields and tasks. This universality makes the pre-trained model have strong generalization ability. At the same time, since the pre-training stage has fully extracted information in multiple fields, the downstream task model can often provide more accurate and in-depth results when processing specific tasks.

[0099] In summary, by unifying data encoding, model pre-training, and pre-trained model fine-tuning, cross-industry compatibility is achieved, enabling the system to process various industry data and improving universality and portability.

[0100] First, the model management system to which the method provided by the embodiments of the present application can be applied is described. FIG. 3 is a schematic diagram of the architecture of a model management system provided by an embodiment of the present application. As shown in FIG. 3, the system includes a computing device cluster 320 and a terminal 310.

[0101] The terminal 310 can be, but is not limited to, various personal computers, notebook computers, smartphones, tablet computers, and portable wearable devices. The exemplary embodiments of the terminal 310 involved in the present solution include, but are not limited to, electronic devices running iOS, android, Windows, Harmony OS, or other operating systems. The type of electronic device is not specifically limited by the embodiments of the present application.

[0102] The computing device cluster 320 can be configured as a standalone physical server, a server cluster or distributed system composed of multiple physical servers, a cloud server or cloud server cluster providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, etc. Basic cloud computing services, and deployed in several cloud data centers; the software can be an application that implements the object control method, but is not limited to the above forms.

[0103] The terminal 310 and the computing device cluster 320 are connected through a network. The network can be a wired network or a wireless network. For example, the wired network can be a cable network, a fiber network, a digital data network (DDN), etc., and the wireless network can be a telecommunication network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a Zig Bee network, a global system for mobile communications (GSM) network, a code division multiple access (CDMA) network, a general packet radio service (CPRS) network, etc., or any combination thereof. It can be understood that the network can use any known network communication protocol to realize communication between different client layers and gateways. The network communication protocol can be various wired or wireless communication protocols, such as an Ethernet, a universal serial bus (USB), a firewire, a global system for mobile communications (GSM), a general packet radio service (GPRS), a code division multiple access (CDMA), a wideband code division multiple access (WCDMA), a time-division code division multiple access (TD-SCDMA), a long term evolution (LTE), a new radio (NR), a Bluetooth, a wireless fidelity (Wi-Fi), etc.

[0104] In a possible scenario, the computing device cluster 320 can be a cloud (a software platform using application virtualization technology, integrating functions such as software search, download, use, management, backup, and the like); in specific use, the computing device cluster 320 can be configured as a cloud management platform 321 and a data center 322, and the terminal 310 and the cloud interact through the cloud management platform 322; in addition, the data center 322 can deploy nodes, and the nodes in the data center 322 can be virtual machine instances, container instances, physical servers, and the like; in the data center 322, the number of nodes is generally massive. The cloud management platform 321 can be independently deployed on a server or a virtual machine in the data center 322, or can be distributedly deployed on multiple servers in the data center 322 or distributedly deployed on multiple virtual machines in the data center 322, or a part of the cloud management platform 321 can be independently or distributedly deployed in a device in an edge environment (also referred to as an edge device), and the other part of the cloud management platform 321 can be independently or distributedly deployed in the data center 322. The edge environment is an environment that is relatively close to the user's terminal computing device in geographical position, and the edge environment includes edge devices, for example, an edge server, an edge small station with computing capability, and the like.

[0105] In an embodiment of the present application, the node can be used for training of the model; in a possible case, when the model is large in size, the model can be distributedly deployed on multiple nodes based on the model parallelism idea for parallel training. In another possible case, the model can be trained on 1 node.

[0106] In a possible scenario, the model training scheme provided in the embodiment of the present application can be implemented through software. Exemplarily, the software can be online model training software or a service. In addition, the software has a client and a server, the terminal 310 runs the client of the software, and the computing device cluster 320 runs the server of the software; subsequently, in the process of running the client of the software by the terminal 310, the server running on the computing device cluster 320 can be called to train the model.

[0107] In addition, for development of the model, a developer can install an AI development framework on the terminal 310, and then develop an AI model locally, or use an AI development framework to develop an AI model on an online platform (for example, an online open-source framework platform, a public cloud AI basic development platform, and the like). AI development frameworks in the industry are generally open source, and typical AI development frameworks used for development of deep student models, also referred to as deep learning frameworks, include PaddlePaddle, Tensorflow, Caffe, Theano, MXNet, Torch, and PyTorch, and the like.

[0108] Next, in combination with the model training system provided above, a model training method provided by an embodiment of the present application is described in detail.

[0109] FIG. 4 is a flowchart of the model training method provided by an embodiment of the present application. The embodiment can be applied on an electronic device, and specifically can be applied on a server or a general computer.

[0110] As shown in FIG. 4, the model training method provided by an embodiment of the present application at least includes the following steps:

[0111] In step 401, the computing device cluster 320 determines a first triple sample corresponding to each modality in the plurality of modalities, the first triple sample being obtained by encoding the sample corresponding to the modality in a triple manner; the triple is composed of a first data point, a second data point, and a relationship between the first data point and the second data point; and the first triple sample is a plurality of instances of the triple arranged in sequence.

[0112] First, the data of the plurality of modalities needs to be collected and preprocessed to obtain the sample of each modality in the plurality of modalities. In the embodiment of the present application, the data preprocessing can include one or more operations such as data selection, data enhancement, data augmentation, data cleaning, clustering analysis, and feature analysis. Exemplarily, the plurality of modalities can be structured data, images, graph data, and sequence data. The sequence data is data arranged in sequence, such as text, and for example, time series data, which can be voice data, temperature data, etc. Exemplarily, the voice data can be a voice signal after data preprocessing, and the data preprocessing includes but is not limited to voice noise reduction, reverberation elimination, and voice enhancement. Exemplarily, the voice data can be voice features after data preprocessing, and the data preprocessing includes but is not limited to voice noise reduction, reverberation elimination, voice enhancement, windowing and framing, and feature extraction (extracting effective information in the voice signal).

[0113] In some embodiments, to train the model, the data of the plurality of modalities needs to be collected and preprocessed to obtain a plurality of samples (generally a large amount) of each modality in the plurality of modalities. In the embodiment of the present application, for each modality, data in the field can be used. For the data of each modality, the user can pre-collect the data according to the actual application scenario, use open source data formed by the industry, or use the data pre-collected according to the actual application scenario and the open source data. For the case of pre-collection by the user according to the actual application scenario, the user can purchase an object storage service (OBS) in the cloud management platform 321, which is an object-based cloud storage service, and the user can store the data in the OBS.

[0114] Then, for each sample of each modality, the sample is data encoded by a triple to obtain a first triple sample, the first triple sample being a plurality of instances of triples arranged in sequence. It should be noted that the instance of the triple is an example of the triple, that is, the real data is filled in the triple. The triple is composed of a first data point, a second data point, and a relationship between the first data point and the second data point, as shown in FIG. 5, which can be denoted as [x i , R ij , x j ], wherein x i may be the first data point, x j may be the second data point, and R ij represents a relationship representation between the first data point x i and the second data point x i .

[0115] For structured data, the field values of adjacent fields in the structured data are located in the same instance of the triple; correspondingly, the first data point x i and the second data point x j are different fields in the structured data, and R ij is the position of the first data point x i in the structured data, which is incremented from 1. Exemplarily, the structured data can include a field sequence and a plurality of records under the field sequence, the field sequence being used to sequentially record a plurality of fields, and each record under the field sequence being used to explain the field values of each field in the field sequence. For each record, the record is data encoded to obtain a first triple sample. Exemplarily, the record is [x1, x2, …, x n ], wherein x1 records the field value in the key-value manner, the key is the field, and the value is the field value of the field. After the record is data encoded by the triple, (n-1) triples can be obtained: [x1, 1, x2], [x2, 2, x3], …, [x n-1 , n, x n ]. According to the order of x1, x2, …, x n , the (n-1) triples are combined to obtain the first triple sample [x1, 1, x2][x2, 2, x3]…[x n-1 , n, x n ].

[0116] For an image, adjacent pixel values in the same row and / or the same column of the image are located in the same instance of the triple; correspondingly, the first data point x i is any pixel point in the image, the second data point x j is the first data point x iAny pixel point in the periphery, R ij is the first data point x i The position in the image is incremented from 1. Exemplarily, the image can be represented by the following matrix:

[0117] In the embodiment of the present application, for the image, several sampling strategies can be flexibly designed according to actual needs, and the sampling strategy indicates the way of reflecting the position relationship of the pixel points in the image and traversing each pixel point in the image.

[0118] In one example, the sampling strategy can be sequentially sampling from the first pixel value at the beginning of the first row to the end of the row, then continuing from the first pixel value at the beginning of the row and the next row, sequentially sampling from the beginning of the second row, until traversing each row in the image; for the pixel value in the first row of the image, according to the way of data encoding by triple record, starting from the first pixel value at the beginning of the first row, data encoding is performed, the beginning of the first row and the beginning of the second row are connected, data encoding is continued from the first pixel value at the beginning of the second row, and the cycle is repeated until the last row is processed, to obtain the first triple sample, exemplarily, the first triple sample is as follows: [x 11 , 1, x 12 ], …, [x 1n-1 , 1n-1, x 1n ], [x 11 , 1n, x 21 ], [x 21 , 1n+1, x 22 ], …, [x (m-1)1 , (m-1)n+1, x m+1 ], …, [x mn-1 , mn-1, x mn ].

[0119] In another example, the sampling strategy can be sequentially sampling from the first pixel value at the beginning of the first column to the end of the column, then continuing from the first pixel value at the beginning of the column and the next column, sequentially sampling from the beginning of the second column, until traversing each column in the image; for the pixel value in the first column of the image, according to the way of data encoding by triple record, starting from the first pixel value at the beginning of the first column, data encoding is performed, the beginning of the first column and the beginning of the second column are connected, data encoding is continued from the first pixel value at the beginning of the second column, to obtain the first triple sample, exemplarily, the first triple sample is as follows: [x 11 , 1, x 12 ], …, [x (m-1)1 , m-1, x m1 ], [x 11 , 1n, x 12 ], [x 12 , 1n+1, x22 ], …, [x 1n , (m-1)n+1, x 2n ], …, [x (m-1)n , mn-1, x mn ].

[0120] For graph data, two nodes connected in the graph data are in the same instance of the triple; correspondingly, the first data point x i is any node in the graph data, the second data point x j is the first data point x i connected node, R ij is the weight of the edge between the first data point x i and the second data point x j , or R ij is the position of the first data point x i .

[0121] In the embodiment of the application, as shown in FIG. 6, the graph data can include a plurality of nodes connected between the plurality of nodes, the plurality of nodes can be divided into root nodes, child nodes and leaf nodes, the child nodes are nodes between the root nodes and the leaf nodes, the link can be formed between the root nodes and the leaf nodes, and the child nodes in the link can have branch links.

[0122] For graph data, several sampling strategies can be flexibly designed in combination with actual needs.

[0123] In one example, the sampling strategy can be to convert the graph data into links without branch links, and encode each link; at this time, R ij may be the weight of the edge between the first data point x i and the second data point x j , or R ij may be the position of the first data point x i in the link, which is incremented from 1; then for each link, data encoding is performed from the first node in the link in the manner of data encoding through triple record, to obtain the first triple sample.

[0124] In one example, the sampling strategy can be to convert the graph data into links with branch links, and encode each link; at this time, R ij is the position of the first data point x i in the link, which is incremented from 1; then for each link, data encoding is performed from the first node in the link in the manner of data encoding through triple record, to obtain the first triple sample. At this time, R ijThe encoding mode of the first triple sample can be various. In one example, the positions of the nodes in the main link are sequentially increased first, and then the positions of the nodes in the branch link are sequentially increased according to the positions of the nodes in the main link from small to large. Exemplarily, for the follow node, the node 21, the node 31, the node 32, the node 41, the node 42 and the node 43 in FIG. 6, the first triple sample is [follow node, 1, node 21] [node 21, 2, node 31] [node 31, 3, node 41] [node 21, 4, node 32] [node 31, 5, node 42] [node 31, 6, node 43]. In another example, the positions of the nodes in the link are sequentially increased first, and then the positions of the nodes in the branch link are sequentially increased, and then the positions of the nodes in the branch link are sequentially increased from the starting node of the branch link. Exemplarily, for the follow node, the node 21, the node 31, the node 32, the node 41, the node 42 and the node 43 in FIG. 6, the first triple sample is [follow node, 1, node 21] [node 21, 2, node 31] [node 21, 3, node 32] [node 31, 4, node 41] [node 31, 5, node 42] [node 31, 6, node 43].

[0125] For the sequence data, two adjacent objects in the sequence data are located in the same instance of the triple; correspondingly, the first data point x i For any object in the sequence data, the second data point x j is the first data point x i , and R ij is the first data point x i in the sequence data, and is sequentially increased from 1. In specific implementation, the objects in the sequence data are encoded according to the mode of encoding the data in the record by the triple, to obtain the first triple sample.

[0126] It should be noted that, in some possible implementation, the data preprocessing and the data encoding of the triple can be implemented by the cloud management platform 321 and the data center 322. In specific implementation, the terminal 310 sends a data processing task to the computing device cluster 320, the data processing task indicating the position of the first data set and the position of the second data set, the first data set being used to store data of multiple modalities, and the second data set being used to store the first triple sample.

[0127] In some possible implementations, the cloud management platform 321 provides a configuration interface; the configuration interface can be an interface for accessing an AI basic development platform. The AI basic development platform is a one-stop AI development platform for developers, and can provide various capabilities in the whole process of AI development. Exemplarily, the capabilities provided by the AI basic development platform can include the following six parts: data preprocessing, model construction and training, model management, model deployment, data optimization, and model optimization and updating. The various capabilities in the AI basic development platform can be integrated for use by a user in the whole process of AI, or can be provided independently for the user. Subsequently, the terminal 310 accesses the configuration interface, obtains the operation of the user on the configuration interface, and configures a data processing task. Subsequently, the data preprocessing function of the AI basic development platform reads the data of multiple modalities from the first data set, and performs data preprocessing and data encoding in a triple mode on the data of multiple modalities, to obtain the first triple sample corresponding to each of the multiple modalities, and store the first triple sample in a second data set. Subsequently, the computing device cluster 320 can directly obtain the first triple sample from the second data set.

[0128] In some embodiments, the user can pre-purchase an object storage service (OBS) in the cloud management platform 321, which is an object-based cloud storage service. The user can store the first data set in the OBS, and then read the data of multiple modalities in the first data set from the OBS when using the data preprocessing function provided by the AI basic development platform. The data can be subjected to one or more operations such as data selection, data labeling, data enhancement, data augmentation, data cleaning, clustering analysis, and feature analysis according to requirements, and then subjected to data encoding in a triple mode to obtain the first triple sample corresponding to each of the multiple modalities.

[0129] Step 402: The computing device cluster 320 trains a model based on the first triple sample corresponding to each of the modalities, to obtain a trained model.

[0130] In the embodiments of the present application, the model is pre-trained by using self-supervised learning in combination with the first triple sample corresponding to multiple modalities, so that the model can fully understand the meaning of the data of multiple modalities, thereby providing rich knowledge and context information for subsequent downstream tasks.

[0131] In the embodiment of the present application, the manner of training the self-supervised learning model can be: for each modality, changing the target element in the first triple sample corresponding to the modality to determine the changed first triple sample; updating the model parameters of the model based on the changed first triple sample and the target element. Here, the model combines data of multiple modalities to express elements, so that the element expression has rich knowledge content. For example, the changing manner can be mask processing and / or adding noise. For example, the target element of the perturbation processing can be several data points in the first triple sample, and / or the relationship between the data points.

[0132] In a specific implementation, for each modality, the target element in the first triple sample corresponding to the modality is changed to determine the vectorization result (for the sake of description and distinction, it can be referred to as the first vectorization result) of the changed first triple sample, the first vectorization result is the vectorization representation (for the sake of description and distinction, it can be referred to as the first vectorization representation) of each instance in the changed first triple sample, the first vectorization representation of the instance is determined by the vectorization representation (for the sake of description and distinction, it can be referred to as the second vectorization representation) of each element in the instance, the dimension of the second vectorization representation of each element is the same, such as 718, and the dimension of the first vectorization representation of each instance in the first triple sample is the same; the first vectorization result is input into the model to obtain the output of the model, the dimension of the model output is the same as the dimension of the first vectorization result, which is the vectorization representation (for the sake of description and distinction, it can be referred to as the fourth vectorization representation) of each instance in the first triple sample; based on the output of the model and the vectorization representation (for the sake of description and distinction, it can be referred to as the third vectorization representation) of the target element, the representation error of the target element is determined, and the model parameters of the model are updated based on the identification error of the target element, such as, for the purpose of minimizing the representation error of the target element, the model parameters of the model are updated.

[0133] wherein the first vectorization representation of the instance is the result of fusion processing of the second vectorization representation of each element in the instance; wherein the fusion processing can be addition, such as, as shown in FIG. 7, the vectorization representation of x i is v i , the vectorization representation of x ij is v ij , the vectorization representation of x j is v j , then the vectorization representation V i of the triple is v i +v ij +v jIn some possible implementation manners, the manner of determining the vectorization of the changed first triple sample can be: vectorizing each element in the changed first triple sample according to a fixed dimension to obtain a second vectorization representation of each element, then performing fusion processing on the second vectorization representations of the elements in the same instance to obtain a first vectorization representation of the instance, replacing the instance in the changed first triple sample with the first vectorization representation of the instance to obtain a first vectorization result. The above manner is merely an example and does not constitute a specific limitation. In another possible implementation manner, for the first triple sample, each element in the first triple sample is vectorized according to a fixed dimension to obtain a vectorization representation of each element, then the vectorization representations of the elements in the same instance are fused to obtain a vectorization representation of the instance, the instance in the first triple sample is replaced with the vectorization representation of the instance to obtain a vectorization result of the first triple sample; the target element in the first triple sample is changed, the first vectorization representation of the instance in which the changed target element is located is determined, the vectorization representation of the instance in which the target element is located in the vectorization result of the first triple sample is replaced with the first vectorization representation of the instance in which the changed target element is located, and a first vectorization result of the changed first triple sample is determined.

[0134] It should be noted that, in the process of vectorization, for the first data point and the second data point in the triple, the vectorization can be implemented by a linear layer, and for the position in the triple, a position vectorization table (used to record the vectorization representation of each position) can be queried to determine the vectorization representation of each position. It should be noted that the vectorization of the first triple sample can map data of multiple modalities to points in a high-dimensional space, and the relative positional relationship between the points can reflect the inherent relationship between the original data.

[0135] In the embodiment of the application, the manner of inputting the first vectorization result of the changed first triple sample to the encoder is: dividing the first vectorization result of the changed first triple sample according to a fixed size to obtain a plurality of vectorization results (for the sake of description and distinction, which can be referred to as second vectorization results) arranged in sequence, and inputting the plurality of second vectorization results arranged in sequence to the hidden nodes in the encoder. For example, as shown in FIGS. 8a to 8c, the elements in the first triple sample are perturbed or masked and then vectorized to obtain the first vectorization result: V1V2…V o V o+1 …V 2o …V (n-1)o+1 …V n0 , divided according to o, to obtain n vectorization results: V1V2…V o , V o+1 …V 2o , …, V(n-1)o+1 …V n0 In some possible implementation manners, the model is an encoder, the encoder is a single hidden layer, or multiple hidden layers are stacked. The hidden layer includes n hidden nodes respectively corresponding to n time points starting from 1 and increasing to n, and the n hidden nodes are sequentially connected. In the case of stacking multiple hidden layers, each hidden node also needs to be connected to a hidden node at the same position in an adjacent hidden layer. Illustratively, the hidden node can be a node in a recurrent neural network (RNN). Illustratively, the hidden node can be a node in a long short-term memory (LSTM). Illustratively, the hidden node can be a node in a gated recurrent unit (GRU). Embodiments of the present application are not intended to limit the structure of the encoder in any way. For example, it can be a recurrent neural network (RNN), a long short-term memory (LSTM), and a gated recurrent unit (GRU). For another example, it can include a neural network formed by stacking multiple hidden layers.

[0136] In some possible implementation manners, the target element is a first data point and / or a second data point in an instance, and the manner of determining the representation error of the target element is as follows:

[0137] determining a fourth vectorized representation of an instance in which the target element is located in the output of the model; performing inverse operation of the fusion operation based on the second vectorized representation of the other elements in the instance in which the target element is located and the fourth vectorized representation, to remove the content of the other elements in the instance in which the target element is located in the fourth vectorized representation, and obtain a fifth vectorized representation of the target element; and determining the representation error of the target element based on the third vectorized representation of the target element and the fifth vectorized representation.

[0138] Illustratively, as shown in FIG. 8a, it is assumed that the target element x i is disturbed or masked, the vectorized representation of x i is v i , the vectorized representation of x ij after disturbance or masking is v ij , the vectorized representation of x j is v j , and the vectorized representation of the triplet in which x i is located is v R o , v o+1 , …, v 2o , …, v (n-1)o+1 , …, v n0 are input into the model, and the output y1y 2… of the model is obtained.2… y n , y2 is y o+1 …y 2o , assuming i = o + 1, the vectorized representation of the target element x i is y i = y o+1 -v ij -v j , the representation error of the target element x i is y i -v i .

[0139] Exemplarily, as shown in FIG. 8b, assuming that the target elements x i , x j are disturbed or masked, the vectorized representation of x i is v i , the vectorized representation of x i after being disturbed or masked is x j vectorized representation is v j , the vectorized representation of x j after being disturbed or masked is R ij vectorized representation is v ij , the vectorized representation of x i is assuming i = o + 1, the vectorized representation of the target element x i is y i = y o+1 -v ij -v j , the representation error of the target element x i is y i -v i , the vectorized representation of the target element x j is y j = y o+1 -v ij -v i , the representation error of the target element x j is y j -v j .

[0140] In some possible implementation manners, the target elements are each element in the instance, and the manner of determining the representation error of the target element is:

[0141] determining a fourth vectorized representation of the target element in the output of the model; and determining an error between the output of the model and the third vectorized representation of the target element based on the third vectorized representation and the fourth vectorized representation of the target element.

[0142] Exemplarily, as shown in FIG. 8c, it is assumed that the target element x i , R ij , x j is disturbed or masked, the vectorization of x i is represented as v i , the vectorization of x i after being disturbed or masked is represented as x j , the vectorization of R j is represented as v j , the vectorization of R ij after being disturbed or masked is represented as R ij , the vectorization of R ij after being disturbed or masked is represented as , then the vectorization of the triad in which x i is located is represented as Assuming i = o + 1, the representation error of V o+1 is y o+1 - V o+1 .

[0143] In a specific implementation, the computing device cluster 320 selects a first triad sample corresponding to any one of the modalities to iteratively train the model until a training completion condition is reached. The training completion condition refers to a condition for stopping the training of the model, including but not limited to that the number of training iterations reaches a maximum number of iterations, the model loss information reaches a preset threshold, and the model parameters no longer change. Specifically, there are multiple first triad samples corresponding to the modality, and for each first triad sample of the modality, the computing device cluster 320 inputs the first vectorization result of the first triad sample in which the target element is changed into the model to obtain the output of the model, determines the representation error of the changed target element in the first triad sample based on the output of the model and the third vectorization representation of the target element, updates the model parameters of the model based on the representation error of the changed target element in each first triad sample to obtain an updated model, and then replaces the previous model with the updated model for the next iteration until the training completion condition is reached. After the training completion condition is reached, the first triad sample corresponding to another modality is selected to continue to iteratively train the model, and the process is repeated until all modal data is traversed, and the model at the time when the training completion condition is reached for the last time is taken as the pre-trained model.

[0144] In an actual use scenario, the cloud management platform 321 provides a configuration interface; the terminal 310 accesses the configuration interface, obtains an operation of a user on the configuration interface, configures a model training task, and the model training task can indicate a location of a data set, a location of a model, and configuration information of a computing resource; the data set can be used to store first triple samples corresponding to each of a plurality of modalities. In an example, the user can store the data set and the model in an OBS. The configuration information of the computing resource can include a number of nodes, a processor type, and a processor specification. For example, the processor type can be a CPU or a GPU; when the processor type is a CPU, the processor specification indicates a number of cores and a size of memory of the CPU; and when the processor type is a GPU, the processor specification indicates a size of a display memory of the GPU, a number of cores, and a size of memory of the CPU. In a possible scenario, when the number of nodes in the model training task is 1, it indicates that one node is used for model training; and when the number of nodes is a plurality, a plurality of nodes are used for training the model, and distributed training is used for model training. Currently, there are two main ideas for distributed training. One idea is model parallelism, that is, a model is divided into a plurality of small models, which are placed on different devices, and each device runs a part of the model. Because the parts of the model are closely related, this way is very inefficient and requires frequent communication between different devices. Another idea is data parallelism, that is, a complete model is on each machine, but the data is divided into multiple parts for each model, and each model inputs different data for training.

[0145] The terminal 310 can send a model training task to the cloud management platform 321. The cloud management platform 321 determines a training node and a training task of the training node based on the model training task, and the training task is used to indicate a location of the first triple sample and a location of the model; and the cloud management platform 321 issues the training task to the training node. For the training node, the training node deploys the model based on the training task, determines the first triple sample corresponding to each of the plurality of modalities, and trains the model based on the first triple sample corresponding to each of the plurality of modalities.

[0146] In step 403, the computing device cluster 320 determines second triple samples corresponding to a target modality in the plurality of modalities, and the second triple samples are obtained by encoding samples corresponding to the target modality according to triples; and the target modality is at least one of the plurality of modalities.

[0147] Firstly, data of a target modality needs to be collected and preprocessed to obtain a plurality of samples (generally a small number) of each modality in the target modality and labels of the samples. In the embodiment of the present application, data of a specific field is used for the data of the target modality. Then, the samples are encoded in a triple mode to obtain second triple samples corresponding to each modality in the target modality. It should be noted that the target modality can be a single modality or multiple modalities. For example, the target modality can be text and image; for example, the target modality can be text, image, and voice. The data preprocessing can include one or more operations such as data selection, data labeling, data enhancement, data augmentation, data cleaning, cluster analysis, and feature analysis.

[0148] In some possible implementations, the data preprocessing can be implemented by the cloud management platform 321 and the data center 322. In a specific implementation, the terminal 310 sends a data processing task to the computing device cluster 320, the data processing task indicating a location of a third data set and a location of a fourth data set, the third data set being used to store data of a target modality, and the fourth data set being used to store second triple samples. The third data set can be pre-collected by a user according to an actual application scenario. Subsequently, the data of the target modality is read from the third data set by using a data preprocessing function (including data labeling) of the AI basic development platform, the data of the target modality is preprocessed and encoded in a triple mode to obtain second triple samples corresponding to each modality in the target modality, and the second triple samples are stored in the fourth data set. Subsequently, the computing device cluster 320 can directly obtain the second triple samples from the fourth data set.

[0149] In some embodiments, the user can pre-purchase an object storage service (OBS) in the cloud management platform 321, which is an object-based cloud storage service. The user can store the third data set in the OBS, and then read the data of the target modality in the third data set from the OBS when using the data preprocessing function provided by the AI basic development platform. The data can be selected, labeled, enhanced, augmented, cleaned, analyzed, and analyzed according to the requirements of one or more operations, and then encoded in a triple mode to obtain second triple samples corresponding to each modality in the target modality.

[0150] The label is determined by labeling. In one possible implementation in this case, the AI basic development platform can have a labeling tool built-in for manual labeling by the user. Different labeling tasks can be labeled in different ways. In addition, after a certain number of samples are manually labeled, the AI basic development platform can automatically train the built-in pre-labeling model based on the manually labeled samples. After training to a certain accuracy, the pre-labeling model can be used to infer the remaining unlabeled samples (i.e., intelligent labeling), and the labeling results can be evaluated. The intelligent labeling sample pairs with high accuracy can be directly stored in a data set such as OBS for subsequent model training. For difficult examples with low accuracy, the user can manually confirm or correct the results in the form of result display, and store the manually confirmed samples in a data set such as OBS.

[0151] In step 404, the computing device cluster 320 trains the trained model based on the second triple sample corresponding to the target modality to obtain a downstream task model.

[0152] In the embodiment of the application, the pre-trained model is the model trained in step 402, which is a pre-trained model. The pre-trained model is fine-tuned by supervised learning combined with data of a specific modality, so that the model quickly adapts to new tasks based on the rich knowledge and context information learned by pre-training.

[0153] The way of fine-tuning the model by supervised learning combined with data of a specific modality can be: iteratively training the pre-trained model and the task model based on the second triple sample corresponding to the target modality until a training completion condition is reached. The training completion condition refers to the condition for stopping model training, including but not limited to the number of training iterations reaching the maximum number of iterations, the model loss information reaching a preset threshold, and the model parameters no longer changing. The task model can be a multi-modal deep learning model (Contrastive Language-Image Pre-training, CLIP). The CLIP model proposes a contrastive learning paradigm that integrates text and image data. By using text information as a weak supervision signal, it is used to supervise related visual task training. The task model can be Audio CLIP, which applies CLIP to the speech field, integrates information of three modalities of text, video and audio, and learns in the CLIP manner.

[0154] Specifically, the computing device cluster 320, for each second triple sample corresponding to the target modality, vectorizes the second triple sample in the manner described in step 402, inputs the vectorization result of the second triple sample into the pre-training model to obtain the output of the pre-training model, inputs the output of the pre-training model into the task model, the task model predicts the label of the second triple sample to obtain a prediction result, and trains the pre-training model and the task model based on the error between the prediction result of each second triple sample and the label of each second triple sample to obtain an updated pre-training model and task model. Then, the updated pre-training model and task model replace the previous pre-training model and task model, and the next iteration is performed until the training completion condition is reached, and the pre-training model and task model when the training completion condition is reached are used as the downstream task model.

[0155] After obtaining the downstream task model, the downstream task model needs to be deployed. The downstream task model usage scheme provided by the embodiments of the present application can be applied to the terminal 310 described above, and can also be applied to the computing device cluster 320. In specific implementation, the downstream task model usage scheme can be run in the terminal 310 or the computing device cluster 320 in a software manner, for example, the software can be a service or an application program. In addition, the technical scheme provided by the embodiments of the present application can be used in many general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers (PCs), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The embodiments of the present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The embodiments of the present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0156] In some possible scenarios, the computing device cluster 320 deploys a cloud management platform 321 and a data center 322, the data center 321 can deploy nodes, and the nodes can be used to deploy downstream task models. The deployed downstream task model can be referred to as an AI application or a part of an AI application. A user can access the AI application online through a web page or through a client app, and when the AI application is used, the deployed downstream task model in the node can be invoked through online calling to provide a response. Thus, the cloud management platform and the downstream task model can be used to implement inference on online request data and return an inference result. In the process of providing online services by using the downstream task model, the cloud management platform can charge according to the number of invocations of the downstream task model, or charge according to the resource consumption of the inference of the downstream task model.

[0157] In a possible scenario, the model deployment scheme provided by the embodiment of the application can be implemented by software. Exemplarily, the software can be online model deployment software or a service. In addition, the software has a client and a server, the terminal 310 runs the client of the software, and the computing device cluster 320 runs the server of the software; subsequently, the terminal 310 can invoke the server running on the computing device cluster 320 in the process of running the client of the software, and deploy the downstream task model on the computing device cluster 320.

[0158] Exemplarily, the computing device cluster 320 includes an execution device 322 and a training device 321; referring to FIG. 9a, the embodiment of the application provides a system architecture.

[0159] The data acquisition device 910 is configured to acquire task data (generally, original data) under a downstream task and store the task data in a database 920. The terminal 310 can process the task data in the database 920 to obtain a plurality of second triple samples and store the second triple samples in the database 920. The training device 321 can obtain the plurality of second triple samples from the database 920, train a pre-training model and a task model, and obtain a downstream task model 301. Details are described above in the description of steps 403 and 404.

[0160] The downstream task model 301 obtained by the training device 321 can be applied in different systems or devices, for example, as shown in FIG. 9a, the downstream task model 301 can be deployed in the execution device 322, and in some other possible scenarios, the downstream task model 301 can also be deployed in the terminal 310.

[0161] The execution device 322 is configured with an I / O interface 3222 to interact with external devices, and a “user” can input data to the I / O interface 3222 through the terminal 310.

[0162] The execution device 322 can invoke data, code, etc. in the data storage system 930, or store data, instructions, etc. in the data storage system 930.

[0163] The computing module 4221 uses the downstream task model 301 to infer the input data to obtain a prediction result. Specifically, the prediction result can be determined according to the downstream task. Subsequently, the computing module 4221 can provide the prediction result as an output result.

[0164] Finally, the I / O interface 3222 returns the output result to the terminal 310 and provides it to the user.

[0165] More deeply, the training device 321 can generate a corresponding downstream task model 301 based on multiple triple samples under different downstream tasks for different downstream tasks, to provide better results for the user.

[0166] In the case shown in FIG. 9a, the user can manually specify the data input into the execution device 322, for example, by operating in the interface provided by the I / O interface 3222. In another case, the terminal 310 can automatically input data into the I / O interface 3222 and obtain an output result. If the terminal 310 needs to obtain authorization from the user for automatic data input, the user can set the corresponding permissions in the terminal 310. The user can view the output result of the execution device 322 in the terminal 310, and the specific presentation form can be a specific way such as display, sound, action, etc. The terminal 310 can also act as a data collection end to store the collected task data into the database 920.

[0167] It is worth noting that FIG. 9a is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the diagram does not constitute any limitation. For example, in FIG. 9a, the data storage system 930 is an external memory relative to the execution device 322, and in other cases, the data storage system 930 can also be placed in the execution device 322.

[0168] Exemplarily, the computing device cluster 320 includes the execution device 322, and the terminal 310 includes the terminal 311 and the terminal 312; see FIG. 9b, an embodiment of the present application provides a system architecture.

[0169] The execution device 322 is implemented by one or more servers, and can be optionally combined with other computing devices such as data storage, routers, load balancers, etc. The execution device 322 can be arranged on one physical site or distributed on multiple physical sites. In an embodiment of the present application, each server in the execution device 322 can be deployed with a downstream task model.

[0170] A user can operate a respective local device (e.g., terminal 311 and terminal 312) to interact with the execution device 322. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet, a smart camera, a smart car, or other types of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0171] Each local device of a user can interact with the execution device 322 through a communication network of any communication mechanism / communication standard, which can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0172] In one implementation, one or more aspects of the execution device 322 can be implemented by each local device. For example, the execution device 322 can deploy a downstream task model, the terminal 311 can provide local data to the execution device 322, and the execution device 322 feeds back the output result of the downstream task model to the terminal 311; subsequently, the execution device 322 can update the model parameters of the downstream task model according to the update of the terminal 311 on the output result of the downstream task model; or, the terminal 311 can deploy a downstream task model and feed back the output result of the downstream task model to the execution device 322; the execution device 322 can update the model parameters of the downstream task model according to the output result of the downstream task model.

[0173] It is noted that all functions of the execution device 322 can also be implemented by the local device. For example, the terminal 311 can deploy a downstream task model and provide inference service for its own user, or provide inference service for the user of the other terminal 312.

[0174] Exemplarily, FIG. 10 is a flow diagram of a downstream task application method provided by an embodiment of the present application. In combination with FIG. 9a and FIG. 9b, the downstream task model is deployed in the computing device cluster 320. As shown in FIG. 10, the downstream task application method provided by an embodiment of the present application at least includes the following steps:

[0175] Step 1001, the terminal 310 sends data of a downstream task to the computing device cluster 320.

[0176] In some possible scenarios, the data of the downstream task can be an image, a text, a graph data, a sequence data, etc.

[0177] In some possible scenarios, the terminal 310 can install software of the downstream task, which can call the downstream task model deployed in the computing device cluster 320. The terminal 310 can collect data of the downstream task according to the data requirement of the downstream task, and send the data to the computing device cluster 320.

[0178] At step 1002, the computing device cluster 320 encodes the data of the downstream task in a manner of triplets to obtain a triplet encoding result, inputs the triplet encoding result into the downstream task model, and obtains an output result of the downstream task model.

[0179] At step 1003, the computing device cluster 320 sends the output result of the downstream task model to the terminal 310.

[0180] As shown in FIG. 11, the embodiment of the present application further provides a model training apparatus, comprising:

[0181] The encoding module is configured to determine a first triplet sample corresponding to each modality in the plurality of modalities, wherein the triplet sample is obtained by encoding a sample in a manner of triplets; the triplet is composed of a first data point, a second data point, and a relationship between the first data point and the second data point; and the triplet sample is a plurality of instances of triplets arranged in sequence.

[0182] The training module is configured to train the model based on the first triplet sample corresponding to each modality to obtain a trained model.

[0183] The encoding module and the training module can be implemented by software or by hardware. For example, the implementation of the encoding module is described below. Similarly, the implementation of the training module can refer to the implementation of the encoding module.

[0184] As an example of a software functional unit, the encoding module can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the encoding module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0185] Likewise, the plurality of hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, usually one VPC is set in one region, and communication between two VPCs in the same region and between VPCs in different regions needs to set a communication gateway in each VPC to realize the interconnection between VPCs through the communication gateway.

[0186] As an example of a hardware functional unit, the encoding module can include at least one computing device, such as a server, etc. Alternatively, the encoding module can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. Among them, the above-mentioned PLD can be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0187] The plurality of computing devices included in the encoding module can be distributed in the same region or in different regions. The plurality of computing devices included in the encoding module can be distributed in the same AZ or in different AZs. Likewise, the plurality of computing devices included in the encoding module can be distributed in the same VPC or in multiple VPCs. Among them, the plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, etc.

[0188] It should be noted that in other embodiments, the encoding module can be used to perform any step of the model training method, the training module can be used to perform any step of the model training method, and the steps responsible for the encoding module and the training module can be specified as needed. The encoding module and the training module respectively implement different steps in the model training method to realize the entire function of the model training device.

[0189] As shown in FIG. 12, the present application also provides a computing device 1200. The computing device includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other through the bus 1202. It should be understood that the present application does not limit the number of processors, memories in the computing device 1200.

[0190] The bus 1202 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus 1204 can include a path for transmitting information between various components (e.g., the memory 1206, the processor 1204, the communication interface 1208) of the computing device 1200.

[0191] The processor 1204 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0192] The memory 1206 can include a volatile memory (e.g., a random access memory (RAM)) and a non-volatile memory (e.g., a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)). The memory 1206 stores executable program codes, and the processor 1204 executes the executable program codes to implement the above-mentioned model training method. Specifically, the memory 1206 stores instructions for executing the model training method by the model training apparatus.

[0193] The communication interface 1203 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, etc. to realize the communication between the computing device 1200 and other devices or communication networks.

[0194] The embodiments of the present application also provide a computing device cluster. As shown in FIG. 13, the computing device cluster comprises at least one computing device 1200. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0195] In some possible implementations, one or more computing devices 1200 in the computing device cluster can also be used to execute part of the instructions in the model training apparatus for executing the model training apparatus. In other words, the combination of one or more computing devices 1200 can collectively execute the instructions in the model training apparatus for executing the model training method.

[0196] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster can also respectively store part of the instructions for executing the model training apparatus. In other words, the combination of one or more computing devices 100 can collectively execute the instructions for executing the model training method.

[0197] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 14 shows a possible implementation. As shown in FIG. 14, two computing devices 1200A and 1200B are connected through a network. Specifically, the communication interface in each computing device is connected to the network. In this type of possible implementation, the memory 106 in the computing device 1200A stores instructions for executing the functions of the encoding module. At the same time, the memory 106 in the computing device 1200B stores instructions for executing the functions of the training module.

[0198] The connection mode between the computing device cluster shown in FIG. 14 can be that, considering that the model training method provided by the present application needs to encode a large amount of data into triplets, the functions implemented by the training module are executed by the computing device 1200A.

[0199] It should be understood that the functions of the computing device 1200A shown in FIG. 14 can also be completed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B can also be completed by multiple computing devices 1200.

[0200] The embodiment of the present application further provides a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be accessed by a computing device including one or more available media. The available media can be a magnetic media (for example, a floppy disk, a hard disk, a magnetic tape), an optical media (for example, a DVD), or a semiconductor media (for example, a solid state disk) and the like. The computer readable storage medium includes instructions that instruct the computing device to execute the above-mentioned model training method applied to the model training device.

[0201] The embodiment of the present application further provides a computer program product containing instructions. The computer program product can be software or a program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to execute the above-mentioned model training method.

[0202] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that, The method includes: A first triplet sample is determined for each mode in multiple modalities. The first triplet sample is obtained by encoding the sample corresponding to the mode according to the triplet method. The triplet consists of a first data point, a second data point, and the relationship between the first data point and the second data point. The first triplet sample is multiple instances of the triplet arranged in order. Based on the first triplet sample corresponding to each modality, the model is trained to obtain the trained model.

2. The method according to claim 1, characterized in that, The training of the model based on the first triplet samples corresponding to each modality includes: For each modality, the target element in the first triplet sample corresponding to the modality is modified to determine the modified first triplet sample; based on the modified first triplet sample and the target element, the model parameters of the model are updated.

3. The method according to claim 2, characterized in that, The changes are made by masking or adding noise.

4. The method according to claim 2 or 3, characterized in that, The step of updating the model parameters based on the modified first triplet sample and the target element includes: Determine the vectorization result, which indicates the first vectorized representation of each instance in the modified first triplet sample, the first vectorized representation being determined by the second vectorized representation of each element in the instance; The vectorization result is input into the model to obtain the output of the model, and the output of the model is the fourth vectorized representation of each instance in the first triplet sample corresponding to the modality; Based on the output of the model and the third vectorized representation of the target element, the representation error of the target element is determined; The model parameters of the model are updated based on the representation error of the target element.

5. The method according to claim 4, characterized in that, The first vectorized representation of the instance is the result of fusing the second vectorized representation of each element in the instance; The determination of the representation error of the target element based on the output of the model and the third vectorized representation of the target element includes: Determine the fourth vectorized representation of the instance containing the target element in the output of the model; Determine the second vectorized representation of other elements in the instance containing the target element; Based on the fourth vectorized representation and the second vectorized representation of the other elements, the fifth vectorized representation of the target element is obtained through the inverse operation of the fusion operation; Based on the third and fifth vectorized representations of the target element, the representation error of the target element is determined.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: A second triplet sample corresponding to the target mode among the multiple modes is determined. The second triplet sample is obtained by encoding the sample corresponding to the target mode according to the triplet method. The target mode is at least one mode among the multiple modes. Based on the second triplet samples corresponding to the target modality, the trained model is trained to obtain the downstream task model.

7. The method according to any one of claims 1 to 6, characterized in that, The multiple modes include any of the following: Structured data, images, graph data, sequence data.

8. The method according to claim 7, characterized in that, For the structured data, the field values ​​of adjacent fields in the structured data are located in the same instance of the triple, and the relationship is the position of the first data point or the second data point in the structured data; For the image, adjacent pixel values ​​in the same row and / or adjacent pixel values ​​in the same column of the image are located in the same instance of the triple, and the relationship is the position of the first data point or the second data point in the structured data; For the graph data, two connected nodes in the graph data are located in the same instance of the triple, and the relationship is the position of the first data point or the second data point in the structured data, or the weight of the edge between the two connected nodes; For the sequence data, two adjacent objects in the sequence data are located in the same instance of the triple.

9. A model training device, characterized in that, include: The encoding module is used to determine the first triplet sample corresponding to each mode in multiple modes. The first triplet sample is obtained by encoding the sample corresponding to the mode according to the triplet method. The triplet consists of a first data point, a second data point, and the relationship between the first data point and the second data point. The first triplet sample is multiple instances of the triplet arranged in order. The training module is used to train the model based on the first triplet samples corresponding to each modality, and obtain the trained model.

10. The apparatus according to claim 9, characterized in that, The training module is used to change the target element in the first triplet sample corresponding to each modality, and determine the changed first triplet sample. The model parameters of the model are updated based on the modified first triplet sample and the target element.

11. The apparatus according to claim 10, characterized in that, The changes are made by masking or adding noise.

12. The apparatus according to claim 10 or 11, characterized in that, The training module is used to determine the vectorization result, which indicates the first vectorized representation of each instance in the modified first triplet sample, and the first vectorized representation is determined by the second vectorized representation of each element in the instance; input the vectorization result into the model to obtain the output of the model, which is the fourth vectorized representation of each instance in the first triplet sample corresponding to the modality; determine the representation error of the target element based on the output of the model and the third vectorized representation of the target element; and update the model parameters based on the representation error of the target element.

13. The apparatus according to claim 12, characterized in that, The first vectorized representation of the instance is the result of fusing the second vectorized representation of each element in the instance; The training module is used to determine the fourth vectorized representation of the instance where the target element is located in the output of the model; determine the second vectorized representation of other elements in the instance where the target element is located; obtain the fifth vectorized representation of the target element by means of the inverse operation of the fusion operation based on the fourth vectorized representation and the second vectorized representation of the other elements; and determine the representation error of the target element based on the third vectorized representation and the fifth vectorized representation of the target element.

14. The apparatus according to any one of claims 9 to 13, characterized in that, The device further includes: The fine-tuning module is used to determine the second triplet sample corresponding to the target modality among the multiple modalities. The second triplet sample is obtained by encoding the sample corresponding to the target modality according to the triplet method. The target modality is at least one of the multiple modalities. Based on the second triplet sample corresponding to the target modality, the trained model is trained to obtain the downstream task model.

15. The apparatus according to any one of claims 9 to 14, characterized in that, The multiple modes include any of the following: Structured data, images, graph data, sequence data.

16. The apparatus according to claim 15, characterized in that, For the structured data, the field values ​​of adjacent fields in the structured data are located in the same instance of the triple, and the relationship is the position of the first data point or the second data point in the structured data; For the image, adjacent pixel values ​​in the same row and / or adjacent pixel values ​​in the same column of the image are located in the same instance of the triple, and the relationship is the position of the first data point or the second data point in the structured data; For the graph data, two connected nodes in the graph data are located in the same instance of the triple, and the relationship is the position of the first data point or the second data point in the structured data, or the weight of the edge between the two connected nodes; For the sequence data, two adjacent objects in the sequence data are located in the same instance of the triple.

17. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 8.

18. A computer program product containing instructions, characterized in that, When the instruction is executed by a cluster of computer devices, the cluster of computer devices causes the cluster of computer devices to perform the method as described in any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 8.

Images