Model training method and data processing method

Abstract

The application discloses a model training method and a data processing method. The model training method comprises the following steps: obtaining training samples, wherein the training samples comprise different types of first training data and second training data; processing the first training data by using a first processing model to obtain first global features and first data block features, wherein the first global features are used to represent semantic features of the first training data, and the first data block features are used to represent features of data blocks in the first training data; processing the second training data by using a second processing model to obtain second global features and second data block features; and adjusting parameters of the first processing model and the second processing model based on global similarities of the first global features and the second global features and data block similarities of the first data block features and the second data block features. The application solves the technical problem of poor accuracy of processing multi-modal data by using a model in the related art.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a model training method and a data processing method. Background Technology

[0002] Information on the internet primarily consists of videos, images, and text, while user search requests are mainly text-based. To achieve information retrieval, neural network models can be used to represent the features of images and text in a unified manner. However, the feature vectors extracted by these models cannot accurately represent the detailed information of images and text, resulting in poor model processing accuracy.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a model training method and a data processing method to at least solve the technical problem of poor accuracy in processing multimodal data through models in related technologies.

[0005] According to one aspect of the embodiments of this application, a model training method is provided, comprising: acquiring training samples, wherein the training samples include: first training data and second training data of different types; processing the first training data using a first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of data blocks in the first training data; processing the second training data using a second processing model to obtain a second global feature and a second data block feature; and adjusting the parameters of the first processing model and the second processing model based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature, wherein the first processing model and the second processing model are machine learning models.

[0006] According to another aspect of the embodiments of this application, a data processing method is also provided, comprising: acquiring first target data of different types and multiple second target data; processing the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; processing the multiple second target data using a second processing model to obtain second feature vectors corresponding to the multiple second target data; matching the first feature vector and the second feature vector to determine push data among the multiple second target data, wherein the push data is used to represent second target data that matches the first target data; wherein the parameters of the first processing model and the second processing model are adjusted based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature, the first global feature and the first data block feature are obtained by processing the first training data using the first processing model, the second global feature and the second data block feature are obtained by processing the second training data using the second processing model, the first training data and the second training data are different types of data, the first global feature is used to represent the semantic features of the first training data, the first data block feature is used to represent the features of the data blocks in the first training data, and the first processing model and the second processing model are machine learning models.

[0007] According to another aspect of the embodiments of this application, a data processing method is also provided, comprising: displaying first target data on the operation interface in response to an input command applied to an operation interface of a client; and displaying push data among a plurality of second target data on the operation interface in response to a push command applied to the operation interface, wherein the push data is used to characterize second target data that matches the first target data, the first target data and the plurality of second target data are of different types, and the push data is obtained by matching a first feature vector corresponding to the first target data with second feature vectors corresponding to the plurality of second target data, the first feature vector being obtained by processing the first target data using a first processing model, and the second feature vector being obtained by processing the first target data using a second processing model. The first and second processing models are obtained by processing the second target data. The parameters of the first and second processing models are adjusted based on the global similarity between the first and second global features and the data block similarity between the first and second data block features. The first global features and the first data block features are obtained by processing the first training data through the first processing model. The second global features and the second data block features are obtained by processing the second training data through the second processing model. The first and second training data are different types of data. The first global features are used to characterize the semantic features of the first training data, and the first data block features are used to characterize the features of the data blocks in the first training data. The first and second processing models are machine learning models.

[0008] According to another aspect of the embodiments of this application, a data processing method is also provided, comprising: displaying first target data on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device; acquiring a plurality of second target data, wherein the first target data and the plurality of second target data are of different types; processing the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; processing the plurality of second target data using a second processing model respectively to obtain second feature vectors corresponding to the plurality of second target data; matching the first feature vector and the second feature vector to determine push data among the plurality of second target data, wherein the push data is used to characterize the second target data that matches the first target data; driving VR or AR devices display pushed data; wherein, the parameters of the first processing model and the second processing model are adjusted based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0009] According to another aspect of the embodiments of this application, a model training method is also provided, comprising: a server obtaining a model training request sent by a client by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being a model training request, the model training request being used to train a first processing model and a second processing model; the server obtaining training samples based on the model training request, wherein the training samples include: first training data and second training data of different types; the server processing the first training data using the first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of data blocks in the first training data; the server processing the second training data using a second processing model to obtain a second global feature and a second data block feature; the server adjusting the parameters of the first processing model and the second processing model based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature; the server outputting the first processing model and the second processing model to the client by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the first processing model and the second processing model, the first processing model and the second processing model being machine learning models.

[0010] According to another aspect of the embodiments of this application, a data processing method is also provided, comprising: a server obtaining first target data sent by a client by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the first target data; the server obtaining a plurality of second target data, wherein the first target data and the plurality of second target data are of different types; the server processing the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; the server processing the plurality of second target data using a second processing model respectively to obtain a second feature vector corresponding to the plurality of second target data; the server matching the first feature vector and the second feature vector to determine push data among the plurality of second target data, wherein the push data is used to characterize the second target data that matches the first target data; The server pushes data to the client by calling a second interface, where the second interface includes a second parameter whose value is the pushed data. The parameters of the first and second processing models are adjusted based on the global similarity between the first and second global features and the data block similarity between the first and second data block features. The first global features and the first data block features are obtained by processing the first training data through the first processing model, and the second global features and the second data block features are obtained by processing the second training data through the second processing model. The first and second training data are different types of data. The first global features are used to characterize the semantic features of the first training data, and the first data block features are used to characterize the features of the data blocks in the first training data. The first and second processing models are machine learning models.

[0011] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is running, it controls the device where the computer-readable storage medium is located to execute any of the methods in the above embodiments.

[0012] According to another aspect of the embodiments of this application, a computer terminal is also provided, including: a processor; a memory connected to the processor, configured to provide the processor with instructions for performing the following processing steps: acquiring training samples, wherein the training samples include: first training data and second training data of different types; processing the first training data using a first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of data blocks in the first training data; processing the second training data using a second processing model to obtain a second global feature and a second data block feature; adjusting the parameters of the first processing model and the second processing model based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature, wherein the first processing model and the second processing model are machine learning models.

[0013] In this embodiment, after obtaining different types of first and second training data, the first training data can be processed using a first processing model to obtain first global features and first data block features. The second training data can then be processed using a second processing model to obtain second global features and second data block features. Finally, based on the global similarity between the first and second global features, and the data block similarity between the first and second data block features, the parameters of the first and second processing models are adjusted to train the feature extraction model. It is noteworthy that since the adjustment of model parameters is based on global similarity and data block similarity, it achieves a more accurate representation of the similarity between images and text, improves model training performance, and further enhances the accuracy of the model in processing multimodal data. This solves the technical problem of poor accuracy in processing multimodal data using models in related technologies. In image and text search scenarios, the first and second processing models can extract more accurate image and text features, making the searched information more accurate and better suited to the user's search needs, thereby improving the accuracy of image and text search. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0015] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a model training method according to an embodiment of this application;

[0016] Figure 2This is a structural block diagram of the computational environment for a model training method according to an embodiment of this application;

[0017] Figure 3 This is a flowchart of a model training method according to an embodiment of this application;

[0018] Figure 4 This is a flowchart of an optional model training method according to an embodiment of this application;

[0019] Figure 5 This is a flowchart of a first data processing method according to an embodiment of this application;

[0020] Figure 6 This is a flowchart of a second data processing method according to an embodiment of this application;

[0021] Figure 7 This is a flowchart of a third data processing method according to an embodiment of this application;

[0022] Figure 8 This is a flowchart of another model training method according to an embodiment of this application;

[0023] Figure 9 This is a flowchart of the fourth data processing method according to an embodiment of this application;

[0024] Figure 10 This is a schematic diagram of a model training apparatus according to an embodiment of this application;

[0025] Figure 11 This is a schematic diagram of a first data processing apparatus according to an embodiment of this application;

[0026] Figure 12 This is a schematic diagram of a second data processing apparatus according to an embodiment of this application;

[0027] Figure 13 This is a schematic diagram of a third data processing apparatus according to an embodiment of this application;

[0028] Figure 14 This is a schematic diagram of another model training apparatus according to an embodiment of this application;

[0029] Figure 15 This is a schematic diagram of a fourth data processing apparatus according to an embodiment of this application;

[0030] Figure 16 This is a structural block diagram of a computer terminal according to an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0034] Multimodal feature learning: By using machine learning techniques to train a feature encoding model, user-input image or text data can be mapped into a feature vector. This model can be used for tasks such as image and text search.

[0035] The Transformer model is a neural network model that extracts features from input images or text. Before inputting an image, it needs to be segmented into multiple image blocks (e.g., into 16*16 non-overlapping blocks), and each image block is an image token. Similarly, before inputting text, it needs to be segmented into multiple text tokens, and each text token can be approximated as a word.

[0036] The InfoNCE (Info Noise-contrastive estimation) loss function aims to make the similarity between positive sample pairs greater than that between negative sample pairs.

[0037] Currently, in multimodal feature learning, CLIP (Contrastive Language-Image Pre-training) models can be used. After inputting image or text data, neural networks such as CNNs or Transformers can be used to represent the image or text data as a global feature vector. In the training results, feature vectors are first extracted from all images and texts. Then, the similarity between the feature vectors of positive sample image-text pairs is required to be greater than the similarity between the feature vectors of negative sample image-text pairs.

[0038] However, by averaging or otherwise compressing image or text data into a global feature vector, this model loses a significant amount of image or text detail data. This makes it impossible to distinguish between positive samples and more difficult negative samples during the training phase, thereby introducing noise into the training process and affecting the model's performance.

[0039] To address the aforementioned issues, this application provides an implementation scheme that establishes a many-to-many correspondence between image tokens and text tokens, thereby more accurately expressing the similarity between image or text data, improving the effect of feature training, further enhancing the accuracy of model processing, and enabling the delivery of more accurate retrieval results to users in image and text search tasks.

[0040] Example 1

[0041] According to an embodiment of this application, a model training method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0042] The method embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a model training method is shown. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0043] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). This data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model training method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned model training method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0046] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0047] Figure 1 The hardware structure block diagram shown can serve not only as an exemplary block diagram of the aforementioned computer terminal (or mobile device), but also as an exemplary block diagram of the aforementioned server. In one optional embodiment, Figure 2 The use of the above is illustrated in a block diagram. Figure 1 The computer terminal (or mobile device) shown is an example of a computing node in computing environment 201. Figure 2 This is a structural block diagram of the computational environment for a model training method according to an embodiment of this application, such as... Figure 2 As shown, computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (shown as 210-1, 210-2, ..., in the diagram). Each computing node contains local processing and memory resources, and end user 202 can remotely run applications or store data within computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3, and 220-4 within computing environment 201, representing services "A", "D", "E", and "H", respectively.

[0048] End user 202 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or requests of end user 202 can be provided to ingress gateway 230. Ingress gateway 230 may include a corresponding agent to handle provisioning and / or requests for service 220 (one or more services provided in computing environment 201).

[0049] Service 220 is provided or deployed based on various virtualization technologies supported by computing environment 201. In some embodiments, service 220 may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization may involve simulating a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization may launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.

[0050] In one embodiment based on container virtualization, several containers of service 220 can be assembled into a POD (e.g., a Kubernetes POD). For example, such as Figure 2 As shown, service 220-2 can be equipped with one or more PODs 240-1, 240-2, ..., 240-N (collectively referred to as POD 240). Each POD 240 can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers 242). One or more containers 242 in POD 240 handle requests related to one or more corresponding functions of the service, and the proxy 245 typically controls service-related network functions such as routing and load balancing. Other services 220 can also be equipped with PODs similar to POD 240.

[0051] During operation, executing a user request from end user 202 may require calling one or more services 220 in computing environment 201, and executing one or more functions of one service 220 may require calling one or more functions of another service 220. For example... Figure 2 As shown, service "A" 220-1 receives user requests from terminal user 202 from ingress gateway 230. Service "A" 220-1 can call service "D" 220-2, and service "D" 220-2 can request service "E" 220-3 to perform one or more functions.

[0052] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.

[0053] Under the aforementioned operating environment, this application provides the following: Figure 3 The model training method shown is illustrated. It should be noted that the model training method in this embodiment can be developed by... Figure 1The computer terminal of the illustrated embodiment is used for execution. Figure 3 This is a flowchart of a model training method according to an embodiment of this application. Figure 3 As shown, the method may include the following steps:

[0054] Step S302: Obtain training samples, wherein the training samples include: first training data and second training data of different types.

[0055] The training samples in the above steps can be multimodal data, that is, the training samples can contain two different types of data, namely the first training data and the second training data. The specific data types can be determined according to the application scenario. In this embodiment of the application, taking the image and text search scenario as an example, the first training data can be image data and the second training data can be text data, but it is not limited to these.

[0056] In addition, to improve the model's performance, training samples can be divided into positive samples and negative samples. Positive samples contain two different types of data that are matched. For example, in the context of image and text search, for positive samples, the text data is "chiffon dress" and the image data can be an image of "chiffon dress"; for negative samples, the text data is "chiffon dress" and the image data can be an image of "celebrity A".

[0057] In one optional embodiment, different users often have different purposes for training the model. To ensure that the trained model can meet the user's needs, multimodal data provided by the user or multimodal data stored in the user's server can be used as training samples. In another optional embodiment, the number of training samples provided by the user is limited. To further improve the accuracy of the model processing, a large amount of multimodal data can be obtained from the Internet as training samples based on the user's needs.

[0058] Step S304: The first training data is processed using the first processing model to obtain the first global feature and the first data block feature, wherein the first global feature is used to characterize the semantic features of the first training data and the first data block feature is used to characterize the features of the data blocks in the first training data.

[0059] The first processing model in the above steps can be a machine learning model capable of extracting features from the first training data. This model can be trained using data of the same type as the first training data. The model can employ a model structure capable of feature extraction from related technologies. In this embodiment, the Transformer model is used as an example, but it is not limited to this. In an image-text search scenario, when the first training data is image data, the first processing model can be an image encoder; when the first training data is text data, the first processing model can be a text encoder.

[0060] In one optional embodiment, when feature extraction is required on the first training data, the first training data is first segmented to obtain multiple data blocks (patchtokens). Then, a learnable embedding vector (classtoken) is added to obtain the final input sequence to the first processing model. Here, the class token can represent the overall semantics of the first training data. After inputting the input sequence into the first processing model for feature extraction, the feature vector corresponding to the classtoken (i.e., the aforementioned first global feature) and the feature vector corresponding to each patch token (i.e., the aforementioned first data block feature) are obtained.

[0061] Step S306: Process the second training data using the second processing model to obtain the second global feature and the second data block feature.

[0062] The second processing model in the above steps can be a machine learning model capable of extracting features from the first training data. Since the first and second training data are two different types of data, the first and second processing models need to be trained separately for the first and second training data to ensure the accuracy of feature extraction. In an image-text search scenario, when the second training data is text data, the second processing model can be a text encoder; when the second training data is image data, the second processing model can be an image encoder.

[0063] Since the first processing model and the second processing model have the same processing purpose, the structure of the second processing model is the same as that of the first processing model, and the processing flow of the second processing model is the same as that of the first processing model, so they will not be described in detail here.

[0064] Step S308: Adjust the parameters of the first processing model and the second processing model based on the global similarity between the first global feature and the second global feature, and the data block similarity between the second data block feature and the second data block feature.

[0065] The global similarity in the above steps can be a one-to-one similarity relationship between the first global feature and the second global feature, and the data block similarity can be a many-to-many similarity relationship between the first data block feature and the second data block feature.

[0066] In one optional embodiment, the training objective of the first processing model and the second processing model can be that the similarity of positive samples is greater than the similarity of negative samples. Therefore, a loss function can be constructed based on global similarity and data block similarity to further adjust the parameters of the first processing model and the second processing model, ensuring that the model accuracy of the two models is the same.

[0067] It should be noted that the specific adjustment process of the parameters can be achieved using methods provided in related technologies, and this application does not impose any specific limitations on this.

[0068] The following is combined with Figure 4 Taking an image and text search scenario as an example, a preferred embodiment of this application will be described in detail. For example... Figure 4 As shown, the training samples consist of images and text. The image depicts a pizza, and the text reads "Sausage pizza plate." Image features are extracted using an image encoder, and text features are extracted using a text encoder. Before processing the image, the image is segmented into six image blocks (six image tokens). These six image tokens, along with a class token, are then input into the image encoder to obtain global image features and six image token features. Similarly, before processing the text, it is segmented into six characters (six text tokens). These six text tokens, along with another class token, are then input into a text editor to obtain global text features and six text token features. The global image and text features are used to construct a global feature alignment loss. The six image tokens and six text tokens are then processed by a token feature alignment module to construct a token feature alignment loss. The parameters of the image encoder and text encoder are updated using the global feature alignment loss and the token feature alignment loss, thus achieving the model training objective. By establishing a one-to-one correspondence between global image features and global text features, as well as a many-to-many correspondence between image tokens and text tokens, the similarity between images and text can be expressed more accurately, thereby improving the effect of feature training.

[0069] The solution provided by the above embodiments of this application, after obtaining different types of first and second training data, can use a first processing model to process the first training data to obtain first global features and first data block features, and use a second processing model to process the second training data to obtain second global features and second data block features. Finally, based on the global similarity between the first and second global features and the data block similarity between the first and second data block features, the parameters of the first and second processing models are adjusted to achieve the purpose of training the feature extraction model. It is noteworthy that since the adjustment of model parameters is based on global similarity and data block similarity, it achieves a more accurate representation of the similarity between images and text, improves the model training effect, and further enhances the technical effect of improving the accuracy of the model in processing multimodal data. This solves the technical problem of poor accuracy in processing multimodal data through models in related technologies. In image and text search scenarios, the first and second processing models can extract image and text features with higher accuracy, further making the searched information more accurate and better meeting the user's search needs, thereby improving the accuracy of image and text search.

[0070] In the above embodiments of this application, the parameters of the first processing model and the second processing model are adjusted based on the global similarity of the first global feature and the second global feature, and the data block similarity of the second data block feature and the second data block feature. This includes: constructing a first loss function based on global similarity; constructing a second loss function based on data block similarity; obtaining a weighted sum of the first loss function and the second loss function to obtain a target loss function; and adjusting the parameters of the first processing model and the second processing model based on the target loss function.

[0071] The first and second loss functions mentioned above can be set according to the actual training objective. For example, in the image and text search scenario, the training objective is the similarity between the feature vectors of positive samples, which is greater than the similarity between the feature vectors of negative samples. Therefore, the first and second loss functions can be the InfoNCE loss function, but are not limited to this.

[0072] In one alternative embodiment, the training samples may include positive and negative samples, for example, such as Figure 4 As shown, the "sausage pizza plate" and the image represent the same item, therefore, Figure 4The images and text shown are positive samples. Therefore, the global similarity includes the first global similarity of positive samples and the second global similarity of negative samples, and the data block similarity includes the first data block similarity of positive samples and the second data block similarity of negative samples. Then, according to the formula of the InfoNCE loss function, the first loss function and the second loss function are constructed respectively. Then, the target loss function is obtained by weighted summing the function values ​​of the two loss functions. If the target loss function is greater than the threshold, the parameters of the first processing model and the second processing model can be adjusted, and training can continue using training samples until the target loss function is less than the threshold. At this point, the adjustment of the parameters of the first processing model and the second processing model is stopped, and the entire training process is considered complete.

[0073] In the above embodiments of this application, constructing a first loss function based on global similarity includes: obtaining a first global similarity corresponding to a first sample and a second global similarity corresponding to a second sample from the global similarity, wherein the two different types of data contained in the first sample match, and the two different types of data contained in the second sample do not match; and constructing a first loss function based on the first global similarity and the second global similarity.

[0074] In an alternative embodiment, the first loss function L can be constructed using the following formula: t :

[0075]

[0076] Among them, s p Indicates the first global similarity. and Let represent the second global similarity, and τ represent the hyperparameter. It should be noted that the above formula provides an implementation scheme for constructing the first loss function using two second global similarities, but it is not limited to this; the number of second global similarities to be used can be determined according to actual needs.

[0077] In the above embodiments of this application, constructing a second loss function based on data block similarity includes: obtaining the first data block similarity corresponding to the first sample and the second data block similarity corresponding to the second sample from the data block similarity; and constructing a second loss function based on the first data block similarity and the second data block similarity.

[0078] It should be noted that both the second loss function and the first loss function are InfoNCE loss functions. Therefore, the construction process of the second loss function is the same as that of the first loss function, and will not be elaborated here.

[0079] In the above embodiments of this application, the method further includes: using a feature alignment module to perform feature alignment on the features of the second data block and the features of the second data block to obtain data block similarity.

[0080] The feature alignment module in the above steps can be any model provided in the related technology that can achieve the purpose of token feature alignment. In this embodiment, the feature alignment module can be implemented by a single layer of cross attention.

[0081] In an optional embodiment, since the number of multimodal data partitions is large in practical applications, and token feature alignment requires processing all token features, in order to improve model training efficiency and save manpower costs, the feature alignment module provided in related technologies can be used to implement token feature alignment, thereby determining the similarity of data blocks.

[0082] In the above embodiments of this application, the method further includes: acquiring first target data; processing the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; processing multiple second target data using a second processing model to obtain second feature vectors corresponding to the multiple second target data; matching the first feature vector and the second feature vector to determine push data among the multiple second target data, wherein the push data is used to characterize the second target data that matches the first target data.

[0083] In one optional embodiment, to achieve multimodal data retrieval and data push, the user can input first target data for which data retrieval or data push is needed. A first processing model trained using the above method is then used to extract features from the first target data, resulting in a first feature vector. Additionally, for multiple second target data items stored in the database (i.e., data to be pushed to the user), a second processing model trained using the above method can be used to extract features from the second target data, resulting in multiple second feature vectors, each corresponding to a second target data item. Then, by matching the first feature vector with each second feature vector, a matching second feature vector can be determined, and the second target data corresponding to that second feature vector is used as the final retrieval result or push data pushed to the user.

[0084] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0086] Example 2

[0087] According to an embodiment of this application, a data processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0088] Figure 5 This is a flowchart of a first data processing method according to an embodiment of this application. For example... Figure 5 As shown, the method may include the following steps:

[0089] Step S502: Obtain different types of first target data and multiple second target data.

[0090] The first target data and multiple second target data in the above steps can be multimodal data, that is, they belong to two different types. For example, in the image and text search scenario, the first target data can be text data and the multiple second target data can be image data, or the first target data can be image data and the multiple second target data can be text data.

[0091] In one alternative embodiment, the first target data can be provided by the user. For example, in the case of text data, the first target data can be text manually entered by the user; in the case of image data, the first target data can be images captured by the user, or images selected by the user from captured images or images on the Internet. Multiple sets of second target data can be data stored in a database or data from the Internet.

[0092] Step S504: Process the first target data using the first processing model to obtain the first feature vector corresponding to the first target data.

[0093] The first feature vector in the above steps can be a vector obtained by compressing the input sequence input to the first processing model. Here, the input sequence can include multiple data blocks obtained by segmenting the first target data, and a class token that represents the overall semantics of the first target data. The first processing model can be a model capable of extracting features from the first target data. This model can be trained using data of the same type as the first target data. The model can employ a model structure capable of feature extraction from related technologies. In this embodiment, the Transformer model is used as an example, but it is not limited to this. In an image-text search scenario, when the first target data is image data, the first processing model can be an image encoder; when the first target data is text data, the first processing model can be a text encoder.

[0094] In one optional embodiment, the first target data is first segmented to obtain multiple data blocks (patchtokens). Then, a learnable embedding vector (class token) is added to obtain the final input sequence to the first processing model. The class token can represent the overall semantics of the first target data. After feature extraction using the first processing model, a compressed vector, i.e., the aforementioned first feature vector, is obtained.

[0095] It should be noted that the first processing model can be deployed on the client or on a cloud server. In order to reduce the operating resources of the client, this embodiment of the application will take the deployment of the first processing model on a cloud server as an example.

[0096] Step S506: The second processing model is used to process the multiple second target data respectively to obtain the second feature vectors corresponding to the multiple second target data.

[0097] Since the first target data and the second target data are two different types of data, it is necessary to train the first processing model and the second processing model separately for the first target data and the second target data, respectively, to ensure the accuracy of feature extraction. In the image-text search scenario, when the second target data is text data, the second processing model can be a text encoder; when the second target data is image data, the second processing model can be an image encoder.

[0098] Since the first processing model and the second processing model have the same processing purpose, the structure of the second processing model is the same as that of the first processing model, and the processing flow of the second processing model is the same as that of the first processing model, so they will not be described in detail here.

[0099] Step S508: Match the first feature vector and the second feature vector to determine the push data among multiple second target data, wherein the push data is used to characterize the second target data that matches the first target data.

[0100] In one optional embodiment, the similarity between the first feature vector and each second feature vector can be calculated separately to match the first feature vector and each second feature vector. The second target data corresponding to the minimum similarity is further determined to be the second target data that matches the first target data. Therefore, the second target data can be determined as the push data to be finally pushed to the user.

[0101] The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0102] It should be noted that the processing of the first processing model and the second processing model can refer to the model training method in embodiment 1 above. The specific training process will not be described in detail here.

[0103] In the above embodiments of this application, after determining the push data, the method further includes: outputting the push data; receiving feedback data, wherein the feedback data is used to characterize the data after the push data has been modified; and adjusting the parameters of the first processing model and the second processing model based on the feedback data.

[0104] In one optional embodiment, when both the first processing model and the second processing model are deployed in a cloud server, after determining the push data from multiple second target data, the push data can be fed back to the user for confirmation. If the push data meets the user's needs, the user can provide confirmation information; if the push data does not meet the user's needs, the user can modify the push data, obtain feedback data, and adjust the parameters of the first and second processing models based on the feedback data to improve server performance.

[0105] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0106] Example 3

[0107] According to an embodiment of this application, a data processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0108] Figure 6 This is a flowchart of a second data processing method according to an embodiment of this application. For example... Figure 6 As shown, the method may include the following steps:

[0109] Step S602: In response to the input command applied to the operation interface of the client, the first target data is displayed on the operation interface.

[0110] The operation interface in the above steps can be the interface provided to the user on the client side. The user can provide the first target data by operating on this interface, and the first target data can be displayed in the first display area of ​​the operation interface. The input command can be a command that the user enters text content in the text box, a command generated by the user clicking the "Shoot" button, or a command generated by the user dragging the first target data to the input area, but it is not limited to these.

[0111] Step S604: In response to the push command applied to the client's operation interface, push data from multiple second target data is displayed on the operation interface. The push data is used to represent the second target data that matches the first target data. The first target data and the multiple second target data are of different types. The push data is obtained by matching the first feature vector corresponding to the first target data with the second feature vectors corresponding to the multiple second target data. The first feature vector is obtained by processing the first target data through a first processing model, and the second feature vector is obtained by processing the second target data through a second processing model.

[0112] The push notification in the above steps can be a notification generated by the user by clicking the "Push" button. In the context of image and text search, the push notification can be an image and text search command, but it is not limited to this.

[0113] The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0114] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 2, but are not limited to the schemes provided in Embodiment 2.

[0115] Example 4

[0116] According to the embodiments of this application, a data processing method applicable to virtual reality scenarios such as virtual reality (VR) devices and augmented reality (AR) devices is also provided. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0117] Figure 7 This is a flowchart of a third data processing method according to an embodiment of this application. For example... Figure 7 As shown, the method may include the following steps:

[0118] Step S702: Display the first target data on the presentation screen of the virtual reality (VR) device or augmented reality (AR) device.

[0119] Step S704: Obtain multiple second target data, wherein the first target data and the multiple second target data are of different types.

[0120] Step S706: Process the first target data using the first processing model to obtain the first feature vector corresponding to the first target data.

[0121] Step S708: The second processing model is used to process the multiple second target data respectively to obtain the second feature vectors corresponding to the multiple second target data.

[0122] Step S710: Match the first feature vector and the second feature vector to determine the push data among multiple second target data, wherein the push data is used to characterize the second target data that matches the first target data.

[0123] Step S712: Drive the VR or AR device to display the pushed data.

[0124] The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0125] Optionally, in this embodiment, the above data processing method can be applied to a hardware environment consisting of a server and a virtual reality device. The pushed data is displayed on the screen of the virtual reality (VR) device or augmented reality (AR) device. The server can be a server corresponding to a media file operator. The aforementioned network includes, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), or a local area network (LAN). The aforementioned virtual reality device is not limited to, for example, a virtual reality headset, virtual reality glasses, or a standalone virtual reality device.

[0126] Optionally, the virtual reality device includes: a memory, a processor, and a transmission device. The memory stores an application that can be used to perform: displaying first target data on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device; acquiring multiple second target data sets, wherein the first target data and the multiple second target data sets are of different types; processing the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; processing the multiple second target data sets separately using a second processing model to obtain second feature vectors corresponding to the multiple second target data sets; matching the first feature vector and the second feature vector to determine push data among the multiple second target data sets, wherein the push data is used to characterize the second target data set that matches the first target data set; and driving the VR device. The system displays and pushes data to an AR device; wherein, the parameters of the first processing model and the second processing model are adjusted based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0127] It should be noted that the data processing method described above for use in VR or AR devices in this embodiment may include... Figure 3 The method of the illustrated embodiment is used to drive VR or AR devices to display pushed data.

[0128] Optionally, the processor in this embodiment can invoke the application stored in the memory via the transmission device to perform the above steps. The transmission device can receive media files sent by the server via a network, and can also be used for data transmission between the processor and the memory.

[0129] Optionally, in a virtual reality device, there is a head-mounted display with eye tracking. The screen in the HMD is used to display the video footage. The eye tracking module in the HMD is used to acquire the real-time movement path of the user's eyes. The tracking system is used to track the user's position and movement information in real three-dimensional space. The computing and processing unit is used to acquire the user's real-time position and movement information from the tracking system and calculate the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of vision orientation in the virtual three-dimensional space.

[0130] In this embodiment, the virtual reality device can be connected to a terminal, and the terminal and the server are connected via a network. The virtual reality device is not limited to virtual reality headsets, virtual reality glasses, virtual reality all-in-one machines, etc., and the terminal is not limited to PCs, mobile phones, tablets, etc. The server can be a server corresponding to a media file operator, and the network includes, but is not limited to, wide area networks, metropolitan area networks, or local area networks.

[0131] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 2, but are not limited to the schemes provided in Embodiment 2.

[0132] Example 5

[0133] According to an embodiment of this application, a model training method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0134] Figure 8 This is a flowchart of another model training method according to an embodiment of this application. For example... Figure 8 As shown, the method may include the following steps:

[0135] In step S802, the server obtains the model training request sent by the client by calling the first interface. The first interface includes a first parameter, the value of which is the model training request. The model training request is used to train the first processing model and the second processing model.

[0136] Step S804: The server obtains training samples based on the model training request, wherein the training samples include: first training data and second training data of different types.

[0137] Step S806: The server processes the first training data using the first processing model to obtain the first global feature and the first data block feature. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data.

[0138] In step S808, the server uses the second processing model to process the second training data to obtain the second global feature and the second data block feature.

[0139] In step S810, the server adjusts the parameters of the first processing model and the second processing model based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature.

[0140] In step S812, the server outputs the first processing model and the second processing model to the client by calling the second interface. The second interface includes a second parameter, the value of which is the first processing model and the second processing model. The first processing model and the second processing model are machine learning models.

[0141] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0142] Example 6

[0143] According to an embodiment of this application, a data processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0144] Figure 9 This is a flowchart of the fourth data processing method according to an embodiment of this application. For example... Figure 9 As shown, the method may include the following steps:

[0145] In step S902, the server obtains the first target data sent by the client by calling the first interface, wherein the first interface includes a first parameter and the parameter value of the first parameter is the first target data.

[0146] In step S904, the server acquires multiple second target data, wherein the first target data and the multiple second target data are of different types.

[0147] In step S906, the server processes the first target data using the first processing model to obtain the first feature vector corresponding to the first target data.

[0148] In step S908, the server uses the second processing model to process multiple second target data respectively to obtain the second feature vectors corresponding to the multiple second target data.

[0149] In step S910, the server matches the first feature vector and the second feature vector to determine the push data among multiple second target data, wherein the push data is used to characterize the second target data that matches the first target data.

[0150] In step S912, the server outputs push data to the client by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the push data.

[0151] The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0152] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 2, but are not limited to the schemes provided in Embodiment 2.

[0153] Example 7

[0154] According to embodiments of this application, a model processing apparatus for implementing the above-described model training method is also provided, such as... Figure 10 As shown, the device 1000 includes: an acquisition module 1002, a first processing module 1004, a second processing module 1006, and an adjustment module 1008.

[0155] The acquisition module 1002 is used to acquire training samples, which include first training data and second training data of different types; the first processing module 1004 is used to process the first training data using a first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of data blocks in the first training data; the second processing module 1006 is used to process the second training data using a second processing model to obtain a second global feature and a second data block feature; the adjustment module 1008 is used to adjust the parameters of the first processing model and the second processing model based on the global similarity between the first global feature and the second global feature, and the data block similarity between the second data block feature and the second data block feature, wherein the first processing model and the second processing model are machine learning models.

[0156] It should be noted that the acquisition module 1002, the first processing module 1004, the second processing module 1006, and the adjustment module 1008 mentioned above correspond to steps S302 to S308 in Embodiment 1. The four modules and their corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run on the computer terminal provided in Embodiment 1.

[0157] In the above embodiments of this application, the adjustment module includes: a first construction unit, a second construction unit, a weighting unit, and an adjustment unit.

[0158] The first construction unit is used to construct a first loss function based on global similarity; the second construction unit is used to construct a second loss function based on data block similarity; the weighting unit is used to obtain the weighted sum of the first loss function and the second loss function to obtain the target loss function; and the adjustment unit is used to adjust the parameters of the first processing model and the second processing model based on the target loss function.

[0159] In the above embodiments of this application, the first construction unit is further configured to obtain the first global similarity corresponding to the first sample and the second global similarity corresponding to the second sample from the global similarity, wherein the two different types of data contained in the first sample match, and the two different types of data contained in the second sample do not match; and to construct a first loss function based on the first global similarity and the second global similarity.

[0160] In the above embodiments of this application, the second construction unit is further configured to obtain the first data block similarity corresponding to the first sample and the second data block similarity corresponding to the second sample from the data block similarity; and construct a second loss function based on the first data block similarity and the second data block similarity.

[0161] In the above embodiments of this application, the device further includes a feature alignment module.

[0162] The feature alignment module is used to align the features of the second data block with the features of the second data block to obtain the data block similarity.

[0163] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0164] Example 8

[0165] According to embodiments of this application, a data processing apparatus for implementing the above-described data processing method is also provided, such as... Figure 11 As shown, the device 1100 includes: an acquisition module 1102, a first processing module 1104, a second processing module 1106, and a matching module 1108.

[0166] The acquisition module 1102 is used to acquire different types of first target data and multiple second target data; the first processing module 1104 is used to process the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; the second processing module 1106 is used to process multiple second target data using a second processing model to obtain a second feature vector corresponding to the multiple second target data; the matching module 1108 is used to match the first feature vector and the second feature vector to determine the push data among the multiple second target data, wherein the push data is used to represent the second target data that matches the first target data, the parameters of the first processing model and the second processing model are adjusted based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature, the first global feature and the first data block feature are obtained by processing the first training data through the first processing model, the second global feature and the second data block feature are obtained by processing the second training data through the second processing model, the first training data and the second training data are different types of data, the first global feature is used to represent the semantic features of the first training data, the first data block feature is used to represent the features of the data blocks in the first training data, and the first processing model and the second processing model are machine learning models.

[0167] It should be noted that the acquisition module 1102, the first processing module 1104, the second processing module 1106, and the matching module 1108 mentioned above correspond to steps S502 to S508 in Embodiment 2. The four modules and their corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 2. It should also be noted that the above modules, as part of the device, can run on the computer terminal provided in Embodiment 1.

[0168] Example 9

[0169] According to an embodiment of this application, a data processing apparatus for implementing the above-described data processing method is also provided, deployed in a client, such as... Figure 12 As shown, the device 1200 includes: a first display module 1202 and a second display module 1204.

[0170] The first display module 1202 is used to respond to input commands applied to the client operation interface and display the first target data on the operation interface; the second display module 1204 is used to respond to push commands applied to the operation interface and display push data from multiple second target data on the operation interface. The push data represents the second target data that matches the first target data. The first target data and the multiple second target data are of different types. The push data is obtained by matching the first feature vector corresponding to the first target data with the second feature vectors corresponding to the multiple second target data. The first feature vector is obtained by processing the first target data using a first processing model, and the second feature vector is obtained by processing the first target data using a second processing model. The second target data is processed to obtain the parameters of the first processing model and the second processing model. The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model. The second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0171] It should be noted that the first display module 1202 and the second display module 1204 mentioned above correspond to steps S602 to S604 in Embodiment 3. The two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 3. It should also be noted that the above modules, as part of the device, can run in the computer terminal provided in Embodiment 1.

[0172] Example 10

[0173] According to embodiments of this application, a data processing apparatus for implementing the above-described data processing method is also provided, such as... Figure 13 As shown, the device 1300 includes: a first display module 1302, an acquisition module 1304, a first processing module 1306, a second processing module 1308, a matching module 1310, and a second display module 1312.

[0174] The system comprises the following modules: a first display module 1302 for displaying first target data on the screen of a virtual reality (VR) device or an augmented reality (AR) device; an acquisition module 1304 for acquiring multiple second target data sets, wherein the first target data and the multiple second target data sets are of different types; a first processing module 1306 for processing the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; a second processing module 1308 for processing the multiple second target data sets using a second processing model to obtain second feature vectors corresponding to the multiple second target data sets; and a matching module 1310 for matching the first feature vector and the second feature vector to determine push data among the multiple second target data sets, wherein the push data is used to characterize data that matches the first target data. The second target data; the second display module 1312 is used to drive VR devices or AR devices to display and push data; wherein, the parameters of the first processing model and the second processing model are adjusted based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0175] It should be noted that the first display module 1302, acquisition module 1304, first processing module 1306, second processing module 1308, matching module 1310, and second display module 1312 mentioned above correspond to steps S702 to S712 in Embodiment 4. The six modules and their corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 4. It should also be noted that the above modules, as part of the device, can run on the computer terminal provided in Embodiment 1.

[0176] Example 11

[0177] According to an embodiment of this application, a model training apparatus for implementing the above-described model training method is also provided, deployed on a server, such as... Figure 14 As shown, the device 1400 includes: a first calling module 1402, an acquisition module 1404, a first processing module 1406, a second processing module 1408, an adjustment module 1410, and a second calling module 1412.

[0178] The system includes the following modules: a first calling module 1402, which obtains a model training request sent by a client by calling a first interface, wherein the first interface includes a first parameter, the value of which is a model training request, and the model training request is used to train a first processing model and a second processing model; an acquisition module 1404, which obtains training samples based on the model training request, wherein the training samples include first training data and second training data of different types; a first processing module 1406, which processes the first training data using the first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of data blocks in the first training data; a second processing module 1408, which processes the second training data using the second processing model to obtain a second global feature and a second data block feature; and an adjustment module 1410, which adjusts the first global feature and the second global feature based on the global similarity, the first data block feature, and the second data block feature. Based on the data block similarity of the block features, the parameters of the first processing model and the second processing model are adjusted; the second calling module 1412 is used to output the first processing model and the second processing model to the client by calling the second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter is the first processing model and the second processing model; wherein the parameters of the first processing model and the second processing model are adjusted based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0179] It should be noted that the first calling module 1402, the acquisition module 1404, the first processing module 1406, the second processing module 1408, the adjustment module 1410, and the second calling module 1412 mentioned above correspond to steps S802 to S812 in Embodiment 5. The six modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 5. It should also be noted that the above modules, as part of the device, can run in the computer terminal provided in Embodiment 1.

[0180] Example 12

[0181] According to an embodiment of this application, a data processing apparatus for implementing the above-described data processing method is also provided, deployed on a server, such as... Figure 15As shown, the device 1500 includes: a first calling module 1502, an acquisition module 1504, a first processing module 1506, a second processing module 1508, a matching module 1510, and a second calling module 1512.

[0182] The system comprises the following modules: a first calling module 1502, which obtains first target data sent by the client by calling a first interface, wherein the first interface includes a first parameter and the parameter value of the first parameter is the first target data; an acquisition module 1504, which obtains multiple second target data, wherein the first target data and the multiple second target data are of different types; a first processing module 1506, which processes the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; a second processing module 1508, which processes the multiple second target data using a second processing model to obtain second feature vectors corresponding to the multiple second target data; a matching module 1510, which matches the first feature vector and the second feature vector to determine the push data among the multiple second target data, wherein the push data is used to represent the second target data that matches the first target data; and a second calling module 1512, which outputs push data to the client by calling a second interface, wherein the second interface includes a second parameter and the parameter value of the second parameter is the push data.

[0183] The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0184] It should be noted that the first calling module 1502, the acquisition module 1504, the first processing module 1506, the second processing module 1508, the matching module 1510, and the second calling module 1512 mentioned above correspond to steps S902 to S912 in Embodiment 6. The six modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 6. It should also be noted that the above modules, as part of the device, can run in the computer terminal provided in Embodiment 1.

[0185] Example 13

[0186] Embodiments of this application may provide a computer terminal, which may be any computer terminal in a cluster of computer terminals. Optionally, in this embodiment, the aforementioned computer terminal may also be replaced with a mobile terminal or other terminal device.

[0187] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0188] In this embodiment, the computer terminal described above can execute the program code for the following steps in the model training method: obtaining training samples, which include: first training data and second training data of different types; processing the first training data using a first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to characterize the semantic features of the first training data and the first data block feature is used to characterize the features of data blocks in the first training data; processing the second training data using a second processing model to obtain a second global feature and a second data block feature; adjusting the parameters of the first processing model and the second processing model based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature, wherein the first processing model and the second processing model are machine learning models.

[0189] Optionally, Figure 16 This is a structural block diagram of a computer terminal according to an embodiment of this application. Figure 16 As shown, the computer terminal A may include one or more (only one is shown in the figure) processors 1602 and memory 1604.

[0190] The memory can be used to store software programs and modules, such as the model training method and apparatus in the embodiments of this application, and the program instructions / modules corresponding to the data processing method and apparatus. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned model training method and data processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0191] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: acquiring training samples, which include first training data and second training data of different types; processing the first training data using a first processing model to obtain first global features and first data block features, wherein the first global features are used to characterize the semantic features of the first training data and the first data block features are used to characterize the features of data blocks in the first training data; processing the second training data using a second processing model to obtain second global features and second data block features; and adjusting the parameters of the first processing model and the second processing model based on the global similarity between the first global features and the second global features, and the data block similarity between the first data block features and the second data block features, wherein the first processing model and the second processing model are machine learning models.

[0192] Optionally, the processor may also execute program code that performs the following steps: constructing a first loss function based on global similarity; constructing a second loss function based on data block similarity; obtaining a weighted sum of the first and second loss functions to obtain a target loss function; and adjusting the parameters of the first and second processing models based on the target loss function.

[0193] Optionally, the processor may also execute program code that performs the following steps: obtaining a first global similarity corresponding to a first sample and a second global similarity corresponding to a second sample from the global similarity, wherein the two different types of data contained in the first sample match, and the two different types of data contained in the second sample do not match; and constructing a first loss function based on the first global similarity and the second global similarity.

[0194] Optionally, the processor may also execute program code that performs the following steps: obtaining the first data block similarity corresponding to the first sample and the second data block similarity corresponding to the second sample from the data block similarity; and constructing a second loss function based on the first data block similarity and the second data block similarity.

[0195] Optionally, the processor may also execute program code that performs the following steps: using a feature alignment module to align the features of the first data block and the features of the second data block to obtain the data block similarity.

[0196] The processor can invoke information and application programs stored in the memory through a transmission device to perform the following steps: acquiring first target data of different types and multiple second target data; processing the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; processing the multiple second target data using a second processing model to obtain second feature vectors corresponding to the multiple second target data; matching the first feature vector and the second feature vector to determine push data among the multiple second target data, wherein the push data is used to represent the second target data that matches the first target data; wherein the parameters of the first processing model and the second processing model are adjusted based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature; the first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model; the first training data and the second training data are different types of data; the first global feature is used to represent the semantic features of the first training data; the first data block feature is used to represent the features of the data blocks in the first training data; and the first processing model and the second processing model are machine learning models.

[0197] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: responding to an input command applied to the client's operating interface, displaying first target data on the operating interface; responding to a push command applied to the operating interface, displaying push data from a plurality of second target data on the operating interface, wherein the push data is used to characterize second target data matching the first target data, the first target data and the plurality of second target data are of different types, and the push data is obtained by matching the first feature vector corresponding to the first target data with the second feature vectors corresponding to the plurality of second target data, the first feature vector being obtained by processing the first target data using a first processing model, and the second feature vector being obtained by processing the first target data using a second processing model. The first and second processing models are obtained by processing the second target data using a processing model. The parameters of the first and second processing models are adjusted based on the global similarity between the first and second global features and the data block similarity between the first and second data block features. The first global features and the first data block features are obtained by processing the first training data using the first processing model. The second global features and the second data block features are obtained by processing the second training data using the second processing model. The first and second training data are different types of data. The first global features are used to characterize the semantic features of the first training data, and the first data block features are used to characterize the features of the data blocks in the first training data. The first and second processing models are machine learning models.

[0198] The processor can access information and applications stored in memory via a transmission device to perform the following steps: displaying first target data on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device; acquiring multiple second target data sets, wherein the first target data and the multiple second target data sets are of different types; processing the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; processing the multiple second target data sets using a second processing model to obtain second feature vectors corresponding to the multiple second target data sets; matching the first feature vector and the second feature vector to determine push data among the multiple second target data sets, wherein the push data is used to characterize the second target data set that matches the first target data set. The system drives VR or AR devices to display and push data. The parameters of the first and second processing models are adjusted based on the global similarity between the first and second global features and the data block similarity between the first and second data block features. The first global features and the first data block features are obtained by processing the first training data through the first processing model. The second global features and the second data block features are obtained by processing the second training data through the second processing model. The first and second training data are different types of data. The first global features are used to characterize the semantic features of the first training data, and the first data block features are used to characterize the features of the data blocks in the first training data. The first and second processing models are machine learning models.

[0199] The processor can invoke information and applications stored in memory via a transmission device to perform the following steps: obtaining a model training request sent by a client by calling a first interface, wherein the first interface includes a first parameter, the value of which is a model training request used to train a first processing model and a second processing model; obtaining training samples based on the model training request, wherein the training samples include: first training data and second training data of different types; processing the first training data using the first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of data blocks in the first training data; processing the second training data using the second processing model to obtain a second global feature and a second data block feature; and performing training based on the global similarity of the first global feature and the second global feature, and the numerical similarity of the first data block feature and the second data block feature. Based on block similarity, the parameters of the first processing model and the second processing model are adjusted; the first processing model and the second processing model are output to the client by calling the second interface, wherein the second interface includes a second parameter, the parameter value of which is the first processing model and the second processing model; wherein the parameters of the first processing model and the second processing model are adjusted based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0200] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: acquiring first target data sent by a client by invoking a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the first target data; acquiring multiple second target data, wherein the first target data and the multiple second target data are of different types; processing the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; processing the multiple second target data using a second processing model respectively to obtain second feature vectors corresponding to the multiple second target data; matching the first feature vector and the second feature vector to determine push data among the multiple second target data, wherein the push data is used to characterize the second target data that matches the first target data; and invoking... The second interface outputs push data to the client. The second interface includes a second parameter, the value of which is the push data. The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model. The second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0201] This application provides a model training scheme. A first processing model processes first training data to obtain first global features and first data block features. A second processing model processes second training data to obtain second global features and second data block features. Finally, based on the global similarity between the first and second global features, and the data block similarity between the first and second data block features, the parameters of the first and second processing models are adjusted to train the feature extraction model. This achieves a more accurate representation of the similarity between images and text, improves model training performance, and further enhances the accuracy of the model in processing multimodal data. This solves the technical problem of poor accuracy in multimodal data processing using models in related technologies. In image and text search scenarios, the first and second processing models can extract more accurate image and text features, making the searched information more accurate and better suited to the user's search needs, thereby improving the accuracy of image and text search.

[0202] Those skilled in the art will understand that Figure 16The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 16 This does not limit the structure of the aforementioned electronic device. For example, computer terminal A may also include components that are more... Figure 16 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 16 The different configurations shown.

[0203] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0204] Example 14

[0205] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the aforementioned computer-readable storage medium can be used to store the program code executed by the model training method and data processing method provided in the above embodiments.

[0206] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a cluster of computer terminals in a computer network, or in any mobile terminal in a cluster of mobile terminals.

[0207] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining training samples, the training samples including: first training data and second training data of different types; processing the first training data using a first processing model to obtain a first global feature and a first data block feature, the first global feature being used to characterize the semantic features of the first training data, and the first data block feature being used to characterize the features of data blocks in the first training data; processing the second training data using a second processing model to obtain a second global feature and a second data block feature; adjusting the parameters of the first processing model and the second processing model based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature, wherein the first processing model and the second processing model are machine learning models.

[0208] Optionally, the aforementioned computer-readable storage medium is further configured to store program code for performing the following steps: constructing a first loss function based on global similarity; constructing a second loss function based on data block similarity; obtaining a weighted sum of the first and second loss functions to obtain a target loss function; and adjusting parameters based on the target loss function.

[0209] Optionally, the aforementioned computer-readable storage medium is further configured to store program code for performing the following steps: obtaining a first global similarity corresponding to a first sample and a second global similarity corresponding to a second sample from the global similarity, wherein the two different types of data contained in the first sample match, and the two different types of data contained in the second sample do not match; and constructing a first loss function based on the first global similarity and the second global similarity.

[0210] Optionally, the aforementioned computer-readable storage medium is further configured to store program code for performing the following steps: obtaining a first data block similarity corresponding to a first sample and a second data block similarity corresponding to a second sample from the data block similarity; and constructing a second loss function based on the first data block similarity and the second data block similarity.

[0211] Optionally, the aforementioned computer-readable storage medium is further configured to store program code for performing the following steps: aligning the features of the first data block and the features of the second data block using a feature alignment module to obtain data block similarity.

[0212] The processor can invoke information and application programs stored in the memory through a transmission device to perform the following steps: acquiring first target data of different types and multiple second target data; processing the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; processing the multiple second target data using a second processing model to obtain second feature vectors corresponding to the multiple second target data; matching the first feature vector and the second feature vector to determine push data among the multiple second target data, wherein the push data is used to represent the second target data that matches the first target data; wherein the parameters of the first processing model and the second processing model are adjusted based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature; the first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model; the first training data and the second training data are different types of data; the first global feature is used to represent the semantic features of the first training data; the first data block feature is used to represent the features of the data blocks in the first training data; and the first processing model and the second processing model are machine learning models.

[0213] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: responding to an input command applied to an operation interface on a client, displaying first target data on the operation interface; responding to a push command applied to the operation interface, displaying push data from a plurality of second target data on the operation interface, wherein the push data is used to characterize second target data matching the first target data, the first target data and the plurality of second target data are of different types, and the push data is obtained by matching a first feature vector corresponding to the first target data with second feature vectors corresponding to the plurality of second target data, the first feature vector being obtained by processing the first target data using a first processing model, and the second feature vector being obtained by... The second processing model is obtained by processing the second target data. The parameters of the first and second processing models are adjusted based on the global similarity between the first and second global features and the data block similarity between the first and second data block features. The first global features and the first data block features are obtained by processing the first training data through the first processing model. The second global features and the second data block features are obtained by processing the second training data through the second processing model. The first and second training data are different types of data. The first global features are used to characterize the semantic features of the first training data, and the first data block features are used to characterize the features of the data blocks in the first training data. The first and second processing models are machine learning models.

[0214] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: displaying first target data on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device; acquiring multiple second target data, wherein the first target data and the multiple second target data are of different types; processing the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; processing the multiple second target data using a second processing model respectively to obtain second feature vectors corresponding to the multiple second target data; matching the first feature vector and the second feature vector to determine push data among the multiple second target data, wherein the push data is used to characterize the second target data that matches the first target data. Data; driving VR or AR devices to display and push data; wherein, the parameters of the first processing model and the second processing model are adjusted based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0215] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining a model training request sent by a client by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is a model training request, and the model training request is used to train a first processing model and a second processing model; obtaining training samples based on the model training request, wherein the training samples include: first training data and second training data of different types; processing the first training data using the first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of data blocks in the first training data; processing the second training data using a second processing model to obtain a second global feature and a second data block feature; adjusting the parameters of the first processing model and the second processing model based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature; and outputting the first processing model and the second processing model to the client by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter is the first processing model and the second processing model, and the first processing model and the second processing model are machine learning models.

[0216] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining first target data sent by the client by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the first target data; obtaining multiple second target data, wherein the first target data and the multiple second target data are of different types; processing the first target data using a first processing model to obtain a first feature vector corresponding to the first target data; processing the multiple second target data using a second processing model respectively to obtain second feature vectors corresponding to the multiple second target data; matching the first feature vector and the second feature vector to determine push data among the multiple second target data, wherein the push data is used to characterize the second target data that matches the first target data; and so on. Data is pushed out by calling a second interface, where the second interface includes a second parameter whose value is the data to be pushed to the client. The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The first processing model and the second processing model are machine learning models.

[0217] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0218] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0219] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0220] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0221] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0222] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0223] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A model training method, characterized in that, include: Obtain training samples, wherein the training samples include: first training data and second training data of different types, wherein the first training data is image data and the second training data is text data; The first training data is processed using a first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The second training data is processed using the second processing model to obtain the second global feature and the second data block feature; The target loss function is determined based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature; If the target loss function is greater than a threshold, the parameters of the first processing model and the second processing model are adjusted until the target loss function is less than the threshold. The first processing model and the second processing model are machine learning models, wherein the model structure of the first processing model and the second processing model are the same, the model accuracy of the first processing model and the second processing model is the same, and the data block similarity is obtained by feature alignment processing of the features of the first data block and the features of the second data block by a feature alignment module.

2. The method according to claim 1, characterized in that, A target loss function is determined based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature; if the target loss function is greater than a threshold, the parameters of the first processing model and the second processing model are adjusted, including: A first loss function is constructed based on the global similarity. A second loss function is constructed based on the similarity of the data blocks; Obtain the weighted sum of the first loss function and the second loss function to obtain the target loss function; The parameters of the first processing model and the second processing model are adjusted based on the target loss function.

3. The method according to claim 2, characterized in that, A first loss function is constructed based on the global similarity, including: Obtain the first global similarity corresponding to the first sample and the second global similarity corresponding to the second sample from the global similarity, wherein the two different types of data contained in the first sample match, and the two different types of data contained in the second sample do not match; The first loss function is constructed based on the first global similarity and the second global similarity.

4. The method according to claim 2, characterized in that, A second loss function is constructed based on the similarity of the data blocks, including: Obtain the first data block similarity corresponding to the first sample and the second data block similarity corresponding to the second sample from the data block similarity; The second loss function is constructed based on the similarity between the first data block and the second data block.

5. A data processing method, characterized in that, include: Acquire different types of primary target data and multiple secondary target data; The first target data is processed using the first processing model to obtain the first feature vector corresponding to the first target data; The second processing model is used to process the multiple second target data respectively to obtain the second feature vectors corresponding to the multiple second target data; The first feature vector and the second feature vector are matched to determine the push data among the plurality of second target data, wherein the push data is used to characterize the second target data that matches the first target data; In this process, the parameters of the first and second processing models are adjusted when the target loss function is greater than a threshold, until the target loss function is less than the threshold. The target loss function is determined based on the global similarity of the first and second global features, and the data block similarity of the first and second data block features. The first global features and the first data block features are obtained by processing the first training data using the first processing model, and the second global features and the second data block features are obtained by processing the second training data using the second processing model. The first and second training data are different types of data: the first training data is image data, and the second training data is text data. The first global features are used to characterize the semantic features of the first training data, and the first data block features are used to characterize the features of data blocks in the first training data. The first and second processing models are machine learning models with the same model structure and the same model accuracy. The data block similarity is obtained by feature alignment processing of the first and second data block features using a feature alignment module.

6. A data processing method, characterized in that, include: In response to input commands applied to the client's user interface, the first target data is displayed on the user interface. In response to a push command applied to the operation interface, push data from multiple second target data sets is displayed on the operation interface. The push data represents second target data that matches the first target data. The first target data and the multiple second target data sets are of different types. The push data is obtained by matching a first feature vector corresponding to the first target data with second feature vectors corresponding to the multiple second target data sets. The first feature vector is obtained by processing the first target data using a first processing model, and the second feature vector is obtained by processing the second target data using a second processing model. The parameters of the first and second processing models are adjusted when the target loss function is greater than a threshold, until the target loss function is less than the threshold. The target loss function is based on the global similarity between the first and second global features, and the data blocks of the first and second data block features. Similarity is determined by processing the first training data using the first processing model with the first global feature and the first data block feature, and processing the second training data using the second processing model with the second global feature and the second data block feature. The first training data and the second training data are different types of data: the first training data is image data, and the second training data is text data. The first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of data blocks in the first training data. The first processing model and the second processing model are machine learning models with the same model structure and the same model accuracy. The data block similarity is obtained by performing feature alignment processing on the first data block feature and the second data block feature using a feature alignment module.

7. A model training method, characterized in that, include: The server obtains the model training request sent by the client by calling a first interface, wherein the first interface includes a first parameter, the value of which is the model training request, and the model training request is used to train a first processing model and a second processing model. The server obtains training samples based on the model training request, wherein the training samples include: first training data and second training data of different types, the first training data being image data and the second training data being text data; The server uses the first processing model to process the first training data to obtain a first global feature and a first data block feature, wherein the first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of the data blocks in the first training data. The server uses the second processing model to process the second training data to obtain the second global feature and the second data block feature; The server determines a target loss function based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. If the target loss function is greater than a threshold, the parameters of the first processing model and the second processing model are adjusted until the target loss function is less than the threshold. The first processing model and the second processing model have the same model structure and the same model accuracy. The data block similarity is obtained by feature alignment processing of the first data block feature and the second data block feature through a feature alignment module. The server outputs the first processing model and the second processing model to the client by calling the second interface, wherein the second interface includes a second parameter, the value of which is the first processing model and the second processing model, and the first processing model and the second processing model are machine learning models.

8. A data processing method, characterized in that, include: The server obtains the first target data sent by the client by calling the first interface, wherein the first interface includes a first parameter and the value of the first parameter is the first target data; The server acquires multiple second target data, wherein the first target data and the multiple second target data are of different types; The server uses a first processing model to process the first target data to obtain a first feature vector corresponding to the first target data; The server uses a second processing model to process the plurality of second target data respectively, and obtains the second feature vectors corresponding to the plurality of second target data; The server matches the first feature vector and the second feature vector to determine the push data among the plurality of second target data, wherein the push data is used to characterize the second target data that matches the first target data; The server outputs the push data to the client by calling a second interface, wherein the second interface includes a second parameter, and the value of the second parameter is the push data; In this process, the parameters of the first and second processing models are adjusted when the target loss function is greater than a threshold, until the target loss function is less than the threshold. The target loss function is determined based on the global similarity of the first and second global features, and the data block similarity of the first and second data block features. The first global features and the first data block features are obtained by processing the first training data using the first processing model, and the second global features and the second data block features are obtained by processing the second training data using the second processing model. The first and second training data are different types of data: the first training data is image data, and the second training data is text data. The first global features are used to characterize the semantic features of the first training data, and the first data block features are used to characterize the features of data blocks in the first training data. The first and second processing models are machine learning models with the same model structure and the same model accuracy. The data block similarity is obtained by feature alignment processing of the first and second data block features using a feature alignment module.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 8.

10. A computer terminal, characterized in that, include: processor; A memory, connected to the processor, is used to provide the processor with instructions to perform the following processing steps: acquiring training samples, wherein the training samples include: first training data and second training data of different types, the first training data being image data and the second training data being text data; processing the first training data using a first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to characterize the semantic features of the first training data, and the first data block feature is used to characterize the features of data blocks in the first training data; processing the second training data using a second processing model to obtain a second global feature and a second data block feature; determining a target loss function based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature; adjusting the parameters of the first processing model and the second processing model if the target loss function is greater than a threshold, until the target loss function is less than the threshold, wherein the first processing model and the second processing model are machine learning models, the first processing model and the second processing model have the same model structure, the first processing model and the second processing model have the same model accuracy, and the data block similarity is obtained by feature alignment processing of the first data block feature and the second data block feature through a feature alignment module.

Images