Neural network model training method and apparatus

Abstract

Embodiments of the present disclosure provide a neural network model training method, device, computer program product and storage medium. The neural network model training method comprises: obtaining a first data set composed of data of a first category and a second data set composed of data of a second category; obtaining anchor data in the data of the first category; calculating a first similarity between each data in the first data set and the anchor data using a first similarity calculation neural network; determining a plurality of positive sample pairs based on a first set of data in the first data set whose first similarity is greater than a first threshold and a second set of data in the second data set corresponding to the data in the first set; and training the neural network using the positive sample pairs. The neural network model training method can effectively improve the problem of limited number of sample data and low accuracy of sample data in the neural network training process, and can improve the training effect of the neural network.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to a method, apparatus, computer program product, and storage medium for training a neural network model. Background Technology

[0002] With the increasing demand for data analysis across industries in the era of big data, the efficient acquisition of knowledge through machine learning has gradually become a major driving force for the development of machine learning technology. How to conduct in-depth analysis of complex and diverse data based on machine learning and utilize information more efficiently has become a major research direction in the current big data environment. Furthermore, in the era of big data, with the continuous acceleration of data generation, the amount of data has grown unprecedentedly, and new types of data requiring analysis are constantly emerging, such as text understanding, sentiment analysis, image retrieval and understanding, and the analysis of graphics and network data. This makes intelligent computing technologies such as big data machine learning and data mining extremely important in the intelligent analysis and processing of big data. Currently, machine learning is widely used in fields such as the internet, finance, and bioinformatics. It is commonly used in scenarios such as data analysis and mining, cross-modal information matching, and pattern recognition. For example, cross-modal information retrieval can be based on machine learning to quickly align and complement different forms of information, thereby acquiring knowledge more comprehensively and accurately. Another example is data classification and matching based on machine learning, providing practical and valuable references for academic research.

[0003] However, many existing neural network training methods for machine learning often suffer from drawbacks such as the huge amount of data required, slow processing speed, and limited analytical accuracy. Therefore, how to train neural networks quickly and effectively in the face of massive amounts of data remains a hot research topic. Summary of the Invention

[0004] To address the aforementioned problems, this disclosure proposes a neural network model training method, comprising: acquiring a first dataset consisting of data of a first category and a second dataset consisting of data of a second category, wherein data in the first dataset corresponds to data in the second dataset; acquiring anchor data in the data of the first category; calculating a first similarity between each data point in the first dataset and the anchor data using a first similarity calculation neural network; for a first set of data in the first dataset whose first similarity is greater than a first threshold, and a second set of data in the second dataset corresponding to the data in the first set, acquiring data in the first set and data in the second set respectively to determine multiple positive sample pairs; iteratively training a first neural network using the positive sample pairs to optimize the network parameters of the first neural network, wherein the first neural network is used to determine the degree of correlation between the data in the first set and the data in the second set. This neural network model training method can improve the accuracy and reliability of neural networks.

[0005] According to an embodiment of this disclosure, the step of calculating the first similarity between each data in the first dataset and the anchor data using a first similarity calculation neural network further includes: extracting the feature matrix of each data in the first dataset using a feature extraction encoder; and calculating the first similarity between each data and the anchor data based on the feature matrix of each data in the first dataset using the first similarity calculation neural network.

[0006] According to an embodiment of this disclosure, the step of obtaining data from the first set and data from the second set of data in the first dataset whose first similarity is greater than a first threshold, and data from the second set of data in the second dataset that correspond to the data in the first set, to determine multiple positive sample pairs further includes: for at least a portion of the data in the first set, combining each data in the at least a portion of the data with multiple data in the second set to form the multiple positive sample pairs.

[0007] According to embodiments of this disclosure, the iterative training of the first neural network using the positive sample pairs further includes: for each of the plurality of positive sample pairs, using a first encoder to perform matrix encoding on the data of a first category in the positive sample pair to obtain a first feature matrix; using a second encoder to perform matrix encoding on the data of a second category in the positive sample pair to obtain a second feature matrix; performing matrix fusion on the first feature matrix and the second feature matrix to obtain a feature matrix of the positive sample pair, thereby obtaining a set of sample feature matrices for the plurality of positive sample pairs; and optimizing the network parameters of the first neural network based on a first objective function and the set of sample feature matrices, wherein the first neural network includes the first encoder and the second encoder.

[0008] According to an embodiment of this disclosure, the neural network model training method further includes: for a third set of data in the first dataset whose first similarity is less than a second threshold, and a fourth set of data in the second dataset corresponding to the data in the third set, respectively obtaining data in the third set and data in the fourth set to determine multiple negative sample pairs; and using the positive sample pairs and the negative sample pairs to iteratively train the first neural network.

[0009] According to embodiments of this disclosure, the first encoder includes a global feature encoder, and the second encoder includes a global feature encoder; or the first encoder includes a local feature encoder, and the second encoder includes a local feature encoder; or the first encoder includes a global feature encoder and a local feature encoder, and the first encoder includes a global feature encoder and a local feature encoder.

[0010] According to embodiments of this disclosure, optimizing the network parameters of the first neural network based on the first objective function and the sample feature matrix set further includes: calculating a second similarity between any two feature matrices in the sample feature matrix set; and optimizing the network parameters of the first neural network based on the first objective function and the calculated second similarity.

[0011] According to embodiments of this disclosure, the first objective function is determined based on at least one of cosine similarity, Euclidean distance, Pearson correlation coefficient, Hamming distance, and Manhattan distance.

[0012] According to embodiments of this disclosure, the first category of data is text data, the second category of data is image data, and the task to be completed includes at least one of: data classification task, image segmentation task, and cross-modal retrieval task, wherein the data classification task is an image classification task or a text classification task, and the cross-modal retrieval task is a text-based image retrieval task or an image-based text retrieval task.

[0013] This disclosure provides a neural network-based data classification method, comprising: acquiring data to be classified, wherein the data to be classified is image data and / or text data; processing the data to be classified using a data classification neural network to obtain classified data; wherein the data classification neural network is determined based on a first neural network and classification training data, and the first neural network is determined using the neural network model training method as described in claim 1, the classification training data includes data belonging to a first category and / or data belonging to a second category, the data in the first category being text data, and the data in the second category being image data.

[0014] This disclosure provides an image segmentation method based on a neural network, comprising: acquiring an image to be segmented; processing the image to be segmented using an image segmentation neural network to obtain an image segmentation result of the image to be segmented; wherein the image segmentation neural network is determined based on a first neural network and image segmentation training data, and the first neural network is determined using the neural network model training method as described in claim 1, and the image segmentation training data includes data belonging to a second category, wherein the data in the second category is image data.

[0015] This disclosure provides a cross-modal retrieval method based on a neural network, comprising: acquiring first data; processing the first data using a cross-modal retrieval neural network to obtain second data corresponding to the first data; wherein the cross-modal retrieval neural network is determined based on a first neural network and cross-modal retrieval training data, and the first neural network is determined using the neural network model training method as described in claim 1, the cross-modal retrieval training data includes data belonging to a first category and data belonging to a second category, the first category data being text data, the second category data being image data, and the first data being image data and the second data being text data, or the first data being text data and the second data being image data.

[0016] Embodiments of this disclosure provide a neural network model training apparatus, comprising: a dataset acquisition module configured to: acquire a first dataset consisting of data of a first category and a second dataset consisting of data of a second category, wherein the data in the first dataset and the data in the second dataset have a corresponding association; an anchor data acquisition module configured to: acquire anchor data in the data of the first category; a similarity calculation module configured to: calculate a first similarity between each data in the first dataset and the anchor data using a first similarity calculation neural network; a sample determination module configured to: acquire data in the first set and data in the second set respectively for a first set of data in the first dataset whose first similarity is greater than a first threshold, and a second set of data in the second dataset corresponding to the data in the first set, to determine multiple positive sample pairs; and a network training module configured to: iteratively train a first neural network using the positive sample pairs to optimize the network parameters of the first neural network, wherein the first neural network is used to acquire the relationship between the data of the first category and the data of the second category.

[0017] Embodiments of this disclosure provide a computer program product including computer software code that, when run by a processor, provides the methods described above.

[0018] Embodiments of this disclosure provide a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, provide the above-described method.

[0019] The neural network model training method disclosed herein effectively addresses the problems of limited sample data quantity and low sample data accuracy during neural network training. By utilizing related datasets to jointly train the neural network model, this disclosure enhances the accuracy and reliability of the neural network. Furthermore, the neural network model training method disclosed herein can be applied to various application scenarios. For example, by applying this neural network model training method to data classification, image segmentation, and cross-modal retrieval, it can effectively improve the speed and accuracy of tasks such as data classification, image segmentation, and cross-modal retrieval. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some exemplary embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0021] In the attached image:

[0022] Figure 1 This is a schematic flowchart illustrating a neural network model training method according to an embodiment of the present disclosure;

[0023] Figure 2 This is a schematic diagram illustrating training samples according to embodiments of the present disclosure;

[0024] Figure 3 This is a schematic diagram illustrating the neural network model training process according to an embodiment of the present disclosure;

[0025] Figure 4 This is a schematic diagram illustrating the implementation process of various tasks based on neural networks according to embodiments of the present disclosure;

[0026] Figures 5A-5C This is a schematic diagram illustrating the inputs and outputs corresponding to different task application scenarios according to embodiments of the present disclosure;

[0027] Figure 6 This is a schematic diagram illustrating the composition of a neural network model training apparatus according to an embodiment of the present disclosure;

[0028] Figure 7 This illustrates the architecture of a computing device according to embodiments of the present disclosure; and

[0029] Figure 8 This is a schematic diagram illustrating a storage medium according to an embodiment of the present disclosure. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0031] Furthermore, in this specification and the accompanying drawings, steps and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted.

[0032] Furthermore, in this specification and accompanying drawings, elements are described in singular or plural forms according to embodiments. However, the singular and plural forms have been suitably chosen for the presented cases merely for ease of explanation and are not intended to limit this disclosure. Thus, a singular form may include a plural form, and a plural form may include a singular form, unless the context clearly indicates otherwise.

[0033] In this specification and accompanying drawings, steps and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these steps and elements are omitted. Furthermore, in the description of this disclosure, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0034] To facilitate the description of this disclosure, the following concepts related to this disclosure are introduced.

[0035] The methods disclosed herein can be based on artificial intelligence (AI). Artificial intelligence is the theory, methods, technology, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. For example, AI-based methods can perform machine learning in a manner similar to human perception, such as training neural networks to match various forms of information, helping people quickly find the information they need from large amounts of data. AI-based information matching methods can provide effective support for data analysis and processing (e.g., data classification, image segmentation, cross-modal retrieval) in the context of the big data era.

[0036] In summary, the solutions provided by the embodiments of this disclosure involve computer technologies such as artificial intelligence, neural networks, big data, and information processing. The embodiments of this disclosure will be further described below with reference to the accompanying drawings.

[0037] Figure 1 This is a schematic flowchart 100 illustrating a neural network model training method according to an embodiment of the present disclosure.

[0038] In step S110, a first dataset consisting of data of a first category and a second dataset consisting of data of a second category are obtained, wherein the data in the first dataset and the data in the second dataset are associated with each other.

[0039] According to embodiments of this disclosure, the first category of data and the second category of data can be one of image data, text data, audio data, and video data, respectively, and the first category of data and the second category of data are different (for example, the first category of data is audio and the second category of data is image; the first category of data is text and the second category of data is audio; the first category of data is text and the second category of data is video, etc.). Optionally, audio data can be converted into text data for further processing, and video data can be converted into one or more combinations of image data, audio data, and text data for further processing.

[0040] In step S120, anchor data from the first category of data is obtained.

[0041] Optionally, the anchor data can be data from the first dataset, or it can be data from other first categories that are not in the first dataset. Typically, anchor data is representative data or data with high similarity to data in the first dataset. Since data from the first category is selected as anchor data, the processing complexity of the first category is less than that of the second category, making the processing simpler (e.g., selecting text data as the first category and image data as the second category).

[0042] To simplify data processing, data that is easier to process can be classified as the first category, and data that is more complex to process can be classified as the second category (for example, text data as the first category and image data as the second category).

[0043] In step S130, a first similarity calculation neural network is used to calculate the first similarity between each data in the first dataset and the anchor data.

[0044] According to embodiments of this disclosure, a feature extraction encoder can be used to extract feature matrices from each data point in the first dataset; then, a first similarity calculation neural network is used to calculate a first similarity between each data point and the anchor data based on the feature matrices of each data point in the first dataset. Depending on the data in the first dataset to be processed, the form of the feature extraction encoder may differ (e.g., different types of text encoders, different types of image encoders, etc.).

[0045] In step S140, for the first set of data in the first dataset whose first similarity is greater than the first threshold, and the second set of data in the second dataset that corresponds to the data in the first set, the data in the first set and the data in the second set are respectively obtained to determine multiple positive sample pairs.

[0046] Optionally, for at least a portion of the data in the first set, each data point in the at least a portion of the data can be combined with multiple data points in the second set to form the multiple positive sample pairs. Alternatively, for each data point in the first set, that data point can be combined with each data point in the second set to form multiple positive sample pairs. For example, for a first set containing M elements (M is a positive integer), a maximum of (M×M) sample pairs can be formed.

[0047] In step S150, the first neural network is iteratively trained using the positive sample pairs to optimize the network parameters of the first neural network, wherein the first neural network is used to determine the degree of correlation between the data of the first set and the data of the second set.

[0048] According to embodiments of this disclosure, for each of the plurality of positive sample pairs, a first encoder can be used to matrix-encode the data of the first category in the positive sample pair to obtain a first feature matrix; a second encoder can be used to matrix-encode the data of the second category in the positive sample pair to obtain a second feature matrix; the first feature matrix and the second feature matrix are fused to obtain the feature matrix of the positive sample pair, thereby obtaining a set of sample feature matrices for the plurality of positive sample pairs. Based on a first objective function and the set of sample feature matrices, the network parameters of the first neural network are optimized, wherein the first neural network includes the first encoder and the second encoder.

[0049] Optionally, the process of fusing the first feature matrix and the second feature matrix can be implemented based on a fusion neural network, which is used to fuse the first feature matrix and the second feature matrix to obtain the feature matrix of the positive sample pair.

[0050] Optionally, the process of fusing the first feature matrix and the second feature matrix can also be implemented through matrix multiplication. For example, each row vector of the first feature matrix is ​​multiplied sequentially by each column vector of the second feature matrix to obtain the corresponding row vector in the fused feature matrix. Specifically, when performing matrix multiplication by multiplying each row vector of the first feature matrix by each column vector of the second feature matrix, the number of columns in the first feature matrix is ​​equal to the number of rows in the second feature matrix. Alternatively, each row vector of the second feature matrix can be multiplied sequentially by each column vector of the first feature matrix to obtain the corresponding row vector in the fused feature matrix. Specifically, when performing matrix multiplication by multiplying each row vector of the second feature matrix by each column vector of the first feature matrix, the number of columns in the second feature matrix is ​​equal to the number of rows in the first feature matrix.

[0051] According to embodiments of this disclosure, in order to increase the diversity and comprehensiveness of samples, thereby improving the training effect of the first neural network, for a third set of data in the first dataset whose first similarity is less than a second threshold, and a fourth set of data in the second dataset corresponding to the data in the third set, data from the third set and data from the fourth set can be obtained respectively to determine multiple negative sample pairs; and the first neural network can be iteratively trained using the positive sample pairs and the negative sample pairs. Optionally, for each data in the third set, the data can be combined with each data in the fourth set to form multiple negative sample pairs. For example, for a third set containing K elements (K is a positive integer), at most (K×K) negative sample pairs can be formed.

[0052] Therefore, when iteratively training the first neural network using the positive and negative sample pairs, for each sample pair consisting of the plurality of negative and positive sample pairs, a first encoder can be used to matrix-encode the data of the first category in the sample pair to obtain a first feature matrix; a second encoder can be used to matrix-encode the data of the second category in the sample pair to obtain a second feature matrix; the first and second feature matrices are then fused to obtain the feature matrix of the sample pair, thereby obtaining a set of sample feature matrices for the plurality of sample pairs. Based on a first objective function and the set of sample feature matrices, the network parameters of the first neural network are optimized, wherein the first neural network includes the first encoder and the second encoder. Optionally, the process of fusing the first and second feature matrices can be implemented based on a fusion neural network, which is used to fuse the first and second feature matrices to obtain the feature matrix of the sample pair.

[0053] According to embodiments of this disclosure, both the first encoder and the second encoder can have various forms. For example, the first encoder may include a global feature encoder, and the second encoder may include a global feature encoder; or the first encoder may include a local feature encoder, and the second encoder may include a local feature encoder; or the first encoder may include both a global feature encoder and a local feature encoder.

[0054] According to embodiments of this disclosure, a second similarity between any two feature matrices in the sample feature matrix set can also be calculated; then, based on the first objective function and the calculated second similarity, the network parameters of the first neural network are optimized.

[0055] It should be noted that the first objective function can be determined based on various similarity functions (e.g., cosine similarity, Euclidean distance, Pearson correlation coefficient, Hamming distance, Manhattan distance, etc.). Optionally, the first objective function can be determined based on one similarity function, or it can be determined based on multiple similarity functions (e.g., a weighted average of multiple similarity functions).

[0056] According to embodiments of this disclosure, after training the first neural network is completed, a second neural network can be constructed based on the trained first neural network according to the task to be completed; and the network parameters of the second neural network can be optimized based on the second objective function determined according to the task to be completed.

[0057] For example, if the first category of data is text data and the second category of data is image data, the task to be completed includes at least one of the following: data classification task, image segmentation task, and cross-modal retrieval task, wherein the data classification task is an image classification task or a text classification task, and the cross-modal retrieval task is a text-based image retrieval task or an image-based text retrieval task.

[0058] More specifically, when the task to be completed is a data classification task, the second neural network can be iteratively trained using classification training data to optimize the network parameters of the second neural network, wherein the classification training data includes data belonging to the first category and / or data belonging to the second category.

[0059] When the task to be completed is an image segmentation task, the second neural network can be iteratively trained using image segmentation training data to optimize the network parameters of the second neural network, wherein the image segmentation training data includes data belonging to the second category.

[0060] When the task to be completed is a cross-modal retrieval task, the second neural network can be iteratively trained using cross-modal retrieval training data to optimize the network parameters of the second neural network. The cross-modal retrieval training data includes data belonging to the first category and data belonging to the second category.

[0061] It should be understood that the data in the first and second categories are not limited to text data and image data. Therefore, the neural network training method of this disclosure can also be applied to tasks involving various data types. For example, the method of this disclosure can be applied to application scenarios such as video and / or audio classification, video and / or audio segmentation, and text-based video and / or audio retrieval.

[0062] Figure 2 This is a schematic diagram illustrating training samples according to an embodiment of the present disclosure.

[0063] Taking medical reports and their corresponding medical images as an example, for the same type of disease, there is a very high degree of similarity between different medical reports and medical images. Figure 2 (Text displayed using the same text format has the same meaning). If only one-to-one corresponding medical reports and medical image pairs are used as positive samples for neural network training, the training data will be insufficient. Furthermore, some medical reports and images with high similarity will be incorrectly pushed apart in the joint embedding space, which will adversely affect neural network training.

[0064] Therefore, a medical report can be selected as the anchor data. Figure 2 The data (A) is then compared with other medical reports to calculate the similarity between each medical report and the anchor data. Optionally, the calculation of the similarity between the medical reports can be based on a neural network. For example, a text encoder can be used to extract the feature matrix of each medical report in the medical report set; then, a neural network can be used to calculate the similarity between each medical report and the anchor data based on the feature matrix of each medical report in the medical report set.

[0065] Optionally, multiple positive sample pairs can be determined by acquiring data from the first set and the second set, respectively, based on a first set of medical reports in the medical report set whose similarity to the anchor data is greater than a first threshold, and a second set of medical images in the medical image set corresponding to the medical reports in the first set. For example, for at least a portion of the medical reports in the first set, each medical report in the at least a portion of the medical reports can be combined with multiple medical images in the second set to form multiple positive sample pairs. Then, the positive sample pairs can be used to iteratively train a first neural network, where the first neural network is used to determine the degree of correlation between the medical reports in the first set and the medical images in the second set.

[0066] Optionally, for a third set of medical reports in the medical report set whose similarity to the anchor data is less than a second threshold, and a fourth set of medical images in the medical image set corresponding to the medical reports in the third set, data from the third set and data from the fourth set can be obtained respectively to determine multiple negative sample pairs. For example, for at least a portion of the medical reports in the third set, each medical report in the at least a portion of the medical reports can be combined with multiple medical images in the fourth set to form multiple negative sample pairs. Then, the first neural network can be iteratively trained using the positive sample pairs and negative sample pairs, wherein the first neural network is used to determine the degree of correlation between the medical reports in the first set and the medical images in the second set.

[0067] Optionally, multiple neutral sample pairs can be determined based on a fifth set of medical reports in the medical report set that are between a first threshold and a second threshold, and a sixth set of medical images in the medical image set that correspond to the medical reports in the fifth set. Optionally, the neutral sample pairs do not participate in the training process of the first neural network.

[0068] for Figure 2 In the example, assuming the similarity is η, when 0.97 < η ≤ 1, the medical image and medical report constitute a positive sample pair; when 0 < η ≤ 0.96, the medical image and medical report constitute a negative sample pair; and when 0.96 < η ≤ 0.97, the medical image and medical report constitute a neutral sample pair. That is, Figure 2 Sample (B) in the sample is a positive sample pair. Figure 2 The (C) sample in the text is a negative sample pair. Figure 2 The (D) sample in the sample is a neutral sample pair.

[0069] This sample processing method can effectively improve the problems of limited sample data quantity, low sample data accuracy, and mutual influence between different samples during neural network training, which may adversely affect the training results.

[0070] Figure 3 This is a schematic diagram illustrating the training process of a neural network model according to an embodiment of the present disclosure.

[0071] like Figure 3 As shown, multiple (N, where N is a positive integer) text reports constitute the first dataset, and multiple (N) images constitute the second dataset. The text reports in the first dataset correspond to the images in the second dataset (e.g., each medical image has a corresponding medical report). Since processing text reports is simpler than processing images, text report data can be selected as anchor data, and a text first similarity calculation neural network is used to calculate the first similarity between each text report in the first dataset and the anchor data.

[0072] Based on a first set of data in a first dataset whose similarity is greater than a first threshold, and a second set of data in a second dataset corresponding to the data in the first set, multiple positive sample pairs can be identified (e.g., in...). Figure 3 In the example, let the first similarity be η, and when 0.97 < η ≤ 1, a positive sample pair is determined. Optionally, for at least a portion of the data in the first set, each data point in the at least a portion of the data can be combined with multiple data points in the second set to form multiple positive sample pairs.

[0073] Similarly, based on a third set of data in the first dataset whose first similarity is less than a second threshold, and a fourth set of data in the second dataset corresponding to the data in the third set, multiple negative sample pairs can be identified (e.g., in...). Figure 3 In the example, when 0 < η ≤ 0.96, a negative sample pair is determined. Alternatively, for each data point in the third set, the data point can be combined with each data point in the fourth set to form multiple negative sample pairs.

[0074] After determining the positive and negative sample pairs, the first neural network can be iteratively trained using the positive and negative sample pairs to optimize its network parameters. The first neural network is used to determine the degree of correlation between the data in the first set (text report data) and the data in the second set (image data).

[0075] More specifically, for each of a plurality of sample pairs consisting of a plurality of negative sample pairs and a plurality of positive sample pairs, the image encoder can be used to process the image data in that sample pair. Figure 3 In this context, I represents the image. The image is then matrix-encoded to obtain the first feature matrix; the text report data (in this case, the text encoder) from the sample pair is then used. Figure 3In the text report (where T represents the text report), the data is matrix-encoded to obtain the second feature matrix. It should be noted that the first neural network includes a first encoder and a second encoder, and both the first encoder and the second encoder can include one or more encoders. The first encoder and the second encoder can include various different types of encoders. For example, the first encoder may include a global feature encoder, and the second encoder may include a global feature encoder; or the first encoder may include a local feature encoder, and the second encoder may include a local feature encoder; or the first encoder may include both a global feature encoder and a local feature encoder (e.g., ...). Figure 3 In this context, g can represent global features; L can represent local features. According to embodiments of this disclosure, in the medical field, for text report data, the first encoder can be implemented based on densely connected convolutional networks (DenseNet), residual neural networks (ResNet), etc.; for image data, the second encoder can be implemented based on bioclinical BERT, bio-language representation models (BioBERT), etc.

[0076] By fusing the first feature matrix and the second feature matrix, the feature matrix of the sample pair can be obtained (for example, by fusing the first feature matrix L). T Second characteristic matrix L I Obtain the sample feature matrix L; then fuse the first feature matrix g. T Second characteristic matrix g I The sample feature matrix g is obtained, and thus a set of sample feature matrices is obtained for the multiple sample pairs.

[0077] Next, based on the first objective function and the sample feature matrix set, the network parameters of the first neural network can be optimized. More specifically, a second similarity between any two feature matrices in the sample feature matrix set can be calculated; and based on the first objective function and the calculated second similarity, the network parameters of the first neural network are optimized. The first objective function can be determined based on similarity calculation functions such as cosine similarity, Euclidean distance, Pearson correlation coefficient, Hamming distance, and Manhattan distance. That is, the first objective function... It can be represented as:

[0078]

[0079] Among them, the positive sample similarity matrix set is The set of negative sample similarity matrices is Local features or global features placeholder, Local features or global features The placeholder is ; sim is the placeholder for the corresponding calculation of global or local feature similarity function, and i, j, k are the sample numbers.

[0080] According to embodiments of this disclosure, in the field of medical report and medical image processing, the first neural network can be implemented based on frameworks such as the Contrastive Learning Framework for Medical Visual Representations in Paired Images and Text (ConVIRT) and the Multimodal Global-Local Representation Learning Framework for Label-Efficient Medical Image Recognition (GLoRIA).

[0081] For example, in the context of medical report and medical image processing, after identifying multiple positive and negative sample pairs composed of medical reports and medical images using the method described above, a first neural network can be constructed based on the ConVIRT framework. This first neural network includes a global medical image encoder (e.g., a medical image encoder based on ResNet) and a global medical report encoder (e.g., a medical report encoder based on BioClinicalBERT). By using the global medical image encoder to perform matrix encoding on the medical images, multiple global image feature matrices g corresponding to multiple medical images can be obtained. I By using a global medical report encoder to perform matrix encoding on medical reports, multiple global report feature matrices g corresponding to multiple medical reports can be obtained. T By analyzing multiple global image feature matrices g I and multiple global reporting feature matrices g T Matrix fusion can yield the global feature matrix g of medical images and medical reports.

[0082] Iterative training and optimization of the first neural network based on the first objective function allows it to learn global features and obtain the correlation between medical images and medical reports. For the first neural network built on the ConVIRT framework, the first objective function... It can be:

[0083]

[0084] Where τ is the temperature coefficient used for scaling, <> is the cosine similarity calculation function, and i and j are the sample numbers.

[0085] In the context of medical report and medical image processing, after identifying multiple positive and negative sample pairs composed of medical reports and medical images using the method described above, a first neural network can be constructed based on the GLoRIA framework. This first neural network includes a medical image encoder (e.g., a medical image encoder implemented based on ResNet) and a medical report encoder (e.g., a medical report encoder implemented based on BioClinicalBERT). Compared to the first neural network constructed based on the ConVIRT framework, the medical image encoder and medical report encoder of the first neural network constructed based on the GLoRIA framework encode not only global features but also local features.

[0086] That is, by using a global medical image encoder to perform matrix encoding on medical images, multiple global image feature matrices g corresponding to multiple medical images can be obtained. I By using a global medical report encoder to perform matrix encoding on medical reports, multiple global report feature matrices g corresponding to multiple medical reports can be obtained. T By utilizing a fusion neural network to analyze multiple global image feature matrices g I and multiple global reporting feature matrices g T Matrix fusion can yield the global feature matrix g of medical images and medical reports.

[0087] By using a local medical image encoder to perform matrix encoding on medical images, multiple local image feature matrices L corresponding to multiple medical images can be obtained. I By using a local medical report encoder to perform matrix encoding on medical reports, multiple local report feature matrices L corresponding to multiple medical reports can be obtained. T By utilizing a fusion neural network to analyze multiple local image feature matrices L I and multiple local report feature matrices L T Matrix fusion can yield the local feature matrix L of medical images and medical reports.

[0088] Iterative training and optimization of the first neural network based on the first objective function allows it to learn by comparing global and local features to obtain the correlation between medical images and medical reports. For the first neural network built on the GLORIA framework, the first objective function... It can be:

[0089]

[0090] Where τ is a temperature coefficient used for scaling, <> is the cosine similarity calculation function, and i and j are the sample numbers. This represents the local feature similarity.

[0091] Since the target image (e.g., an image of the tissue or organ to be analyzed) usually only occupies a portion of a medical image, in order to enhance the alignment of local features and make the keywords in the medical report correspond to sub-regions in the medical image, the medical image features can be converted into word-based attention representations.

[0092] Assume the local image feature matrix output by the intermediate convolutional layers of the ResNet network is C is the feature dimension, M is the number of sub-regions, and the local reporting feature matrix. If the matrix consists of K words with feature dimension C, then the local image feature matrix can be represented as:

[0093]

[0094] in,

[0095] Optionally, The following formula can be used to calculate:

[0096]

[0097] Where σ is the scaling coefficient and k is the number of the local feature matrix.

[0098] Based on the first objective function and sample feature matrix set described above, the first neural network can be iteratively trained to optimize its network parameters. When the number of iterations reaches a predetermined upper limit, or when the training effect of the first neural network meets the requirements (e.g., by inputting test samples into the first neural network to test whether the accuracy of the first neural network's analysis of the test samples meets the requirements), the training of the first neural network can be considered complete. The first neural network can be used to determine the degree of correlation between the data in the first set and the data in the second set (e.g., in the above-mentioned medical report and medical image processing scenario, the first neural network can be used to determine the degree of correlation between medical reports and medical images).

[0099] Figure 4 This is a schematic diagram illustrating the implementation process of various tasks based on neural networks according to embodiments of the present disclosure.

[0100] for Figure 3In the embodiments described, the first neural network trained can be further used to construct a second neural network based on the trained first neural network, according to the task to be completed. Furthermore, a second training process is performed based on a specific objective function determined according to the task to be completed, to optimize the network parameters of the second neural network. For example, when the first category of data is text data and the second category of data is image data, the task to be completed may include at least one of: a data classification task, an image segmentation task, and a cross-modal retrieval task. More specifically, the data classification task may be an image classification task or a text classification task, and the cross-modal retrieval task may be a text-based image retrieval task or an image-based text retrieval task.

[0101] For data classification tasks, the processing computer can acquire the data to be classified, and then use a data classification neural network to process the data to obtain classified data. Optionally, the data to be classified can be input into the processing computer through terminal A, and the classified data can be output through terminal B. Terminal A and terminal B can be the same or different. Furthermore, the processing of the data to be classified can be performed either on terminal A and / or terminal B, or it can be implemented by other terminals or servers different from terminal A and terminal B (for example, after receiving the data to be classified, terminal A can send the data to be classified to the server for processing. After processing the data to be classified, the server obtains the data classification result and provides the data classification result to terminal B).

[0102] More specifically, the processing computer can acquire image data to be classified, and then use a data classification neural network to process the image data to obtain classified image data. In this case, the data classification neural network is determined based on a first neural network and classification training data, which includes image data.

[0103] Optionally, the processing computer can also acquire the text data to be classified, and then use a data classification neural network to process the image data to be classified to obtain the classified text data. In this case, the data classification neural network is determined based on the first neural network and the classification training data, which includes text data.

[0104] Optionally, the processing computer can also simultaneously acquire the image data to be classified and the corresponding text data, and then use a data classification neural network to process the image data to be classified and the corresponding text data to obtain classified image data and / or text data. In this case, the data classification neural network is determined based on the first neural network and classification training data, which includes image data and text data. For example, for image data and the corresponding text data, the data classification network can be used to classify the text data, and then the classification result of the image data can be obtained based on the classification result of the text data, thereby reducing the computational load of classifying the image data.

[0105] For image segmentation tasks, the processing computer can acquire the image to be segmented; then, it uses an image segmentation neural network to process the image to obtain the image segmentation result. In this case, the image segmentation neural network is determined based on a first neural network and image segmentation training data, which includes image data.

[0106] Optionally, the image to be segmented can be input into the processing computer via terminal A, and the image segmentation result can be output via terminal B. Terminal A and terminal B can be the same or different. Furthermore, the processing of the image to be segmented can be performed on terminal A and / or terminal B, or it can be implemented by other terminals or servers different from terminal A and terminal B (for example, after receiving the image to be segmented, terminal A can send the image to be segmented to the server for processing. After processing the image to be segmented, the server obtains the image segmentation result and provides the image segmentation result to terminal B).

[0107] For cross-modal retrieval tasks, the processing computer can acquire first data; then, it can process the first data using a cross-modal retrieval neural network to obtain second data corresponding to the first data; wherein, the cross-modal retrieval neural network is determined based on the first neural network and cross-modal retrieval training data. According to embodiments of this disclosure, text can be retrieved based on images, or images can be retrieved based on text. That is, the first data is image data and the second data is text data, or the first data is text data and the second data is image data.

[0108] Optionally, the image or text to be retrieved can be input into the processing computer via terminal A, and the retrieval result can be output via terminal B. Terminal A and terminal B can be the same or different. Furthermore, the processing of the image or text to be retrieved can be performed on terminal A and / or terminal B, or it can be implemented by other terminals or servers different from terminal A and terminal B (for example, after receiving the image or text to be retrieved, terminal A can send it to a server for processing. After processing the image or text, the server obtains the retrieval result and provides it to terminal B).

[0109] based on Figure 4 The processing procedure shown is as follows: Figure 5A This is a schematic diagram illustrating the input and output corresponding to a data classification task application scenario according to an embodiment of the present disclosure.

[0110] Taking medical image classification as an example, a medical image to be classified can be input into terminal A, and then the classification result of the medical image can be determined based on a neural network. The classification result can then be output on terminal B to assist in diagnosis. Optionally, multiple images can be classified separately based on a neural network to quickly obtain corresponding analysis results for a large number of medical images.

[0111] Figure 5B This is a schematic diagram illustrating the input and output of an image segmentation task application scenario according to an embodiment of the present disclosure.

[0112] Taking medical image segmentation as an example, a medical image to be segmented can be input through terminal A, and then the image segmentation result can be determined based on a neural network. The image segmentation result can then be output on terminal B, thereby quickly identifying the lesion region and assisting in diagnosis. Optionally, multiple images can be segmented based on a neural network to quickly obtain image segmentation results for a large number of medical images.

[0113] Figure 5C This is a schematic diagram illustrating the input and output corresponding to a cross-modal retrieval task application scenario according to an embodiment of the present disclosure.

[0114] Taking the task of retrieving medical images and medical reports as an example, the medical image to be retrieved can be input into terminal A, and then the corresponding medical report retrieval result can be determined based on a neural network. This retrieval result can then be output into terminal B, thereby assisting in diagnosis. Alternatively, the medical report to be retrieved can be input into terminal A, and then the corresponding medical image retrieval result can be determined based on a neural network. This retrieval result can then be output into terminal B, thereby assisting in diagnosis.

[0115] Figure 6 This is a schematic diagram illustrating the composition of a neural network model training apparatus according to an embodiment of the present disclosure.

[0116] According to embodiments of the present disclosure, the neural network model training device 600 may include: a dataset acquisition module 610, an anchor data acquisition module 620, a similarity calculation module 630, a sample determination module 640, and a network training module 650.

[0117] The dataset acquisition module 610 can be configured to acquire a first dataset consisting of data of a first category and a second dataset consisting of data of a second category, wherein the data in the first dataset and the data in the second dataset have a corresponding relationship.

[0118] Optionally, the data in the first category and the data in the second category can be one of image data, text data, audio data, and video data, respectively, and the data in the first category and the data in the second category are different.

[0119] The anchor data acquisition module 620 can be configured to acquire anchor data from the data of the first category.

[0120] Optionally, the anchor data can be data from the first dataset, or it can be data from other first categories that are not in the first dataset. Typically, the anchor data is representative data or data with a high degree of similarity to the data in the first dataset.

[0121] The similarity calculation module 630 is configured to: calculate the first similarity between each data in the first dataset and the anchor data using a first similarity calculation neural network.

[0122] Optionally, a feature extraction encoder can be used to extract the feature matrix of each data in the first dataset; a neural network can be used to calculate the first similarity between each data and the anchor data based on the feature matrix of each data in the first dataset.

[0123] The sample determination module 640 can be configured to: for a first set of data in the first dataset whose first similarity is greater than a first threshold, and a second set of data in the second dataset that corresponds to the data in the first set, respectively obtain the data in the first set and the data in the second set to determine multiple positive sample pairs.

[0124] Optionally, for at least a portion of the data in the first set, each data point in the at least a portion of the data can be combined with multiple data points in the second set to form multiple positive sample pairs.

[0125] According to an embodiment of this disclosure, the sample determination module 640 can also be configured to: for a third set of data in the first dataset whose first similarity is less than a second threshold, and a fourth set of data in the second dataset that corresponds to the data in the third set, respectively obtain the data in the third set and the data in the fourth set to determine multiple negative sample pairs.

[0126] Optionally, for at least a portion of the data in the third set, each data point in the at least a portion of the data can be combined with multiple data points in the fourth set to form multiple negative sample pairs.

[0127] The network training module 650 can be configured to iteratively train a first neural network using the positive sample pairs to optimize the network parameters of the first neural network, wherein the first neural network is used to obtain the relationship between the data of the first category and the data of the second category.

[0128] Optionally, for each sample pair, the data of the first category in the sample pair can be matrix-encoded using a first encoder to obtain a first feature matrix; the data of the second category in the sample pair can be matrix-encoded using a second encoder to obtain a second feature matrix; the first feature matrix and the second feature matrix can be fused to obtain the feature matrix of the sample pair, thereby obtaining a set of sample feature matrices for the multiple sample pairs. Based on the first objective function and the set of sample feature matrices, the network parameters of the first neural network are optimized, wherein the first neural network includes the first encoder and the second encoder.

[0129] Optionally, a second similarity can be calculated between any two feature matrices in the sample feature matrix set; and the network parameters of the first neural network can be optimized based on the first objective function and the calculated second similarity. The first objective function can be determined based on various similarity functions.

[0130] It should be understood that both the first encoder and the second encoder can have multiple forms. For example, the first encoder may include a global feature encoder and the second encoder may include a global feature encoder; or the first encoder may include a local feature encoder and the second encoder may include a local feature encoder; or the first encoder may include both a global feature encoder and a local feature encoder.

[0131] According to embodiments of this disclosure, the neural network model training device 600 can also be applied to various application task scenarios, such as data classification tasks, image segmentation tasks, and cross-modal retrieval tasks. When training a neural network for a specific application task, it is only necessary to construct a second neural network based on the trained first neural network according to the task to be completed; and optimize the network parameters of the second neural network based on the second objective function determined according to the task to be completed.

[0132] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0133] For example, the method or apparatus according to embodiments of this disclosure can also be used by means of Figure 7 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 7 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the methods provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 7 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 7 One or more components in the computing device shown.

[0134] According to another aspect of this disclosure, a computer-readable storage medium is also provided. Figure 8 A schematic diagram 4000 of a storage medium according to the present disclosure is shown.

[0135] like Figure 8As shown, computer-readable instructions 4010 are stored on computer storage medium 4020. When the computer-readable instructions 4010 are executed by a processor, the methods according to embodiments of the present disclosure described with reference to the above figures can be performed. The computer-readable storage medium in the embodiments of the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0136] Embodiments of this disclosure also provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method according to an embodiment of this disclosure.

[0137] In summary, embodiments of this disclosure provide a neural network model training method, apparatus, computer program product, and storage medium. The neural network model training method proposed in this disclosure includes: acquiring a first dataset composed of data of a first category and a second dataset composed of data of a second category, wherein data in the first dataset corresponds to data in the second dataset; acquiring anchor data in the data of the first category; calculating a first similarity between each data point in the first dataset and the anchor data using a first similarity calculation neural network; for a first set of data in the first dataset whose first similarity is greater than a first threshold, and a second set of data in the second dataset corresponding to the data in the first set, acquiring data in the first set and data in the second set respectively to determine multiple positive sample pairs; iteratively training a first neural network using the positive sample pairs to optimize the network parameters of the first neural network, wherein the first neural network is used to determine the degree of correlation between the data in the first set and the data in the second set.

[0138] The neural network model training method disclosed herein effectively addresses the problems of limited sample data quantity and low sample data accuracy during neural network training. By utilizing related datasets to jointly train the neural network model, this disclosure enhances the accuracy and reliability of the neural network. Furthermore, the neural network model training method disclosed herein can be applied to various application scenarios. For example, by applying this neural network model training method to data classification, image segmentation, and cross-modal retrieval, it can effectively improve the speed and accuracy of tasks such as data (including text, images, audio, and video) classification, image segmentation, and cross-modal retrieval (e.g., retrieving images, audio, or video by text, retrieving text or video by images, and retrieving text or images by audio).

[0139] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] This disclosure uses specific terms to describe embodiments of the present disclosure. Terms such as "first / second embodiment," "an embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of the present disclosure. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, certain features, structures, or characteristics in one or more embodiments of the present disclosure can be appropriately combined.

[0141] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in a common dictionary shall be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning, unless expressly defined herein.

[0142] The foregoing description is illustrative of the invention and should not be construed as limiting it. Although several exemplary embodiments of the invention have been described, those skilled in the art will readily understand that many modifications can be made to the exemplary embodiments without departing from the novel teachings and advantages of the invention. Therefore, all such modifications are intended to be included within the scope of the invention as defined in the claims. It should be understood that the foregoing description is illustrative of the invention and should not be construed as limiting it to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The invention is defined by the claims and their equivalents.

Claims

1. A method for training a neural network model, comprising: Obtain a first dataset consisting of data of a first category and a second dataset consisting of data of a second category, wherein the data of the first category and the data of the second category are respectively one of image data, text data, audio data, and video data, and the data of the first category and the data of the second category are different, and the data in the first dataset and the data in the second dataset are associated accordingly; Retrieve anchor data from the data in the first category; The first similarity between each data point in the first dataset and the anchor data is calculated using a first similarity calculation neural network. For a first set of data in the first dataset whose first similarity is greater than a first threshold, and a second set of data in the second dataset that corresponds to the data in the first set, data in the first set and data in the second set are respectively obtained to determine multiple positive sample pairs, wherein the multiple positive sample pairs are obtained by combining each data in at least a portion of the data in the first set with multiple data in the second set respectively; The first neural network is iteratively trained using the positive samples to optimize its network parameters, wherein the first neural network is used to determine the degree of correlation between the data in the first set and the data in the second set.

2. The neural network model training method as described in claim 1, wherein, The step of using a first similarity calculation neural network to calculate the first similarity between each data point in the first dataset and the anchor data includes: The feature matrix of each data point in the first dataset is extracted using a feature extraction encoder. Using the first similarity calculation neural network, based on the feature matrix of each data in the first dataset, the first similarity between each data and the anchor data is calculated.

3. The neural network model training method as described in claim 1, wherein, The first neural network includes a first encoder and a second encoder, and the iterative training of the first neural network using the positive sample pairs includes: For each of the plurality of positive sample pairs The first encoder is used to perform matrix encoding on the data of the first category in the positive sample pair to obtain the first feature matrix; The second encoder is used to matrix-encode the data of the second category in the positive sample pair to obtain the second feature matrix; The first feature matrix and the second feature matrix are fused to obtain the feature matrix of the positive sample pair, thereby obtaining a set of sample feature matrices for the multiple positive sample pairs; Based on the first objective function and the sample feature matrix set, the network parameters of the first neural network are optimized.

4. The neural network model training method as described in claim 1, further comprising: For the third set of data in the first dataset whose first similarity is less than the second threshold, and the fourth set of data in the second dataset that corresponds to the data in the third set, the data in the third set and the data in the fourth set are respectively obtained to determine multiple negative sample pairs; and, The first neural network is iteratively trained using the positive sample pairs and the negative sample pairs.

5. The neural network model training method as described in claim 4, wherein, The step of determining multiple negative sample pairs by acquiring data from the third set of data in the first dataset whose first similarity is less than the second threshold, and the fourth set of data in the second dataset corresponding to the data in the third set, respectively, further includes: for at least a portion of the data in the third set... Each data point in the at least one portion of the data is combined with multiple data points in the fourth set to form the multiple negative sample pairs.

6. The neural network model training method as described in claim 4, wherein, The first neural network includes a first encoder and a second encoder, and the iterative training of the first neural network using the positive sample pairs and the negative sample pairs further includes: For each of the multiple sample pairs consisting of the multiple negative sample pairs and the multiple positive sample pairs, The first encoder is used to perform matrix encoding on the data of the first category in the sample pair to obtain the first feature matrix; The second encoder is used to matrix-encode the data of the second category in the sample pair to obtain the second feature matrix; Matrix fusion is performed on the first feature matrix and the second feature matrix to obtain the feature matrix of the sample pair, thereby obtaining a set of sample feature matrices for the multiple sample pairs. Based on the first objective function and the sample feature matrix set, the network parameters of the first neural network are optimized.

7. The neural network model training method as described in claim 3 or 6, wherein, The first encoder includes a global feature encoder, and the second encoder includes a global feature encoder; or The first encoder includes a local feature encoder, and the second encoder includes a local feature encoder; or The first encoder includes a global feature encoder and a local feature encoder, and the second encoder includes a global feature encoder and a local feature encoder.

8. The neural network model training method as described in claim 3 or 6, wherein, The optimization of the network parameters of the first neural network based on the first objective function and the sample feature matrix set further includes: Calculate the second similarity between any two feature matrices in the sample feature matrix set; Based on the first objective function and the calculated second similarity, the network parameters of the first neural network are optimized.

9. The neural network model training method as described in claim 1, further comprising: Based on the task to be completed, a second neural network is constructed using the trained first neural network; and, The network parameters of the second neural network are optimized based on the second objective function determined according to the task to be completed.

10. The neural network model training method as described in claim 9, wherein, The first category of data is text data, the second category of data is image data, and the task to be completed is a data classification task, which includes either image classification or text classification. The neural network model training method further includes: The second neural network is iteratively trained using classification training data to optimize its network parameters, wherein the classification training data includes data belonging to the first category and / or data belonging to the second category.

11. The neural network model training method as described in claim 9, wherein, The first category of data is text data, the second category of data is image data, the task to be completed is image segmentation, and the neural network model training method further includes: The second neural network is iteratively trained using image segmentation training data to optimize its network parameters, wherein the image segmentation training data includes data belonging to the second category.

12. The neural network model training method as described in claim 9, wherein, The first category of data is text data, the second category of data is image data, and the task to be completed is a cross-modal retrieval task, which includes: a text-based image retrieval task or an image-based text retrieval task. The neural network model training method further includes: The second neural network is iteratively trained using cross-modal retrieval training data to optimize its network parameters. The cross-modal retrieval training data includes data belonging to the first category and data belonging to the second category.

13. A data classification method based on neural networks, comprising: Obtain the data to be classified, wherein the data to be classified is image data and / or text data; The data to be classified is processed using a data classification neural network to obtain classified data; The data classification neural network is determined based on the first neural network and classification training data, and the first neural network is determined using the neural network model training method as described in claim 1. The classification training data includes data belonging to a first category and / or data belonging to a second category, wherein the data in the first category is text data and the data in the second category is image data.

14. An image segmentation method based on a neural network, comprising: Obtain the image to be segmented; The image to be segmented is processed using an image segmentation neural network to obtain the image segmentation result of the image to be segmented; The image segmentation neural network is determined based on the first neural network and image segmentation training data, and the first neural network is determined using the neural network model training method as described in claim 1. The image segmentation training data includes data belonging to the second category, which is image data.

15. A cross-modal retrieval method based on neural networks, comprising: Get the first data; The first data is processed using a cross-modal retrieval neural network to obtain the second data corresponding to the first data; The cross-modal retrieval neural network is determined based on the first neural network and the cross-modal retrieval training data, and the first neural network is determined using the neural network model training method as described in claim 1. The cross-modal retrieval training data includes data belonging to the first category and data belonging to the second category. The data in the first category is text data, and the data in the second category is image data. The first data is image data, and the second data is text data, or the first data is text data and the second data is image data.

16. A neural network model training device, comprising: The dataset acquisition module is configured to acquire a first dataset consisting of data of a first category and a second dataset consisting of data of a second category, wherein the data of the first category and the data of the second category are respectively one of image data, text data, audio data, and video data, and the data of the first category and the data of the second category are different, and the data in the first dataset and the data in the second dataset are associated accordingly; The anchor data acquisition module is configured to: acquire anchor data from the data of the first category; The similarity calculation module is configured to: calculate the first similarity between each data in the first dataset and the anchor data using a first similarity calculation neural network; The sample determination module is configured to: for a first set of data in the first dataset whose first similarity is greater than a first threshold, and a second set of data in the second dataset that corresponds to the data in the first set, respectively obtain data in the first set and data in the second set to determine multiple positive sample pairs, wherein the multiple positive sample pairs are obtained by combining each data in at least a portion of the data in the first set with multiple data in the second set respectively; The network training module is configured to iteratively train a first neural network using the positive sample pairs to optimize the network parameters of the first neural network, wherein the first neural network is used to obtain the relationship between the data of the first category and the data of the second category.

17. A computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-15.

Images