Training method of data retrieval model, data retrieval method and related equipment

Abstract

The present disclosure relates to a training method of a data retrieval model, a data retrieval method and related equipment, comprising: obtaining a data sample; inputting the data sample into a data retrieval model to obtain a feature vector of the data sample and a plurality of weight vectors; determining a type prediction probability of the data sample based on the feature vector of the data sample, the plurality of weight vectors and a label indicating a true data type of the data sample; calculating a loss according to the type prediction probability of the data sample; and training the data retrieval model by adjusting parameters of the data retrieval model according to the loss. In this way, by setting multiple sub-centers for the same data type, i.e. by setting multiple weight vectors, the feature vector of a certain special visual form of a certain data can be matched with the feature center having the smallest included angle, so as to ensure that the trained data retrieval model can cope with various complex changes in vision of data and improve the retrieval accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to training methods for data retrieval models, data retrieval methods, and related equipment. Background Technology

[0002] With the development of technology, various types of multimedia resource recommendation platforms have emerged. Each type of platform has its own logo, meaning that the logos of different platforms are distinct. The logo indicates the platform affiliation of the multimedia resources and plays an important role in combating piracy.

[0003] However, the logo of the same multimedia resource recommendation platform may have multiple different visual forms. For example, adjusting the color, font, background, etc., of the same logo can result in multiple visually similar logo forms, even though these similar-looking logo forms actually belong to the same type of logo. In related technologies, logo retrieval systems may encounter problems with poor retrieval accuracy due to the complexity and variety of logo styles. Summary of the Invention

[0004] This disclosure provides a training method for a data retrieval model, a data retrieval method, and related equipment to at least solve the problem of poor retrieval accuracy caused by the complex and varied styles of station logos in the aforementioned related technologies.

[0005] According to a first aspect of the present disclosure, a method for training a data retrieval model is provided, comprising: acquiring a data sample, wherein the data sample has a label indicating the true data type of the data sample; inputting the data sample into the data retrieval model to obtain a feature vector and a plurality of weight vectors of the data sample, wherein the feature vector is used to characterize the visual features of the data sample, the plurality of weight vectors include weight vectors corresponding to each of the predetermined plurality of data types, each weight vector corresponding to each data type being used to characterize the probability that the data sample belongs to each visual form under each data type, and each visual form under each data type being each of the predetermined plurality of visual forms under each data type; determining a type prediction probability of the data sample based on the feature vector of the data sample, the plurality of weight vectors and the label indicating the true data type of the data sample, wherein the type prediction probability is a prediction probability that the data type of the data sample is predicted to be the true data type of the data sample; calculating a loss based on the type prediction probability of the data sample; and training the data retrieval model by adjusting the parameters of the data retrieval model according to the loss.

[0006] Optionally, determining the type prediction probability of the data sample based on the feature vector of the data sample, the plurality of weight vectors, and the label indicating the true data type of the data sample includes: calculating a first inner product between the weight vector of each of the predetermined plurality of visual forms under the true data type of the data sample and the feature vector of the data sample; calculating a second inner product between the feature vector of the data sample and each of the plurality of weight vectors; and determining the type prediction probability of the data sample based on the first inner product and the second inner product.

[0007] Optionally, the step of obtaining data samples includes: obtaining original data samples; randomly selecting at least one sample enhancement strategy from a predetermined plurality of sample enhancement strategies; using the selected at least one sample enhancement strategy to enhance the original data samples to obtain enhanced data samples with a plurality of different visual forms; and using the original data samples and the enhanced data samples as the data samples.

[0008] Optionally, the predetermined multiple sample enhancement strategies include the following: randomly erasing parts of the original data sample, rotating the original data sample at a random angle, translating the original data sample in a specified direction, randomly adjusting the arrangement order of the color channels of the pixels contained in the original data sample, adjusting the brightness of the original data sample, and adjusting the contrast of the original data sample.

[0009] According to a second aspect of the present disclosure, a data retrieval method is provided, comprising: acquiring target data to be detected; inputting the target data into a data retrieval model trained according to the training method of the present disclosure to obtain a feature vector and a plurality of weight vectors of the target data, wherein the feature vector is used to characterize the visual features of the target data, the plurality of weight vectors include weight vectors corresponding to each of a predetermined plurality of data types, each weight vector corresponding to each data type is used to characterize the probability that the target data belongs to each visual form under each data type, and each visual form under each data type is each visual form among a predetermined plurality of visual forms under each data type; and determining the data type of the target data based on the feature vector of the target data and the plurality of weight vectors.

[0010] Optionally, determining the data type of the target data based on the feature vector of the target data and the plurality of weight vectors includes: calculating the inner product of the feature vector of the target data with each of the plurality of weight vectors; and determining the data type corresponding to the weight vector with the largest inner product as the data type of the target data.

[0011] According to a third aspect of the present disclosure, a training apparatus for a data retrieval model is provided, comprising: a data sample acquisition module configured to acquire data samples, wherein the data samples have labels indicating the true data type of the data samples; and a data sample input module configured to input the data samples into the data retrieval model to obtain feature vectors and multiple weight vectors of the data samples, wherein the feature vectors are used to characterize the visual features of the data samples, and the multiple weight vectors include weight vectors corresponding to each of a predetermined plurality of data types, each weight vector corresponding to each data type being used to characterize the probability that the data sample belongs to each visual morphology under each data type. Each visual form under a data type is each of a predetermined plurality of visual forms under each data type; the type prediction probability determination module is configured to determine the type prediction probability of the data sample based on the feature vector of the data sample, the plurality of weight vectors and a label indicating the true data type of the data sample, wherein the type prediction probability is the prediction probability of predicting the data type of the data sample as the true data type of the data sample; the loss calculation module is configured to calculate the loss based on the type prediction probability of the data sample; the training module is configured to train the data retrieval model by adjusting the parameters of the data retrieval model according to the loss.

[0012] Optionally, the type prediction probability determination module is configured to: calculate a first inner product between the weight vector of each of the predetermined multiple visual forms under the true data type of the data sample and the feature vector of the data sample; calculate a second inner product between the feature vector of the data sample and each of the multiple weight vectors; and determine the type prediction probability of the data sample based on the first inner product and the second inner product.

[0013] Optionally, the data sample acquisition module is configured to: acquire original data samples; randomly select at least one sample enhancement strategy from a predetermined plurality of sample enhancement strategies; perform sample enhancement on the original data samples using the selected at least one sample enhancement strategy to obtain enhanced data samples with a plurality of different visual forms; and use the original data samples and the enhanced data samples as the data samples.

[0014] Optionally, the predetermined multiple sample enhancement strategies include the following: randomly erasing parts of the original data sample, rotating the original data sample at a random angle, translating the original data sample in a specified direction, randomly adjusting the arrangement order of the color channels of the pixels contained in the original data sample, adjusting the brightness of the original data sample, and adjusting the contrast of the original data sample.

[0015] According to a fourth aspect of the present disclosure, a data retrieval apparatus is provided, comprising: a target data acquisition module configured to acquire target data to be detected; a target data input module configured to input the target data into a data retrieval model trained according to the training method of the present disclosure to obtain a feature vector and a plurality of weight vectors of the target data, wherein the feature vector is used to characterize the visual features of the target data, the plurality of weight vectors include weight vectors corresponding to each of a predetermined plurality of data types, each weight vector corresponding to each data type is used to characterize the probability that the target data belongs to each visual form under each data type, and each visual form under each data type is each visual form among a predetermined plurality of visual forms under each data type; and a data type determination module configured to determine the data type of the target data based on the feature vector and the plurality of weight vectors of the target data.

[0016] Optionally, the data type determination module is configured to: calculate the inner product of the feature vector of the target data with each of the plurality of weight vectors; and determine the data type corresponding to the weight vector with the largest inner product as the data type of the target data.

[0017] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a training method or a data retrieval method for a data retrieval model according to the present disclosure.

[0018] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a training method or a data retrieval method according to a data retrieval model of the present disclosure.

[0019] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a training method or a data retrieval method for a data retrieval model according to the present disclosure.

[0020] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0021] By setting multiple sub-centers for the same data type, i.e., setting multiple weight vectors, the feature vector of a certain visual form of a certain data can be matched with the feature center with the smallest angle. In other words, the feature vector of a certain visual form of a certain data can be matched with the weight vector with the smallest angle. This ensures that the trained data retrieval model can cope with various complex visual deformations of data, thereby improving retrieval accuracy and precision.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0024] Figure 1 This is a flowchart illustrating a training method for a data retrieval model according to an exemplary embodiment of the present disclosure;

[0025] Figure 2 This is a schematic diagram illustrating data augmentation of a raw data sample according to an exemplary embodiment of the present disclosure;

[0026] Figure 3 This is a schematic diagram illustrating a comparison between an existing Softmax classification according to an exemplary embodiment of the present disclosure and a multicenter classification in the present disclosure;

[0027] Figure 4 This is a flowchart illustrating a data retrieval method according to an exemplary embodiment of the present disclosure;

[0028] Figure 5 This is a block diagram illustrating a training apparatus for a data retrieval model according to an exemplary embodiment of the present disclosure;

[0029] Figure 6 This is a block diagram illustrating a data retrieval apparatus according to an exemplary embodiment of the present disclosure;

[0030] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0032] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0033] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0034] Figure 1 This is a flowchart illustrating a training method for a data retrieval model according to an exemplary embodiment of the present disclosure.

[0035] Reference Figure 1 In step 101, data samples can be obtained. These data samples may have labels indicating the actual data type of the data sample. For example, in this disclosure, a "data sample" can be a "logo sample," where "logo" can be the logo of various multimedia resource recommendation platforms, and each logo sample can be a single logo image.

[0036] According to exemplary embodiments of this disclosure, raw data samples can be obtained, and at least one sample enhancement strategy can be randomly selected from a plurality of predetermined sample enhancement strategies. Then, the raw data samples can be enhanced using the selected at least one sample enhancement strategy to obtain enhanced data samples with various visual morphologies. Next, the raw data samples and enhanced data samples can be used as the aforementioned data samples.

[0037] In this way, various combinations of sample augmentation strategies can be randomly selected to augment the original data samples according to actual needs, so that the training data can cover various morphological changes of the data. This ensures that the trained data retrieval model can cope with various complex visual deformations of the data, thereby improving the retrieval accuracy and precision of the data retrieval model.

[0038] According to exemplary embodiments of this disclosure, the predetermined multiple sample enhancement strategies may include the following:

[0039] Randomly erase portions of the original data sample; for example, randomly delete portions of the original logo image.

[0040] The original data sample can be rotated at a random angle. For example, the original logo image can be rotated clockwise or counterclockwise by a random angle.

[0041] The original data sample is translated in a specified direction. For example, the original logo image can be translated a certain distance in the X or Y direction.

[0042] Randomly adjust the color channel order of the pixels in the original data sample. For example, assuming the original logo image contains pixels with RGB color channels, the color channel order can be adjusted to BRG or GRB, etc.

[0043] Adjust the brightness of the original data samples; for example, you can adjust the brightness of the original logo image.

[0044] Adjust the contrast of the original data sample; for example, you can adjust the contrast of the original logo image.

[0045] It should be noted that when performing data augmentation on the original data sample, one or a combination of the above strategies can be selected for sample augmentation. Furthermore, the strategies for sample augmentation are not limited to the above-mentioned ones; other strategies may also exist. The aforementioned sample augmentation strategies are merely illustrative examples.

[0046] Figure 2 This is a schematic diagram illustrating data augmentation of a raw data sample according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 2 Suppose that a certain original data sample is a heart-shaped image, and there is a letter A in the heart-shaped image. The original font of the letter A is SimSun, the original position of the heart-shaped image is vertical, and the original background color of the letter A contained in the heart-shaped image is white.

[0047] When performing sample augmentation on the original data sample, you can perform a clockwise rotation operation (a) on the original data sample, which will result in a heart-shaped image with a tilted position; you can adjust the font of the letter A contained in the original data sample (b), for example, you can change the font of the letter A contained in the original data sample from SimSun to KaiTi; you can adjust the background color of the letter A contained in the heart-shaped image of the original data sample (c), for example, you can change the background color of the letter A contained in the heart-shaped image of the original data sample from white to gray.

[0048] In this way, by augmenting the original data samples, the number of training samples is increased and the types of training samples are made more diverse. Even if the training data covers various morphological changes of the data, the robustness and retrieval generalization ability of the data retrieval model can be improved, ensuring that the trained data retrieval model can cope with various complex visual deformations of the data, thereby improving retrieval accuracy and precision.

[0049] It should be noted that, in addition to sample augmentation, i.e. data augmentation, this disclosure can also propose a "multi-center classification" method to deal with the changes in various visual forms of data. The core idea of ​​"multi-center classification" is that there may be multiple feature centers for each type of data instead of one, that is, there may be multiple weight vectors for each type of data instead of one, and it is expected that multiple feature centers correspond to multiple visual forms of the same type of data.

[0050] In step 102, data samples can be input into a data retrieval model to obtain feature vectors and multiple weight vectors for the data samples. The "feature vector" characterizes the visual features of the data sample, and the "multiple weight vectors" can include weight vectors corresponding to multiple predetermined data types. Each weight vector corresponding to each data type can characterize the probability that the data sample belongs to each visual form within that data type. Each visual form within each data type represents each of the predetermined multiple visual forms within that data type. In other words, the multiple weight vectors can include weight vectors characterizing the probability that a data sample belongs to each of the predetermined multiple visual forms within each of the predetermined multiple data types. Similarly, the multiple weight vectors can include weight vectors characterizing the probability that a logo sample belongs to each of the predetermined multiple visual forms within each of the predetermined multiple logo types. The "feature vector" X of the data sample can also be called a "visual feature." The "visual feature" acts similarly to a fingerprint, uniquely identifying which data sample it is.

[0051] Figure 3 This is a schematic diagram illustrating a comparison between a conventional Softmax classification according to an exemplary embodiment of the present disclosure and a multicenter classification in the present disclosure. (Refer to...) Figure 3 The circular image is used to show the decision boundary of the classifier, and different colors represent the coverage of different logo types.

[0052] The circular image on the left represents a standard Softmax classification, showing three logo types: Logo Type 1, Logo Type 2, and Logo Type 3. The three arrows represent the feature centers (weight vectors) for each of the three logo types. The three adjacent colored boundaries represent the decision boundaries between two logo types: the logo type of the target logo is determined by the feature center closest to its feature vector (i.e., the weight vector with which the angle between the target logo's feature vector and the weight vector is smallest).

[0053] The circular image on the right shows the feature centers and decision boundaries with multi-center optimization. While the decision boundary doesn't change significantly, multiple feature centers may exist within the corresponding color region for each logo type. The advantage of multiple feature centers is that the feature vector of a specific visual form of a logo can be matched with the feature center that has the smallest angle with it; that is, the feature vector of a specific visual form of a logo can be matched with the weight vector that has the smallest angle with it. Multi-center optimization helps improve the learning accuracy of the logo retrieval model, unlike existing Softmax classification which treats samples with large angles to feature centers as noise samples. For example, Figure 3 In the circular image on the left, the classification interval of logo 3 is very large. If a sample has a large angle between its distance from the feature center of logo 3, it is easy to treat it as a noise sample. However, the multi-center algorithm in this disclosure will not do so.

[0054] In step 103, the type prediction probability of a data sample can be determined based on the feature vector of the data sample, multiple weight vectors, and a label indicating the true data type of the data sample. The type prediction probability can be the probability of predicting the data sample's data type as its true data type.

[0055] According to exemplary embodiments of this disclosure, a first inner product can be calculated between the weight vector of each of a predetermined plurality of visual forms under the real data type of the data sample and the feature vector of the data sample. A second inner product can also be calculated between the feature vector of the data sample and each of the aforementioned plurality of weight vectors. Next, the type prediction probability of the data sample can be determined based on the first and second inner products. In this way, when training the data retrieval model, multiple sub-centers can be set for the same data type, i.e., multiple weight vectors can be set, and each weight vector corresponds to a visual form under a data type. This means that various deformations of data in real-world situations have been considered during the training of the data retrieval model, ensuring that the trained data retrieval model can cope with various complex visual deformations of data, thus improving retrieval accuracy and precision.

[0056] For example, the probability P of predicting the type of a data sample can be expressed by the following formula:

[0057]

[0058] Where X can be the feature vector of the data sample extracted by the data retrieval model, and P(y=j / X) can be the type prediction probability of the data retrieval model predicting the data type of the data sample corresponding to X as the true data type y=j of the data sample.

[0059] W jk W can be the weight vector of the k-th visual form among the j-th data type and the various visual forms under the aforementioned multiple data types. jk W can be the k-th feature center among the predetermined feature centers under the j-th data type mentioned above. mk W can be the weight vector of the k-th visual form among the predefined multiple visual forms under the m-th data type mentioned above. mk It can be the k-th feature center among the predetermined feature centers under the m-th data type mentioned above;

[0060] max k X T W jk It can represent the maximum value of the inner product of the feature vector X of the data sample and the weight vector of each visual form in a predetermined set of visual forms under the j-th data type; max k X T W mk It can represent the maximum value of the inner product of the feature vector X of the data sample and the weight vector of each visual form in the predetermined multiple visual forms under the m-th data type; m can be the number of the aforementioned predetermined multiple data types.

[0061] In step 104, the probability can be predicted based on the type of data sample, and the loss can be calculated.

[0062] According to exemplary embodiments of this disclosure, the above-mentioned loss can be expressed by the following formula:

[0063]

[0064] Where J can be the aforementioned loss, X, P(y = j / X, W) jk W mk ,max k X T W jk ,max k X T W mk The meanings of 'm' and 'm' are explained in detail above and will not be repeated here.

[0065] In this way, by setting multiple sub-centers for the same data type, i.e., setting multiple weight vectors, the feature vector of a certain visual form of a certain data can be matched with the feature center with the smallest angle. In other words, the feature vector of a certain visual form of a certain data can be matched with the weight vector with the smallest angle. This ensures that the trained data retrieval model can cope with various complex visual deformations of data, thereby improving retrieval accuracy and precision.

[0066] In step 105, the data retrieval model can be trained by adjusting the parameters of the data retrieval model based on the loss.

[0067] It should be noted that the data retrieval model in this disclosure can be applied to the classification or retrieval of data. For example, it can be applied to the classification or retrieval of station logos. In addition to station logos, the data can also be any image-based identifier, and this disclosure does not impose any specific limitations on it.

[0068] Figure 4 This is a flowchart illustrating a data retrieval method according to an exemplary embodiment of the present disclosure.

[0069] Reference Figure 4 In step 401, the target data to be detected can be obtained. For example, the "data" in this disclosure can be a "platform logo," which can be the logo of various multimedia resource recommendation platforms, and each logo can be a logo image.

[0070] In step 402, the target data can be input into the data retrieval model trained according to the training method of this disclosure to obtain the feature vector and multiple weight vectors of the target data.

[0071] Among them, the "feature vector" is used to characterize the visual features of the target data, and the "multiple weight vectors" can include weight vectors corresponding to each of the predetermined multiple data types. Each weight vector corresponding to each data type can be used to characterize the probability that the target data belongs to each visual form under each data type. Each visual form under each data type can be each visual form among the predetermined multiple visual forms under each data type.

[0072] In other words, multiple weight vectors can include weight vectors representing the probability that the target data belongs to each of the predetermined visual forms within each of the predetermined multiple data types. Similarly, multiple weight vectors can include weight vectors representing the probability that the logo image belongs to each of the predetermined multiple visual forms within each of the predetermined multiple logo types. Furthermore, the weight vector for each logo type is the feature center for that logo type. The feature vectors of the target data can also be called the "visual features" of the target data. The "visual features" function similarly to fingerprints, uniquely identifying which data type the target data belongs to.

[0073] In step 403, the data type of the target data can be determined based on the feature vector of the target data and the aforementioned multiple weight vectors. That is, the data type of the target data can be determined based on the feature vector of the target data and the predetermined multiple feature centers under each of the aforementioned predetermined multiple data types.

[0074] In this way, by setting multiple sub-centers for the same data type, i.e., setting multiple weight vectors, the feature vector of a certain visual form of a certain data can be matched with the feature center with the smallest angle. In other words, the feature vector of a certain visual form of a certain data can be matched with the weight vector with the smallest angle. This ensures that the trained data retrieval model can cope with various complex visual deformations of data, thereby improving retrieval accuracy and precision.

[0075] According to an exemplary embodiment of this disclosure, the inner product of the feature vector of the target data with each of the plurality of weight vectors can be calculated, that is, the inner product of the feature vector of the target data with each of the plurality of predetermined feature centers under each of the plurality of predetermined data types can be calculated. Then, the data type corresponding to the weight vector with the largest inner product can be determined as the data type of the target data, that is, the data type corresponding to the feature center with the largest inner product can be determined as the data type of the target data.

[0076] Thus, the larger the inner product, the smaller the angle between the feature vector of the target data and the corresponding weight vector, meaning the feature vector of the target data is closer to the corresponding weight vector, and the more similar the visual form of the target data is to the visual form corresponding to the weight vector. Therefore, the data type corresponding to the weight vector with the largest inner product can be determined as the data type of the target data, which can ensure the accuracy of determining the data type of the target data.

[0077] For example, suppose there are a total of 10 predetermined data types, i.e., m = 10, and each data type has a total of 3 predetermined visual forms, i.e., k = 3. In this case, there are a total of m × k = 10 × 3 = 30 weight vectors. The inner product of the feature vector of the target data with each of the 30 weight vectors can be calculated, resulting in 30 inner product results. Next, the data type corresponding to the weight vector with the largest inner product among the 30 inner product results can be determined as the data type of the target data.

[0078] It should be noted that the matrix size of the weight parameter W in the "multi-center classification" of this disclosure is (m*k, dim), where m is the number of the aforementioned predetermined multiple data types, k represents that each data type contains k sub-centers, that is, k represents that each data type contains k weight vectors, and dim represents the length of the feature center. The "multi-center classification" in this disclosure finds the data type of the target data by calculating the maximum value of the inner product between the feature vector of the target data and the predetermined multiple feature centers under each of the predetermined multiple data types, thus completing data matching.

[0079] Figure 5 This is a block diagram illustrating a training apparatus for a data retrieval model according to an exemplary embodiment of the present disclosure.

[0080] Reference Figure 5 The device 500 may include a data sample acquisition module 501, a data sample input module 502, a type prediction probability determination module 503, a loss calculation module 504, and a training module 505.

[0081] The data sample acquisition module 501 can acquire data samples. These data samples may have labels indicating the actual data type of the data sample. For example, in this disclosure, a "data sample" can be a "logo sample," where "logo" can be the logo of various multimedia resource recommendation platforms, and each logo sample can be a single logo image.

[0082] According to an exemplary embodiment of this disclosure, the data sample acquisition module 501 can acquire original data samples and can also randomly select at least one sample enhancement strategy from a predetermined plurality of sample enhancement strategies. Then, the data sample acquisition module 501 can use the selected at least one sample enhancement strategy to enhance the original data samples, obtaining enhanced data samples with various visual morphologies. Next, the data sample acquisition module 501 can use the original data samples and the enhanced data samples as the aforementioned data samples.

[0083] In this way, various combinations of sample augmentation strategies can be randomly selected to augment the original data samples according to actual needs, so that the training data can cover various morphological changes of the data. This ensures that the trained data retrieval model can cope with various complex visual deformations of the data, thereby improving the retrieval accuracy and precision of the data retrieval model.

[0084] According to exemplary embodiments of this disclosure, the predetermined multiple sample enhancement strategies may include the following:

[0085] Randomly erase portions of the original data sample; for example, randomly delete portions of the original logo image.

[0086] The original data sample can be rotated at a random angle. For example, the original logo image can be rotated clockwise or counterclockwise by a random angle.

[0087] The original data sample is translated in a specified direction. For example, the original logo image can be translated a certain distance in the X or Y direction.

[0088] Randomly adjust the color channel order of the pixels in the original data sample. For example, assuming the original logo image contains pixels with RGB color channels, the color channel order can be adjusted to BRG or GRB, etc.

[0089] Adjust the brightness of the original data samples; for example, you can adjust the brightness of the original logo image.

[0090] Adjust the contrast of the original data sample; for example, you can adjust the contrast of the original logo image.

[0091] It should be noted that when performing data augmentation on the original data sample, one or a combination of the above strategies can be selected for sample augmentation. Furthermore, the strategies for sample augmentation are not limited to the above-mentioned ones; other strategies may also exist. The aforementioned sample augmentation strategies are merely illustrative examples.

[0092] In this way, by augmenting the original data samples, the number of training samples is increased and the types of training samples are made more diverse. Even if the training data covers various morphological changes of the data, the robustness and retrieval generalization ability of the data retrieval model can be improved, ensuring that the trained data retrieval model can cope with various complex visual deformations of the data, thereby improving retrieval accuracy and precision.

[0093] It should be noted that, in addition to sample augmentation, i.e. data augmentation, this disclosure can also propose a "multi-center classification" method to deal with the changes in various visual forms of data. The core idea of ​​"multi-center classification" is that there may be multiple feature centers for each type of data instead of one, that is, there may be multiple weight vectors for each type of data instead of one, and it is expected that multiple feature centers correspond to multiple visual forms of the same type of data.

[0094] The data sample input module 502 can input data samples into the data retrieval model to obtain feature vectors and multiple weight vectors of the data samples. Among them, the "feature vectors" are used to characterize the visual features of the data samples, and the "multiple weight vectors" can include weight vectors corresponding to each of the predetermined multiple data types. Each weight vector corresponding to each data type can be used to characterize the probability that the data sample belongs to each visual form under each data type. Each visual form under each data type is each visual form among the predetermined multiple visual forms under each data type.

[0095] In other words, multiple weight vectors can include weight vectors representing the probability that a data sample belongs to each of the predetermined visual forms within each of the predetermined multiple data types. Similarly, multiple weight vectors can include weight vectors representing the probability that a logo sample belongs to each of the predetermined multiple logo types within each of the predetermined multiple visual forms. The "feature vector" X of a data sample can also be called a "visual feature." The "visual feature" acts similarly to a fingerprint, uniquely identifying which data sample it is.

[0096] The type prediction probability determination module 503 can determine the type prediction probability of a data sample based on the data sample's feature vector, multiple weight vectors, and a label indicating the true data type of the data sample. The type prediction probability can be the probability of predicting the data sample's data type as its true data type.

[0097] According to an exemplary embodiment of this disclosure, the type prediction probability determination module 503 can calculate the first inner product of the weight vector of each of the predetermined multiple visual forms under the true data type of the data sample and the feature vector of the data sample, and can also calculate the second inner product of the feature vector of the data sample and each of the aforementioned multiple weight vectors. Next, the type prediction probability of the data sample can be determined based on the first and second inner products. In this way, when training the data retrieval model, multiple sub-centers can be set for the same data type, i.e., multiple weight vectors can be set, and each weight vector corresponds to a visual form under a data type. This means that various deformations of data in real-world situations have been considered during the training of the data retrieval model, ensuring that the trained data retrieval model can cope with various complex visual deformations of data, thus improving retrieval accuracy and precision.

[0098] The loss calculation module 504 can predict probabilities based on the type of data samples and calculate the loss. In this way, by setting multiple sub-centers (i.e., multiple weight vectors) for the same data type, the feature vector of a specific visual form of a data point can be matched with the feature center that has the smallest angle with it. This ensures that the trained data retrieval model can handle various complex visual deformations of data, improving retrieval accuracy and precision.

[0099] The training module 505 can train the data retrieval model by adjusting the parameters of the data retrieval model based on the loss.

[0100] Figure 6 This is a block diagram illustrating a data retrieval apparatus according to an exemplary embodiment of the present disclosure.

[0101] Reference Figure 6 The device 600 may include a target data acquisition module 601, a target data input module 602, and a data type determination module 603.

[0102] The target data acquisition module 601 can acquire the target data to be detected. For example, the "data" in this disclosure can be a "platform logo", and the "platform logo" can be the logo of various multimedia resource recommendation platforms, and each logo can be a logo image.

[0103] The target data input module 602 can input the target data into the data retrieval model trained according to the training method of this disclosure, and obtain the feature vector and multiple weight vectors of the target data.

[0104] Among them, the "feature vector" is used to characterize the visual features of the target data, and the "multiple weight vectors" can include weight vectors corresponding to each of the predetermined multiple data types. Each weight vector corresponding to each data type can be used to characterize the probability that the target data belongs to each visual form under each data type. Each visual form under each data type can be each visual form among the predetermined multiple visual forms under each data type.

[0105] In other words, multiple weight vectors can include weight vectors representing the probability that the target data belongs to each of the predetermined visual forms within each of the predetermined multiple data types. Similarly, multiple weight vectors can include weight vectors representing the probability that the logo image belongs to each of the predetermined multiple visual forms within each of the predetermined multiple logo types. Furthermore, the weight vector for each logo type is the feature center for that logo type. The feature vectors of the target data can also be called the "visual features" of the target data. The "visual features" function similarly to fingerprints, uniquely identifying which data type the target data belongs to.

[0106] The data type determination module 603 can determine the data type of the target data based on the feature vector of the target data and the aforementioned multiple weight vectors. That is, it can determine the data type of the target data based on the feature vector of the target data and the predetermined multiple feature centers under each of the aforementioned multiple predetermined data types.

[0107] In this way, by setting multiple sub-centers for the same data type, i.e., setting multiple weight vectors, the feature vector of a certain visual form of a certain data can be matched with the feature center with the smallest angle. In other words, the feature vector of a certain visual form of a certain data can be matched with the weight vector with the smallest angle. This ensures that the trained data retrieval model can cope with various complex visual deformations of data, thereby improving retrieval accuracy and precision.

[0108] According to an exemplary embodiment of this disclosure, the data type determination module 603 can calculate the inner product of the feature vector of the target data with each of the aforementioned plurality of weight vectors, that is, it can calculate the inner product of the feature vector of the target data with each of the aforementioned plurality of predetermined feature centers under each of the predetermined plurality of data types. Then, the data type corresponding to the weight vector with the largest inner product can be determined as the data type of the target data, that is, the data type corresponding to the feature center with the largest inner product can be determined as the data type of the target data. In this way, since the larger the inner product, the smaller the angle between the feature vector of the target data and the corresponding weight vector, that is, the closer the feature vector of the target data is to the corresponding weight vector, and the more similar the visual form of the target data is to the visual form corresponding to the corresponding weight vector, the data type corresponding to the weight vector with the largest inner product can be determined as the data type of the target data, which can ensure the accuracy of determining the data type of the target data.

[0109] It should be noted that the matrix size of the weight parameter W in the "multi-center classification" of this disclosure is (m*k, dim), where m is the number of the aforementioned predetermined multiple data types, k represents that each data type contains k sub-centers, that is, k represents that each data type contains k weight vectors, and dim represents the length of the feature center. The "multi-center classification" in this disclosure finds the data type of the target data by calculating the maximum value of the inner product between the feature vector of the target data and the predetermined multiple feature centers under each of the predetermined multiple data types, thus completing data matching.

[0110] Figure 7 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment of the present disclosure.

[0111] Reference Figure 7 The electronic device 700 includes at least one memory 701 and at least one processor 702. The at least one memory 701 stores instructions that, when executed by the at least one processor 702, execute a training method or a data retrieval method for a data retrieval model according to an exemplary embodiment of the present disclosure.

[0112] As an example, electronic device 700 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned instructions. Here, electronic device 700 is not necessarily a single electronic device, but may be a collection of any devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 700 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0113] In electronic device 700, processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0114] The processor 702 can execute instructions or code stored in the memory 701, which can also store data. Instructions and data can also be sent and received via a network through a network interface device, which can employ any known transmission protocol.

[0115] The memory 701 can be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 701 can include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 701 and the processor 702 can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 702 to read files stored in the memory.

[0116] In addition, the electronic device 700 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 700 can be interconnected via a bus and / or network.

[0117] According to exemplary embodiments of this disclosure, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the training method or data retrieval method of the data retrieval model described above. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0118] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, including a computer program that, when executed by a processor, implements a training method or a data retrieval method for a data retrieval model according to the present disclosure.

[0119] According to the training method, data retrieval method and related equipment of the data retrieval model disclosed herein, by setting multiple sub-centers for the same data type, that is, setting multiple weight vectors, the feature vector of a certain visual form of a certain data can be matched with the feature center with the smallest angle, that is, the feature vector of a certain visual form of a certain data can be matched with the weight vector with the smallest angle, ensuring that the trained data retrieval model can cope with various complex visual deformations of data, thereby improving retrieval accuracy and precision.

[0120] Furthermore, various combinations of sample augmentation strategies can be randomly selected to augment the original data samples according to actual needs, so that the training data can cover various morphological changes of the data. This ensures that the trained data retrieval model can cope with various complex visual deformations of the data, thereby improving the retrieval accuracy and precision of the data retrieval model.

[0121] Furthermore, by performing sample augmentation on the original data samples, the number of training samples is increased and the types of training samples are made more diverse. Even if the training data covers various morphological changes of the data, it can improve the robustness and retrieval generalization ability of the data retrieval model, ensuring that the trained data retrieval model can cope with various complex visual deformations of the data, thereby improving retrieval accuracy and precision.

[0122] Furthermore, when training the data retrieval model, multiple sub-centers can be set for the same data type, that is, multiple weight vectors can be set, and each weight vector corresponds to a visual form under a data type. In other words, the various deformations of data in real-world situations have been taken into account during the training of the data retrieval model, which can ensure that the trained data retrieval model can cope with various complex visual deformations of data, thereby improving retrieval accuracy and precision.

[0123] Furthermore, since the larger the inner product, the smaller the angle between the feature vector of the target data and the corresponding weight vector, that is, the closer the feature vector of the target data is to the corresponding weight vector, the more similar the visual form of the target data is to the visual form corresponding to the weight vector. Therefore, the data type corresponding to the weight vector with the largest inner product can be determined as the data type of the target data, which can ensure the accuracy of determining the data type of the target data.

[0124] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0125] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for a data retrieval model, characterized in that, include: Obtain data samples, wherein the data samples have labels indicating the true data type of the data samples, and the data samples include logo image samples, wherein the logo is the logo of a multimedia resource recommendation platform; The data sample is input into the data retrieval model to obtain the feature vector and multiple weight vectors of the data sample. The feature vector is used to characterize the visual features of the data sample. The multiple weight vectors include weight vectors corresponding to each of the predetermined multiple data types. Each weight vector corresponding to each data type is used to characterize the probability that the data sample belongs to each visual form under each data type. Each visual form under each data type is each visual form among the predetermined multiple visual forms under each data type. Based on the feature vector of the data sample, the plurality of weight vectors, and the label indicating the true data type of the data sample, the type prediction probability of the data sample is determined, wherein the type prediction probability is the prediction probability of predicting the data type of the data sample as the true data type of the data sample. Based on the type of the data sample, predict the probability and calculate the loss; The data retrieval model is trained by adjusting the parameters of the data retrieval model based on the loss. The step of determining the type prediction probability of the data sample based on the feature vector of the data sample, the plurality of weight vectors, and the label indicating the true data type of the data sample includes: Calculate the first inner product between the weight vector of each visual form in a predetermined plurality of visual forms under the real data type of the data sample and the feature vector of the data sample; Calculate the second inner product between the feature vector of the data sample and each of the plurality of weight vectors; Based on the first inner product and the second inner product, the type prediction probability of the data sample is determined.

2. The training method as described in claim 1, characterized in that, The acquisition of data samples includes: Obtain raw data samples; Randomly select at least one sample augmentation strategy from a predetermined number of sample augmentation strategies; The original data samples are augmented using the selected at least one sample augmentation strategy to obtain augmented data samples with various visual morphologies. The original data sample and the enhanced data sample are used as the data sample.

3. The training method as described in claim 2, characterized in that, The predetermined multiple sample augmentation strategies include the following: Randomly erase parts of the original data sample, rotate the original data sample at a random angle, translate the original data sample in a specified direction, randomly adjust the color channel arrangement of the pixels contained in the original data sample, adjust the brightness of the original data sample, and adjust the contrast of the original data sample.

4. A data retrieval method, characterized in that, include: Acquire the target data to be detected; The target data is input into a data retrieval model trained by the training method according to any one of claims 1 to 3 to obtain a feature vector and multiple weight vectors of the target data. The feature vector is used to characterize the visual features of the target data. The multiple weight vectors include weight vectors corresponding to each of the predetermined multiple data types. Each weight vector corresponding to each data type is used to characterize the probability that the target data belongs to each visual form under each data type. Each visual form under each data type is each visual form among the predetermined multiple visual forms under each data type. Based on the feature vector of the target data and the multiple weight vectors, the data type of the target data is determined.

5. The data retrieval method as described in claim 4, characterized in that, Determining the data type of the target data based on the feature vector and the plurality of weight vectors includes: Calculate the inner product between the feature vector of the target data and each of the multiple weight vectors; The data type corresponding to the weight vector with the largest inner product is determined as the data type of the target data.

6. A training device for a data retrieval model, characterized in that, include: The data sample acquisition module is configured to acquire data samples, wherein the data samples have labels indicating the true data type of the data samples, and the data samples include logo image samples, wherein the logo is the logo of a multimedia resource recommendation platform; A data sample input module is configured to input the data sample into the data retrieval model to obtain a feature vector and multiple weight vectors of the data sample. The feature vector is used to characterize the visual features of the data sample. The multiple weight vectors include weight vectors corresponding to each of a predetermined plurality of data types. Each weight vector corresponding to each data type is used to characterize the probability that the data sample belongs to each visual form under each data type. Each visual form under each data type is each visual form among a predetermined plurality of visual forms under each data type. The type prediction probability determination module is configured to determine the type prediction probability of the data sample based on the feature vector of the data sample, the plurality of weight vectors and a label indicating the true data type of the data sample, wherein the type prediction probability is the prediction probability of predicting the data type of the data sample as the true data type of the data sample; The loss calculation module is configured to predict the probability based on the type of the data sample and calculate the loss. The training module is configured to train the data retrieval model by adjusting the parameters of the data retrieval model according to the loss. The type prediction probability determination module is configured as follows: Calculate the first inner product between the weight vector of each visual form in a predetermined plurality of visual forms under the real data type of the data sample and the feature vector of the data sample; Calculate the second inner product between the feature vector of the data sample and each of the plurality of weight vectors; Based on the first inner product and the second inner product, the type prediction probability of the data sample is determined.

7. A data retrieval device, characterized in that, include: The target data acquisition module is configured to acquire the target data to be detected. The target data input module is configured to input the target data into a data retrieval model trained by the training method according to any one of claims 1 to 3, and obtain a feature vector and a plurality of weight vectors of the target data, wherein the feature vector is used to characterize the visual features of the target data, and the plurality of weight vectors include weight vectors corresponding to each of a predetermined plurality of data types, and each weight vector corresponding to each data type is used to characterize the probability that the target data belongs to each visual form under each data type, wherein each visual form under each data type is each visual form among a predetermined plurality of visual forms under each data type; The data type determination module is configured to determine the data type of the target data based on the feature vector of the target data and the multiple weight vectors.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the training method of the data retrieval model as described in any one of claims 1 to 3, or to implement the data retrieval method as described in any one of claims 4 to 5.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the training method of the data retrieval model as described in any one of claims 1 to 3, or perform the data retrieval method as described in any one of claims 4 to 5.

Images