Image text similarity detection method and device, and terminal

Abstract

An image text similarity detection method and device, and a terminal, the method comprising: determining a plurality of images to be detected, and text to be detected corresponding to each image to be detected; determining an image numerical vector of the image to be detected, and a text numerical vector of the text to be detected; using an alignment algorithm to perform alignment processing on the image numerical vector and / or the text numerical vector, so that the number of floating point values contained in the image numerical vector and the text numerical vector is equal, and the image numerical vector and the text numerical vector are one-to-one aligned; iteratively optimizing one or more of the following items: the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm, to minimize a preset loss function to determine the optimized items; and using the optimized items to determine the similarity result between the image numerical vector and the text numerical vector. The present application can effectively improve the accuracy of detection.

Description

Technical Field

[0001] This invention relates to the field of digital media technology, and in particular to an image-text similarity detection method, apparatus, and terminal. Background Technology

[0002] With the rapid development of digital media technology, multimedia resources, including text and images, are growing in scale. Self-image text recognition or self-text image drawing has gradually become a research hotspot. For example, based on the visual presentation of text content, image retrieval of text content can be performed, or based on text recognition of image content, text retrieval of image content can be performed.

[0003] However, current technologies lack the ability to assess the accuracy and effectiveness of conversions between text derived from image recognition and images derived from text rendering. In other words, the accuracy of similarity assessments using existing image-text similarity detection techniques is relatively low.

[0004] There is a need for an image-text similarity detection method that can detect the similarity between images and text and effectively improve the accuracy of the detection. Summary of the Invention

[0005] The technical problem solved by this invention is to provide an image-text similarity detection method, device, and terminal, which can effectively improve the accuracy of detection.

[0006] To address the aforementioned technical problems, this invention provides an image-text similarity detection method, comprising: determining multiple images to be detected and corresponding text to be detected for each image; using an image feature extraction algorithm to extract features from each image to determine an image value vector containing a first preset number of floating-point values; and using a text feature extraction algorithm to extract features from the corresponding text to determine a text value vector containing a second preset number of floating-point values; using an alignment algorithm to align the image value vector and / or the text value vector so that the number of floating-point values ​​in the image value vector and the text value vector are equal, and the image value vector and the text value vector are aligned one-to-one; iterating over one or more of the following optimization terms to minimize a preset loss function to determine optimized optimization terms: the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm; and using the optimized optimization terms to determine the similarity result between the image value vector and the text value vector.

[0007] Optionally, the preset loss function is determined based on the similarity between the image value vectors of each aligned image to be detected and the corresponding text value vectors, as well as the preset target similarity.

[0008] Optionally, the loss function can be determined using the following formula:

[0009]

[0010] or,

[0011] Where L(Y',Y) represents the loss function, Y i Y is used to represent the similarity between the aligned image value vector of the i-th image to be detected and the corresponding text value vector. i The value is used to represent the preset target similarity, and k is used to represent the number of images to be detected, where 1≤i≤k and i and k are both positive integers.

[0012] Optionally, an alignment algorithm is used to align image numerical vectors and / or text numerical vectors, including: constructing an N×M neural network matrix; using the product of the numerical vector to be aligned and the neural network matrix as the numerical vector to be aligned; wherein M represents the number of floating-point values ​​contained in the numerical vector to be aligned, and N represents the number of floating-point values ​​contained in the numerical vector to be aligned.

[0013] Optionally, the numerical vector to be aligned is one of the image numerical vector and the text numerical vector, and the numerical vector being aligned is the other of the image numerical vector and the text numerical vector.

[0014] Optionally, the alignment algorithm is iterated to minimize a preset loss function to determine the optimized alignment algorithm, including: updating the N×M neural network matrix using a gradient descent algorithm; based on the N×M neural network matrix obtained in each update, determining the image value vector and the corresponding text value vector of each image to be detected after alignment, and calculating the loss function until the loss function converges to the target value or reaches the preset number of iterations, to obtain the optimized N×M neural network matrix.

[0015] Optionally, in the first p iterations, the N×M neural network matrix is ​​updated using the first learning rate of the gradient descent algorithm, and in the subsequent q iterations, the N×M neural network matrix is ​​updated using the second learning rate of the gradient descent algorithm; wherein, the first learning rate is greater than the second learning rate; P is used to represent the total number of iterations in the gradient descent algorithm, 1 < p + q ≤ P, and p, q, and P are all positive integers.

[0016] Optionally, the image feature extraction algorithm and / or the text feature extraction algorithm are iterated to minimize a preset loss function to determine the optimized image feature extraction algorithm and / or the text feature extraction algorithm. This includes: updating the parameters in the image feature extraction algorithm and / or the text feature extraction algorithm using a gradient descent algorithm; determining the image value vector and the corresponding text value vector of each image to be detected based on each updated algorithm; then calculating the loss function based on the aligned image value vector and the corresponding text value vector of each image to be detected; and so on until the loss function converges to the target value or reaches a preset number of iterations, thereby obtaining the optimized parameters in the image feature extraction algorithm and / or the text feature extraction algorithm.

[0017] Optionally, the image feature extraction algorithm includes multiple layers of differentiable nonlinear components, and / or, the text feature extraction algorithm includes multiple layers of differentiable nonlinear components; updating the parameters in the image feature extraction algorithm and / or the text feature extraction algorithm includes: updating only the parameters in the terminal first preset number layer of the image feature extraction algorithm, and / or, updating only the parameters in the terminal second preset number layer of the text feature extraction algorithm; wherein, the first preset number and the second preset number are the same or different.

[0018] Optionally, the image feature extraction algorithm includes a fully connected layer, and the fully connected layer is located at the end of the image feature extraction algorithm; and / or, the text feature extraction algorithm includes a fully connected layer, and the fully connected layer is located at the end of the text feature extraction algorithm; updating only the parameters in the first preset number of layers at the end of the image feature extraction algorithm includes updating only the parameters in the fully connected layer of the image feature extraction algorithm; updating only the parameters in the first preset number of layers at the end of the text feature extraction algorithm includes updating only the parameters in the fully connected layer of the text feature extraction algorithm.

[0019] Optionally, one or more of the following conditions must be met: the image feature extraction algorithm is selected from the Deep Residual Network (WRN) algorithm and the Convolutional Neural Network (VGG) algorithm; the text feature extraction algorithm is selected from the Bidirectional Encoding Representation (BERT) algorithm and the Long Short-Term Memory (LSTM) algorithm.

[0020] Optionally, the image numerical vector and the text numerical vector are floating-point values ​​of one-dimensional vectors.

[0021] To address the aforementioned technical problems, this invention provides an image-text similarity detection device, comprising: an image-text determination module, configured to determine multiple images to be detected and text to be detected corresponding to each image to be detected; and a numerical vector determination module, configured to perform feature extraction on each image to be detected using an image feature extraction algorithm to determine an image numerical vector of the image to be detected, the image numerical vector containing a first preset number of floating-point values, and to perform feature extraction on the text to be detected corresponding to the image to be detected using a text feature extraction algorithm to determine a text numerical vector of the text to be detected, the text numerical vector containing a second preset number of floating-point values. The system comprises: a point value module; an alignment processing module, used to align image value vectors and / or text value vectors using an alignment algorithm, so that the number of floating-point values ​​contained in the image value vectors and text value vectors are equal, and the image value vectors and text value vectors are aligned one-to-one; an optimization module, used to iterate over one or more of the following optimization terms to minimize a preset loss function to determine the optimized optimization terms: the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm; and a similarity determination module, used to determine the similarity result between the image value vectors and text value vectors using the optimized optimization terms.

[0022] To address the aforementioned technical problems, embodiments of the present invention provide a computer-readable storage medium storing a computer program thereon, wherein the computer program, when run by a processor, executes the steps of the aforementioned image-text similarity detection method.

[0023] To address the aforementioned technical problems, this invention provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes the steps of the aforementioned image-text similarity detection method.

[0024] Compared with the prior art, the technical solution of the embodiments of the present invention has the following beneficial effects:

[0025] In this embodiment of the invention, image value vectors containing floating-point values ​​of the image to be detected and text value vectors containing floating-point values ​​are obtained through feature extraction. Alignment processing can obtain floating-point values ​​of equal quantity and one-to-one correspondence between the image value vectors and text value vectors, thereby utilizing the precision of floating-point values ​​to improve the effectiveness of comparison. Furthermore, in the optimization process, a preset loss function is used to optimize one or more of the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm, which is beneficial to improve the image-text similarity detection algorithm more comprehensively and more specifically, thereby obtaining more accurate similarity results.

[0026] Furthermore, the preset loss function is determined based on the similarity between the image value vectors of each aligned image to be detected and the corresponding text value vectors, as well as the preset target similarity. By using the scheme implemented in this invention to determine the loss function, the consistency with the final determined result can be improved, thereby making the selection of the loss function more targeted and accurate.

[0027] Furthermore, the alignment processing of image numerical vectors and / or text numerical vectors includes: constructing an N×M neural network matrix; using the product of the numerical vector to be aligned and the neural network matrix as the numerical vector to be aligned. By adopting the scheme of the present invention, an N×M neural network matrix is ​​constructed based on matrix multiplication, which can provide a more suitable iterative basis for subsequent optimization steps while obtaining the alignment result.

[0028] Furthermore, in determining the optimized alignment algorithm, the gradient descent algorithm is used to update the N×M neural network matrix. Based on the N×M neural network matrix obtained in each update, the image value vector and the corresponding text value vector of each image to be detected after alignment are determined, and the loss function is calculated until the loss function converges to the target value or reaches the preset number of iterations, thus obtaining the optimized N×M neural network matrix. Using the scheme of this embodiment, an optimal N×M neural network matrix can be obtained through iterative optimization, improving the alignment accuracy and achieving a better floating-point numerical correspondence between images and text.

[0029] Furthermore, in the first p iterations, the N×M neural network matrix is ​​updated using a first learning rate in the gradient descent algorithm, and in the subsequent q iterations, the N×M neural network matrix is ​​updated using a second learning rate in the gradient descent algorithm; wherein the first learning rate is greater than the second learning rate. By employing the scheme of this embodiment, higher optimization efficiency can be obtained by utilizing a large learning rate in the first p iterations, and higher accuracy can be obtained by utilizing a small learning rate in the subsequent q iterations.

[0030] Furthermore, in the process of optimizing the image feature extraction algorithm and / or the text feature extraction algorithm, a gradient descent algorithm is used to update the parameters in the image feature extraction algorithm and / or the text feature extraction algorithm. Based on each updated algorithm, the image value vector and the corresponding text value vector of each image to be detected are determined. Then, based on the aligned image value vectors of each image to be detected and the corresponding text value vectors, the loss function is calculated until the loss function converges to the target value or reaches the preset number of iterations, thus obtaining the optimized parameters in the image feature extraction algorithm and / or the text feature extraction algorithm. Using the scheme of this embodiment, the preferred parameters in the image feature extraction algorithm and / or the text feature extraction algorithm can be obtained through iterative optimization, improving the accuracy of the algorithm and making the extraction of floating-point values ​​of images and text more accurate.

[0031] Furthermore, only the parameters in the first preset number of layers at the end of the image feature extraction algorithm are updated, and / or only the parameters in the second preset number of layers at the end of the text feature extraction algorithm are updated; wherein the first preset number and the second preset number are the same or different. By adopting the scheme of this embodiment of the invention, updating only the end layers allows for small-scale optimization (fine-tuning) of the algorithm, even when the number of samples learned by the algorithm is large, whereas the number of samples in this embodiment of the invention is small, while avoiding large-scale modifications to the algorithm.

[0032] Furthermore, updating only the parameters in the fully connected layers of the image feature extraction algorithm, and / or updating only the parameters in the fully connected layers of the text feature extraction algorithm, can improve the stability of the algorithm by updating each layer in the fully connected layers during small-scale optimization.

[0033] Furthermore, the image numerical vector and the text numerical vector are floating-point values ​​of one-dimensional vectors, which can improve the consistency of alignment based on one-dimensional vectors and further enhance the reliability of the solution in the embodiment of the present invention. Attached Figure Description

[0034] Figure 1 This is a flowchart of an image-text similarity detection method according to an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of a working scenario for a method of determining image numerical vectors and text numerical vectors according to an embodiment of the present invention;

[0036] Figure 3 yes Figure 1 A flowchart of a specific implementation of step S13;

[0037] Figure 4This is a schematic diagram of the structure of an image-text similarity detection device according to an embodiment of the present invention. Detailed Implementation

[0038] In existing technologies, there is a lack of accuracy and effectiveness assessment for the conversion of text obtained from image recognition and images obtained from text rendering. In other words, the accuracy of similarity judgments obtained using existing image-text similarity detection technologies is relatively low.

[0039] Research has revealed that existing image-text similarity detection methods typically employ keyword extraction and comparison. For example, keywords / words are extracted from the text; paragraphs can be labeled as keywords, and the relative frequencies of these keywords can generate metrics such as term frequency-inverse document frequency (TF-IDF), which are then searched within the image. Another approach involves identifying keywords within the image, for instance, by creating feature variables based on pixel values, hue, saturation, and value (HSV), color, and image texture, and then searching within the text. However, these image-text similarity detection methods rely on statistical results from image pixels or text markers, rather than on the true semantics conveyed by the image and text. This can lead to lower accuracy due to the influence of keyword extraction / recognition algorithms. Furthermore, the recognition process is independent of the similarity detection process, making it difficult to improve the recognition algorithm based on the similarity detection results.

[0040] In this embodiment of the invention, image value vectors containing floating-point values ​​of the image to be detected and text value vectors containing floating-point values ​​are obtained through feature extraction. Alignment processing yields an equal number of floating-point values ​​that correspond one-to-one with the image value vectors and text value vectors. This leverages the low computational complexity and high accuracy of floating-point values ​​to improve the effectiveness of comparison. Furthermore, during the optimization process, a preset loss function is used to optimize one or more of the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm. This facilitates a more comprehensive and targeted improvement of the image-text similarity detection algorithm, resulting in more accurate similarity results.

[0041] To make the above-mentioned objectives, features and beneficial effects of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0042] Reference Figure 1 , Figure 1 This is a flowchart of an image-text similarity detection method according to an embodiment of the present invention. The image-text similarity detection method may include steps S11 to S15:

[0043] Step S11: Determine multiple images to be detected, and the text to be detected corresponding to each image;

[0044] Step S12: Use an image feature extraction algorithm to extract features from each image to be detected to determine the image value vector of the image to be detected. The image value vector contains a first preset number of floating-point values. Use a text feature extraction algorithm to extract features from the text to be detected corresponding to the image to be detected to determine the text value vector of the text to be detected. The text value vector contains a second preset number of floating-point values.

[0045] Step S13: Use an alignment algorithm to align the image numerical vector and / or text numerical vector so that the number of floating-point values ​​contained in the image numerical vector and the text numerical vector are equal, and the image numerical vector and the text numerical vector are aligned one-to-one.

[0046] Step S14: Iterate over one or more of the following items to be optimized, and minimize the preset loss function to determine the optimized items: the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm;

[0047] Step S15: Using the optimized terms to be optimized, determine the similarity result between the image numerical vector and the text numerical vector.

[0048] In the specific implementation of step S11, the image to be detected and the text to be detected that have a corresponding relationship are determined.

[0049] In one specific embodiment of the present invention, the corresponding image to be detected and the text to be detected can be obtained based on a recognition algorithm, such as a text recognition algorithm or an image recognition algorithm. Thus, after obtaining the similarity result using the scheme of the present invention, the accuracy of the recognition algorithm can also be verified.

[0050] In another specific embodiment of the present invention, the image to be detected and the text to be detected that have a corresponding relationship can also be randomly selected, thereby improving the robustness of the detection method by performing detection based on images and texts with a larger similarity span.

[0051] In the specific implementation of step S12, the image numerical vector and the text numerical vector are determined by feature extraction.

[0052] Reference Figure 2 , Figure 2 This is a schematic diagram illustrating a working scenario of a method for determining image numerical vectors and text numerical vectors in an embodiment of the present invention.

[0053] like Figure 2As shown, the branch network layer contains two branch networks, which respectively use appropriate image feature extraction algorithms to extract features from the image to be detected, and appropriate text feature extraction algorithms to extract features from the text to be detected. The two branch networks can be connected to the prediction network layer to convert the two modalities into a common modality. Then, similarity detection (also known as similarity comparison) can be performed on the obtained image numerical vectors and the text numerical vectors.

[0054] Furthermore, the image feature extraction algorithm can be selected from the Deep ResNet (WRN) algorithm and the Visual Geometry Group (VGG) convolutional neural network algorithm.

[0055] Specifically, by employing the WRN algorithm, its ability to more effectively widen residual blocks at shallower depths can be leveraged to improve the performance of image classification and object detection.

[0056] Without limitation, the WRN can, for example, employ Wide ResNet 50-2 to improve image feature extraction performance.

[0057] Furthermore, the Wide ResNet 50-2 can have a depth of 50 and a widening factor of 2 to improve both the accuracy and efficiency of feature extraction.

[0058] Specifically, by employing the VGG algorithm, its high level of maturity can be leveraged to improve the stability of image feature extraction.

[0059] Without limitation, the VGG can be, for example, VGG16, thereby improving the image feature extraction effect.

[0060] Furthermore, the text feature extraction algorithm can be selected from bidirectional encoder representation (BERT) and long short-term memory (LSTM) models.

[0061] Specifically, the BERT algorithm is characterized by generating embedding vectors for each word by learning from the context. Its core idea is to pre-train the network on a large corpus in order to generate general semantic embeddings. Compared with using pre-designed statistical algorithms (such as TF-IDF mentioned above), it has better text feature extraction performance.

[0062] Without limitation, the BERT algorithm may, for example, employ BERT-base to improve text feature extraction performance.

[0063] Furthermore, BERT-base can contain 12 encoder layers (Transformers), each encoder can contain 12 attention heads, and the number of neurons in the last hidden layer can be selected from 768, totaling 110 million parameters, in order to improve both the accuracy and efficiency of feature extraction.

[0064] Specifically, by employing the LSTM algorithm, its ability to model temporal information in text can be utilized to extract text features.

[0065] It should be noted that subsequent steps also include improvements to one or more algorithms, including the image feature extraction algorithm and the text feature extraction algorithm, utilizing the optimized... Figure 2 The model embeds images and text into a common modality of images and text, represented as numerical vectors, and calculates the distance between vectors to capture semantic consistency between image and text pairs. This model connects two isolated processes into a unified framework, resulting in better synergy and contributing to more accurate consistency measurements.

[0066] The image numerical vector can contain a first preset number of floating-point values, and the text numerical vector can contain a second preset number of floating-point values. This allows for the utilization of the low computational complexity and high precision of floating-point values ​​to improve the effectiveness of comparison.

[0067] Furthermore, the image numerical vector and the text numerical vector can be floating-point values ​​of one-dimensional vectors.

[0068] In this embodiment of the invention, the image numerical vector and the text numerical vector are floating-point values ​​of one-dimensional vectors, which can improve the consistency of alignment based on one-dimensional vectors and further enhance the reliability of the solution in this embodiment of the invention.

[0069] Continue to refer to Figure 1 In the specific implementation of step S13, it is necessary to align the image numerical vector and / or text numerical vector. This can be achieved by making the processed image numerical vector and text numerical vector equal in numerical quantity when the first preset quantity is not equal to the second preset quantity. Furthermore, based on the one-to-one correspondence between each floating-point value of the processed image numerical vector and each floating-point value of the text numerical vector, the correlation between the corresponding floating-point values ​​can be improved.

[0070] Reference Figure 3 , Figure 3 yes Figure 1A flowchart illustrating a specific implementation of step S13. The method for aligning image numerical vectors and / or text numerical vectors may include steps S31 to S32:

[0071] Step S31: Construct an N×M neural network matrix;

[0072] Where M represents the number of floating-point values ​​contained in the numerical vector to be aligned, and N represents the number of floating-point values ​​contained in the numerical vector being aligned.

[0073] The numerical vector to be aligned can be an image numerical vector and / or a text numerical vector.

[0074] Furthermore, the numerical vector to be aligned is one of the image numerical vector and the text numerical vector, and the numerical vector being aligned is the other of the image numerical vector and the text numerical vector.

[0075] In one non-limiting embodiment, the image value vector contains 128 floating-point values, and the text value vector contains 256 floating-point values. Therefore, the image value vector can be used as the value vector to be aligned, and the text value vector can be used as the value vector to be aligned, thus obtaining an aligned image value vector with 256 floating-point values. Conversely, an aligned text value vector with 128 floating-point values ​​can also be obtained.

[0076] Furthermore, the number of floating-point values ​​contained in the numerical vector to be aligned can also be other appropriate values, that is, values ​​other than those contained in image numerical vectors and text numerical vectors. Thus, when the number of floating-point values ​​in the numerical vector to be aligned is too large or too small, better processing efficiency or processing results can be obtained by setting other appropriate values.

[0077] In another non-limiting embodiment, the image value vector contains 128 floating-point values ​​and the text value vector contains 256 floating-point values. In this case, both the image value vector and the text value vector can be used as value vectors to be aligned, and 64 floating-point values ​​can be used as the value vectors to be aligned, thereby obtaining an aligned image value vector and an aligned text value vector with 64 floating-point values.

[0078] Step S32: The product of the numerical vector to be aligned and the neural network matrix is ​​used as the numerical vector to be aligned.

[0079] Specifically, by using matrix multiplication, the processed numerical vector to be aligned can have the required number of floating-point values.

[0080] It should be noted that the initial values ​​of the N×M neural network matrix can be determined according to the actual situation. For example, they can be all zeros or set randomly.

[0081] In this embodiment of the invention, by constructing an N×M neural network matrix, the product of the numerical vector to be aligned and the neural network matrix is ​​used as the numerical vector to be aligned. By adopting the scheme of this embodiment of the invention, an N×M neural network matrix is ​​constructed based on matrix multiplication, which can provide a more suitable iterative basis for subsequent optimization steps while obtaining the alignment result.

[0082] It should be noted that the alignment processing method of the present invention is not limited to an N×M neural network matrix. Other appropriate methods can also be used for alignment processing. For example, zeros can be padded to align a numerical vector with a small number of floating-point values.

[0083] Continue to refer to Figure 1 In the specific implementation of step S14, one or more of the following items to be optimized can be iterated to minimize the preset loss function to determine the optimized items: the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm.

[0084] In this embodiment of the invention, by optimizing one or more of the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm, it is beneficial to balance comprehensiveness and specificity in the process of improving the image-text similarity detection algorithm.

[0085] Furthermore, the preset loss function is determined based on the similarity between the image value vectors of each aligned image to be detected and the corresponding text value vectors, as well as the preset target similarity.

[0086] In one specific embodiment of the present invention, the preset loss function can be determined based on the difference between the similarity between the image value vectors of each aligned image to be detected and the corresponding text value vectors, and the preset target similarity.

[0087] In this embodiment of the invention, the preset loss function is determined based on the similarity between the image value vectors of each aligned image to be detected and the corresponding text value vectors, as well as the preset target similarity. By using the scheme implemented in this invention to determine the loss function, the consistency with the final determined result can be improved, thereby making the selection of the loss function more targeted and accurate.

[0088] Furthermore, the loss function can be determined using the following formula:

[0089]

[0090] or,

[0091] Where L(Y',Y) represents the loss function, Y i Y is used to represent the similarity between the aligned image value vector of the i-th image to be detected and the corresponding text value vector. i The value is used to represent the preset target similarity, and k is used to represent the number of images to be detected, where 1≤i≤k and i and k are both positive integers.

[0092] The target similarity can be manually determined or obtained based on historical experience data.

[0093] In another specific embodiment of the present invention, the adjustment factor can be multiplied by the similarity and / or target similarity to obtain the adjusted similarity and / or target similarity, and then the loss function can be determined based on the difference between the adjusted similarity and / or target similarity.

[0094] Furthermore, the step of iterating the alignment algorithm and minimizing the preset loss function to determine the optimized alignment algorithm may include: updating the N×M neural network matrix using the gradient descent algorithm; based on the N×M neural network matrix obtained in each update, determining the image value vector and the corresponding text value vector of each image to be detected after alignment; and calculating the loss function until the loss function converges to the target value or reaches the preset number of iterations, thereby obtaining the optimized N×M neural network matrix.

[0095] The loss function converges to a target value, for example, it may converge to a preset target similarity.

[0096] In this embodiment of the invention, during the process of determining the optimized alignment algorithm, the gradient descent algorithm is used to update the N×M neural network matrix. Based on the N×M neural network matrix obtained from each update, the image value vector and the corresponding text value vector of each image to be detected after alignment are determined, and the loss function is calculated until the loss function converges to the target value or reaches the preset number of iterations, thus obtaining the optimized N×M neural network matrix. Using the scheme of this embodiment of the invention, an optimal N×M neural network matrix can be obtained through iterative optimization, improving the alignment accuracy and achieving a better floating-point numerical correspondence between images and text.

[0097] Furthermore, in the first p iterations, the N×M neural network matrix is ​​updated using the first learning rate of the gradient descent algorithm, and in the subsequent q iterations, the N×M neural network matrix is ​​updated using the second learning rate of the gradient descent algorithm; wherein, the first learning rate is greater than the second learning rate; P is used to represent the total number of iterations in the gradient descent algorithm, 1 < p + q ≤ P, and p, q, and P are all positive integers.

[0098] In this embodiment of the invention, the N×M neural network matrix is ​​updated using a first learning rate in the gradient descent algorithm during the first p iterations, and then updated using a second learning rate in the gradient descent algorithm during the subsequent q iterations; wherein the first learning rate is greater than the second learning rate. By employing the scheme of this embodiment, higher optimization efficiency can be achieved by utilizing a larger learning rate in the first p iterations, and higher accuracy can be achieved by utilizing a smaller learning rate in the subsequent q iterations.

[0099] Further, the step of iterating the image feature extraction algorithm and / or the text feature extraction algorithm to minimize a preset loss function to determine the optimized image feature extraction algorithm and / or the text feature extraction algorithm may include: updating the parameters in the image feature extraction algorithm and / or the text feature extraction algorithm using a gradient descent algorithm; determining the image value vector and the corresponding text value vector of each image to be detected based on each updated algorithm; then calculating the loss function based on the aligned image value vector and the corresponding text value vector of each image to be detected; until the loss function converges to the target value or reaches a preset number of iterations, the optimized parameters in the image feature extraction algorithm and / or the text feature extraction algorithm are obtained.

[0100] The parameters in the updated image feature extraction algorithm and / or text feature extraction algorithm can be pre-selected parameters, or some or all of the learned parameters.

[0101] In this embodiment of the invention, during the optimization of the image feature extraction algorithm and / or the text feature extraction algorithm, a gradient descent algorithm is used to update the parameters of the image feature extraction algorithm and / or the text feature extraction algorithm. Based on each updated algorithm, the image value vector and the corresponding text value vector of each image to be detected are determined. Then, based on the aligned image value vectors of each image to be detected and the corresponding text value vectors, the loss function is calculated until the loss function converges to the target value or reaches a preset number of iterations, thus obtaining the optimized parameters of the image feature extraction algorithm and / or the text feature extraction algorithm. Using the scheme of this embodiment of the invention, the optimal parameters of the image feature extraction algorithm and / or the text feature extraction algorithm can be obtained through iterative optimization, improving the accuracy of the algorithm and making the extraction of floating-point values ​​of images and text more accurate.

[0102] Furthermore, the image feature extraction algorithm includes multiple differentiable nonlinear components, and / or the text feature extraction algorithm includes multiple differentiable nonlinear components; updating the parameters in the image feature extraction algorithm and / or the text feature extraction algorithm includes: updating only the parameters in the first preset number layer at the end of the image feature extraction algorithm, and / or updating only the parameters in the second preset number layer at the end of the text feature extraction algorithm; wherein the first preset number and the second preset number are the same or different.

[0103] In this embodiment of the invention, only the parameters in the first preset number of layers at the end of the image feature extraction algorithm are updated, and / or only the parameters in the second preset number of layers at the end of the text feature extraction algorithm are updated; wherein the first preset number and the second preset number are the same or different. By adopting the scheme of this embodiment of the invention, updating only the end layers allows for small-scale optimization (fine-tuning) of the algorithm, even when the number of samples learned by the algorithm is large, whereas the number of samples in this embodiment of the invention is small, without requiring large-scale modifications to the algorithm.

[0104] Furthermore, the image feature extraction algorithm includes a fully connected layer, and the fully connected layer is located at the end of the image feature extraction algorithm; and / or, the text feature extraction algorithm includes a fully connected layer, and the fully connected layer is located at the end of the text feature extraction algorithm; updating only the parameters in the first preset number of layers at the end of the image feature extraction algorithm includes updating only the parameters in the fully connected layer of the image feature extraction algorithm; updating only the parameters in the first preset number of layers at the end of the text feature extraction algorithm includes updating only the parameters in the fully connected layer of the text feature extraction algorithm.

[0105] In this embodiment of the invention, by updating only the parameters in the fully connected layer of the image feature extraction algorithm, and / or only the parameters in the fully connected layer of the text feature extraction algorithm, the stability of the algorithm can be improved during small-scale optimization by updating each layer in the fully connected layer and freezing the parameters before the fully connected layer.

[0106] It should be noted that the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm can also be iterated together to minimize a preset loss function to determine the optimized alignment algorithm, image feature extraction algorithm, and text feature extraction algorithm.

[0107] Without limitation, the gradient descent algorithm can be used to update the parameters in the alignment algorithm, image feature extraction algorithm, and text feature extraction algorithm. Based on each updated algorithm, the image value vector and the corresponding text value vector of each image to be detected are determined. Then, based on the aligned image value vector and the corresponding text value vector of each image to be detected, the loss function is calculated until the loss function converges to the target value or reaches the preset number of iterations, thus obtaining the optimized alignment algorithm, image feature extraction algorithm, and text feature extraction algorithm.

[0108] In the specific implementation of step S15, the optimized term to be optimized is used to determine the similarity result between the image numerical vector and the text numerical vector.

[0109] Specifically, appropriate algorithms can be used to calculate the distance between image numerical vectors and text numerical vectors. For example, cosine similarity can be used as a model to detect the congruence between image numerical vectors and text numerical vectors.

[0110] Without limitation, the cosine similarity can range from -1 to 1, and the greater the cosine similarity, the closer the semantics between the image and the text.

[0111] In this embodiment of the invention, image value vectors containing floating-point values ​​of the image to be detected and text value vectors containing floating-point values ​​are obtained through feature extraction. Alignment processing can obtain floating-point values ​​of equal quantity and one-to-one correspondence between the image value vectors and text value vectors, thereby utilizing the precision of floating-point values ​​to improve the effectiveness of comparison. Furthermore, in the optimization process, a preset loss function is used to optimize one or more of the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm, which is beneficial to improve the image-text similarity detection algorithm more comprehensively and more specifically, thereby obtaining more accurate similarity results.

[0112] Reference Figure 4 , Figure 4This is a schematic diagram of the structure of an image-text similarity detection device according to an embodiment of the present invention. The image-text similarity detection device may include:

[0113] Image text determination module 41 is used to determine multiple images to be detected, and the text to be detected corresponding to each image to be detected;

[0114] The numerical vector determination module 42 is used to perform feature extraction on each image to be detected using an image feature extraction algorithm to determine the image numerical vector of the image to be detected, wherein the image numerical vector contains a first preset number of floating-point values, and to perform feature extraction on the text to be detected corresponding to the image to be detected using a text feature extraction algorithm to determine the text numerical vector of the text to be detected, wherein the text numerical vector contains a second preset number of floating-point values.

[0115] Alignment processing module 43 is used to perform alignment processing on image numerical vectors and / or text numerical vectors using an alignment algorithm, so that the number of floating-point values ​​contained in the image numerical vectors and text numerical vectors are equal, and the image numerical vectors and text numerical vectors are aligned one-to-one.

[0116] Optimization module 44 is used to iterate over one or more of the following items to be optimized, and minimize a preset loss function to determine the optimized items: the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm;

[0117] The similarity determination module 45 is used to determine the similarity result between the image numerical vector and the text numerical vector using the optimized terms to be optimized.

[0118] about Figure 4 For more information on the working principle, operation mode, and beneficial effects of the image-text similarity detection method and device shown, please refer to the preceding text and... Figures 1 to 3 The relevant descriptions will not be repeated here.

[0119] This invention also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it performs the steps of the image-text similarity detection method described above. The storage medium may include ROM, RAM, a disk, or an optical disk, etc. The storage medium may also include non-volatile memory or non-transitory memory, etc.

[0120] This invention also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it performs the steps of the image-text similarity detection method described above. The terminal includes, but is not limited to, mobile phones, computers, tablets, servers, cloud platforms, and other terminal devices.

[0121] It should be understood that in the embodiments of this application, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0122] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0123] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.

[0124] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or in the form of hardware plus software functional units. For example, for various devices or products applied to or integrated into a chip, each module / unit contained therein can be implemented in hardware such as circuits, or at least some modules / units can be implemented in software programs that run on the processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in hardware such as circuits; for various devices or products applied to or integrated into a chip module, each module / unit contained therein can be implemented in hardware such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented in software programs that run on the processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in hardware such as circuits.

[0125] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0126] In the embodiments of this application, "multiple" refers to two or more.

[0127] The descriptions of "first," "second," etc., appearing in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They have no order and do not indicate any special limitation on the number of devices in the embodiments of this application, nor do they constitute any limitation on the embodiments of this application.

[0128] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. An image-text similarity detection method, characterized in that, include: Identify multiple images to be detected, and the corresponding text to be detected for each image; An image feature extraction algorithm is used to extract features from each image to be detected in order to determine the image value vector of the image to be detected. The image value vector contains a first preset number of floating-point values. A text feature extraction algorithm is used to extract features from the text to be detected corresponding to the image to be detected in order to determine the text value vector of the text to be detected. The text value vector contains a second preset number of floating-point values. An alignment algorithm is used to align the image numerical vector and / or the text numerical vector so that the number of floating-point values ​​contained in the image numerical vector and the text numerical vector are equal, and the image numerical vector and the text numerical vector are aligned one-to-one. The following one or more optimization items are iterated to minimize a preset loss function to determine the optimized optimization items: the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm; The preset loss function is determined based on the difference between the similarity between the image value vectors of each aligned image to be detected and the corresponding text value vectors, and the preset target similarity. Using the optimized terms to be optimized, the similarity results between the image numerical vector and the text numerical vector are determined. An alignment algorithm is used to align image numerical vectors and / or text numerical vectors, including: Construct an N×M neural network matrix; The product of the numerical vector to be aligned and the neural network matrix is ​​used as the numerical vector to be aligned. Where M represents the number of floating-point values ​​contained in the numerical vector to be aligned, and N represents the number of floating-point values ​​contained in the numerical vector being aligned.

2. The method according to claim 1, characterized in that, The loss function is determined using the following formula: ； or, ; in, Used to represent the loss function, This is used to represent the similarity between the aligned image value vector of the i-th image to be detected and the corresponding text value vector. The value is used to represent the preset target similarity, and k is used to represent the number of images to be detected, where 1≤i≤k and i and k are both positive integers.

3. The method according to claim 1, characterized in that, The numerical vector to be aligned is one of the image numerical vector and the text numerical vector, and the numerical vector being aligned is the other of the image numerical vector and the text numerical vector.

4. The method according to claim 1, characterized in that, The alignment algorithm is iterated to minimize a pre-defined loss function to determine the optimized alignment algorithm, including: The N×M neural network matrix is ​​updated using a gradient descent algorithm. Based on the N×M neural network matrix obtained in each update, the image value vector and the corresponding text value vector of each image to be detected after alignment are determined, and the loss function is calculated until the loss function converges to the target value or reaches the preset number of iterations, thus obtaining the optimized N×M neural network matrix.

5. The method according to claim 4, characterized in that, In the first p iterations, the N×M neural network matrix is ​​updated using the first learning rate of the gradient descent algorithm, and in the next q iterations, the N×M neural network matrix is ​​updated using the second learning rate of the gradient descent algorithm. Wherein, the first learning rate is greater than the second learning rate; P is used to represent the total number of iterations in the gradient descent algorithm, 1 < p + q ≤ P, where p, q, and P are all positive integers.

6. The method according to claim 4, characterized in that, Iterating through the image feature extraction algorithm and / or the text feature extraction algorithm to minimize a preset loss function to determine the optimized image feature extraction algorithm and / or the text feature extraction algorithm includes: The parameters in the image feature extraction algorithm and / or the text feature extraction algorithm are updated using a gradient descent algorithm. Based on each updated algorithm, the image value vector and the corresponding text value vector of each image to be detected are determined. Then, based on the aligned image value vectors and the corresponding text value vectors of each image to be detected, the loss function is calculated until the loss function converges to the target value or reaches the preset number of iterations, thus obtaining the optimized parameters in the image feature extraction algorithm and / or the text feature extraction algorithm.

7. The method according to claim 6, characterized in that, The image feature extraction algorithm includes multiple differentiable nonlinear components, and / or the text feature extraction algorithm includes multiple differentiable nonlinear components. Updating the parameters in the image feature extraction algorithm and / or the text feature extraction algorithm includes: Only update the parameters in the first preset number of layers at the end of the image feature extraction algorithm, and / or only update the parameters in the second preset number of layers at the end of the text feature extraction algorithm; The first preset quantity may be the same as or different from the second preset quantity.

8. The method according to claim 7, characterized in that, The image feature extraction algorithm includes a fully connected layer, and the fully connected layer is located at the end of the image feature extraction algorithm; and / or, the text feature extraction algorithm includes a fully connected layer, and the fully connected layer is located at the end of the text feature extraction algorithm. The step of only updating the parameters in the first preset number of layers at the end of the image feature extraction algorithm includes: only updating the parameters in the fully connected layers of the image feature extraction algorithm; The step of updating only the parameters in the first preset number of layers at the end of the text feature extraction algorithm includes updating only the parameters in the fully connected layers of the text feature extraction algorithm.

9. The method according to claim 7, characterized in that, Meet one or more of the following: The image feature extraction algorithm is selected from the Deep Residual Network (WRN) algorithm and the VGG convolutional neural network. The text feature extraction algorithm is selected from the bidirectional encoding representation algorithm BERT and the Long Short-Term Memory (LSTM) model algorithm.

10. The method according to claim 1, characterized in that, The image numerical vector and the text numerical vector are floating-point values ​​of one-dimensional vectors.

11. An image-text similarity detection device, characterized in that, include: The image text determination module is used to determine multiple images to be detected, and the text to be detected corresponding to each image; The numerical vector determination module is used to extract features from each image to be detected using an image feature extraction algorithm to determine the image numerical vector of the image to be detected, wherein the image numerical vector contains a first preset number of floating-point values, and to extract features from the text to be detected corresponding to the image to be detected using a text feature extraction algorithm to determine the text numerical vector of the text to be detected, wherein the text numerical vector contains a second preset number of floating-point values. The alignment processing module is used to perform alignment processing on the image value vector and / or the text value vector using an alignment algorithm, so that the number of floating-point values ​​contained in the image value vector and the text value vector are equal, and the image value vector and the text value vector are aligned in a one-to-one correspondence. The optimization module is used to iterate over one or more of the following items to be optimized, and minimize a preset loss function to determine the optimized items: the alignment algorithm, the image feature extraction algorithm, and the text feature extraction algorithm; The preset loss function is determined based on the difference between the similarity between the image value vectors of each aligned image to be detected and the corresponding text value vectors, and the preset target similarity. The similarity determination module is used to determine the similarity result between the image numerical vector and the text numerical vector using the optimized terms to be optimized; The alignment processing module is used to construct an N×M neural network matrix; The product of the numerical vector to be aligned and the neural network matrix is ​​used as the numerical vector to be aligned; where M represents the number of floating-point values ​​contained in the numerical vector to be aligned, and N represents the number of floating-point values ​​contained in the numerical vector to be aligned.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run by a processor, performs the steps of the image-text similarity detection method according to any one of claims 1 to 10.

13. A terminal comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the image-text similarity detection method according to any one of claims 1 to 10.

Images