Similar image acquisition method and device, electronic equipment and storage medium
By combining image processing models and encoders, textual description information of target images is extracted and encoded, solving the problem in existing technologies where image feature vectors are difficult to combine with geographic time, and enabling the rapid and accurate acquisition of similar images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING REALAI TECH CO LTD
- Filing Date
- 2023-06-28
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, image feature vector extraction methods are difficult to combine with geographic time, making it difficult to quickly and accurately obtain similar images to the target image.
Textual description information of the target image is extracted using an image processing model. Combined with location and time information, the target feature vector is obtained by encoding through an encoder. The similarity with the feature vector in the image database is then calculated to determine similar images.
It enables the rapid and accurate acquisition of similar images from the image library, improving the efficiency and accuracy of similar image acquisition.
Smart Images
Figure CN116775928B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for acquiring similar images. Background Technology
[0002] When searching for similar images based on a target image, feature vectors of the target image and the image to be compared are usually extracted, and similar images of the target image are determined based on the similarity of the feature vectors between the target image and the image to be compared.
[0003] Current methods for extracting image feature vectors include: Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Oriented Fast and Rotated Brief (ORB), and deep learning-based feature extraction methods.
[0004] Current methods for comparing feature vector similarity typically involve enumerating the feature vector similarity between the target image and the image to be compared, or using the K-Nearest Neighbor (KNN) method to calculate the feature vector similarity between images.
[0005] Current image feature vector extraction methods struggle to combine extracted image features with geographic time, making it difficult to quickly and accurately obtain similar images to the target image. Summary of the Invention
[0006] This disclosure provides a method, apparatus, electronic device, and storage medium for acquiring similar images to solve the above-mentioned problems.
[0007] A first aspect of this disclosure provides a method for acquiring similar images, including:
[0008] The target image is processed using an image processing model to obtain the text description information of the target image, and the text description information of multiple images in the image library is obtained. The text description information of each image in the multiple images includes the location information and time information of the corresponding image.
[0009] Location and time information are added to the text description information of the target image to obtain the target text description information;
[0010] The target text description information is encoded using an encoder to obtain a target feature vector, and multiple feature vectors corresponding to the text description information of each of the multiple images are obtained.
[0011] Calculate the similarity between the target feature vector and the plurality of feature vectors, and determine the similar images of the target image based on the similarity calculation results.
[0012] In one embodiment of this disclosure, the step of encoding the target text description information using an encoder to obtain a target feature vector includes:
[0013] Based on the target text description information, determine the target input embedding matrix;
[0014] The target input embedding matrix is transformed to obtain the input signal of the target text description information;
[0015] The input signal of the target text description information is processed using the N-layer structure of the encoder to obtain the target feature vector, where N is an integer greater than 1. Each layer of the N-layer structure includes a multi-head attention sublayer, a first normalization sublayer corresponding to the multi-head attention sublayer, a forward feedback sublayer, and a second normalization sublayer corresponding to the forward feedback sublayer.
[0016] In one embodiment of this disclosure, the step of transforming the target input embedding matrix to obtain the input signal of the target text description information includes:
[0017] The position of the words in the target text description information is determined as the time coordinate, and the value of the column vector in the target input embedding matrix is determined as the signal strength, thereby obtaining the semantic signal of the target text description information;
[0018] The semantic signal of the target text description information is subjected to discrete transformation processing to obtain the input signal of the target text description information.
[0019] In one embodiment of this disclosure, the step of performing discrete transformation processing on the semantic signal of the target text description information to obtain the input signal of the target text description information includes:
[0020] A one-dimensional discrete Fourier transform is performed on the semantic signal of the target text description information to obtain the target frequency domain signal, and the target frequency domain signal is determined as the target input signal.
[0021] In one embodiment of this disclosure, the step of performing discrete transformation processing on the semantic signal of the target text description information to obtain the input signal of the target text description information includes:
[0022] The semantic signal of the target text description information is subjected to discrete cosine transform to obtain the target frequency domain signal, and the target time domain signal is determined as the target input signal.
[0023] In one embodiment of this disclosure, calculating the similarity between the target feature vector and the plurality of feature vectors, and determining similar images of the target image based on the similarity calculation results, includes:
[0024] Cosine similarity calculation is performed on the target feature vector and the plurality of feature vectors to obtain multiple cosine similarities between the target feature vector and the plurality of feature vectors respectively;
[0025] The cosine similarities are sorted in descending order, and a list of similar images to the target image is determined based on the descending order result.
[0026] In one embodiment of this disclosure, before obtaining the text description information of each of the multiple images in the image library, the method further includes:
[0027] The image processing model is used to process the multiple images to obtain text description information for each of the multiple images;
[0028] The encoder is used to encode the text description information of each of the multiple images to obtain the multiple feature vectors.
[0029] A second aspect of this disclosure provides a similar image acquisition apparatus, comprising:
[0030] The text description information acquisition module is used to process the target image using an image processing model to obtain the text description information of the target image, and to acquire the text description information of multiple images in the image library. The text description information of each image in the multiple images includes the location information and time information of the corresponding image.
[0031] The information adding module is used to add location information and time information to the text description information of the target image to obtain the target text description information;
[0032] The feature encoding module is used to encode the target text description information using an encoder to obtain a target feature vector, and to obtain multiple feature vectors corresponding to the text description information of each of the multiple images.
[0033] A similar image determination module is used to calculate the similarity between the target feature vector and the plurality of feature vectors, and to determine similar images of the target image based on the similarity calculation results.
[0034] In one embodiment of this disclosure, the feature encoding module is used to determine a target input embedding matrix based on the target text description information; the feature encoding module is also used to transform the target input embedding matrix to obtain an input signal of the target text description information; the feature encoding module is also used to process the input signal of the target text description information using the N-layer structure of the encoder to obtain the target feature vector, wherein N is an integer greater than 1, and each layer of the N-layer structure includes a multi-head attention sublayer, a first normalization sublayer corresponding to the multi-head attention sublayer, a forward feedback sublayer, and a second normalization sublayer corresponding to the forward feedback sublayer.
[0035] In one embodiment of this disclosure, the feature encoding module is used to determine the position of the words in the target text description information as time coordinates, and to determine the value of the column vector in the target input embedding matrix as signal strength, thereby obtaining the semantic signal of the target text description information; the feature encoding module is also used to perform discrete transformation processing on the semantic signal of the target text description information to obtain the input signal of the target text description information.
[0036] In one embodiment of this disclosure, the feature encoding module is used to perform a one-dimensional discrete Fourier transform on the semantic signal of the target text description information to obtain a target frequency domain signal, and to determine the target frequency domain signal as the target input signal.
[0037] In one embodiment of this disclosure, the feature encoding module is used to perform discrete cosine transform on the semantic signal of the target text description information to obtain the target frequency domain signal, and to determine the target time domain signal as the target input signal.
[0038] In one embodiment of this disclosure, the similar image determination module is used to calculate the cosine similarity between the target feature vector and the plurality of feature vectors to obtain a plurality of cosine similarities between the target feature vector and the plurality of feature vectors respectively; the similar image determination module is further used to sort the plurality of cosine similarities in descending order and determine a list of similar images of the target image based on the descending order result.
[0039] In one embodiment of this disclosure, the similar image acquisition device further includes:
[0040] The preprocessing module is used to process the multiple images using the image-text processing model to obtain text description information for each of the multiple images, and to encode the text description information for each of the multiple images using the encoder to obtain the multiple feature vectors.
[0041] A third aspect of this disclosure provides an electronic device, characterized in that it includes:
[0042] Memory, used to store computer program products;
[0043] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the method described in the first aspect above.
[0044] A fourth aspect of this disclosure provides a computer-readable storage medium having computer program instructions stored thereon, characterized in that, when executed by a processor, the computer program instructions implement the method described in the first aspect above.
[0045] The similar image acquisition method, apparatus, electronic device, and storage medium of this disclosure utilize a pre-trained image processing model to process a target image to obtain text description information of the target image. By combining location information and time information, the target text description information can be obtained. Then, an encoder is used to encode the target text description information to obtain a target feature vector. The similarity between the target feature vector and the feature vector corresponding to the image in the image library is calculated. Based on the similarity calculation result, similar images of the target image can be obtained quickly and accurately.
[0046] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0047] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0048] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:
[0049] Figure 1 This is a flowchart of a similar image acquisition method in one embodiment of the present disclosure;
[0050] Figure 2 This is a schematic diagram of the encoder working principle in one example of this disclosure;
[0051] Figure 3 This is a structural block diagram of a similar image acquisition device in one embodiment of the present disclosure;
[0052] Figure 4 This is a structural block diagram of an electronic device in one embodiment of the present disclosure. Detailed Implementation
[0053] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.
[0054] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.
[0055] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.
[0056] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.
[0057] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.
[0058] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.
[0059] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.
[0060] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0061] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0062] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.
[0063] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.
[0064] Figure 1 This is a flowchart of a similar image acquisition method in one embodiment of this disclosure. Figure 1 As shown, the method for obtaining similar images includes the following steps:
[0065] S1: The target image is processed using an image processing model to obtain its textual description information, and the textual description information of multiple images in the image library is also obtained. The textual description information of each image includes its location and time information.
[0066] The target image is processed using a pre-trained image processing model to obtain the text description information of the target image.
[0067] The image processing model is trained based on sample images and their corresponding text description labels. In one example disclosed herein, the sample images may include at least a portion of the images in an image library, or all the images in the image library.
[0068] S2: Add location and time information to the text description information of the target image to obtain the target text description information.
[0069] If the text description information of the target image meets the user's needs, location and time information can be added to the text description information to obtain the target text description information. If the text description information of the target image does not meet the user's needs, the system can receive the user's correction instruction for the text description information of the target image. After adjusting the text description information according to the correction instruction, location and time information are added to obtain the target text description information. The user can browse the text description information of the target image on the terminal and determine whether the text description information output by the image processing model meets the user's needs by pressing the "meets needs" and "does not meet needs" buttons.
[0070] S3: Use an encoder to encode the target text description information to obtain the target feature vector, and obtain multiple feature vectors corresponding to the text description information of multiple images.
[0071] The target text description information is processed to obtain the encoder's input signal. This input signal is then fed into the encoder for further processing to obtain the target feature vector, which is the feature vector corresponding to the target text description information. Furthermore, the multiple feature vectors corresponding to the text description information of multiple images can be obtained by pre-processing the text description information of each image using this encoder.
[0072] S4: Calculate the similarity between the target feature vector and multiple feature vectors, and determine the similar images of the target image based on the similarity calculation results.
[0073] In one example of this disclosure, the number of similar images can be set, the similarity between the target feature vector and multiple feature vectors can be sorted, and similar images can be selected in descending order of similarity.
[0074] In another example of this disclosure, a vector similarity threshold can be set, and feature vectors with a similarity greater than the vector similarity threshold can be selected from multiple feature vectors. The images corresponding to the feature vectors with a similarity greater than the vector similarity threshold are determined as similar images to the target image.
[0075] In this embodiment, a pre-trained image processing model is used to process the target image to obtain the text description information of the target image. Combined with the location information and time information, the target text description information can be obtained. Then, the target text description information is encoded by an encoder to obtain the target feature vector. The similarity between the target feature vector and the feature vector corresponding to the image in the image library is calculated. Based on the similarity calculation result, similar images of the target image can be accurately obtained.
[0076] In one embodiment of this disclosure, step S3, which involves encoding the target text description information using an encoder to obtain the target feature vector, includes:
[0077] S3-1: Determine the target input embedding matrix based on the target text description information.
[0078] In one example disclosed herein, the target text description is "I am a robot" of length L=4. Each word in the target text description is transformed into a feature vector of dimension d using word embedding methods such as word2vec, resulting in the target input embedding matrix representation of the target text description as follows:
[0079]
[0080] S3-2: Transform the target input embedding matrix to obtain the input signal of the target text description information.
[0081] S3-3: The input signal of the target text description information is processed by the N-layer structure of the encoder to obtain the target feature vector, where N is an integer greater than 1. Each layer in the N-layer structure includes a multi-head attention sub-layer, a first normalization sub-layer corresponding to the multi-head attention sub-layer, a forward feedback sub-layer, and a second normalization sub-layer corresponding to the forward feedback sub-layer.
[0082] Figure 2 This is a schematic diagram illustrating the working principle of an encoder in one example of this disclosure. For example... Figure 2 As shown, the input signal of the encoder is obtained after transforming the target input embedding matrix. This input signal is then fed into an N-layer structure of the encoder, which processes the input signal to output the target feature vector. Within any layer of the N-layer structure, a multi-head attention sublayer is used to detect the relationships between words in the target input embedding matrix; a first normalization sublayer normalizes the parameters of the target input embedding matrix; a forward feedback sublayer processes the results of the multi-head attention sublayer, increasing the difference between different layers; and a second normalization sublayer normalizes the parameters based on the inputs of the first normalization sublayer and the forward feedback sublayer.
[0083] In this embodiment, an encoder with an N-layer structure, each layer including a multi-head attention sublayer, a first normalization sublayer, a forward feedback sublayer, and a second normalization sublayer, can quickly obtain the target feature vector corresponding to the target text description information. The use of a multi-head attention sublayer ensures the encoder's high generalization ability.
[0084] In one embodiment of this disclosure, step S3-2 includes:
[0085] S3-2-1: The position of the words in the target text description information is determined as the time coordinate, and the value of the column vector in the target input embedding matrix is determined as the signal strength, thus obtaining the semantic signal of the target text description information.
[0086] For example, if the target text description is "I am a robot", the positions of words (t=0, 1, ..., L-1) in a sentence of length L=4 are determined as time coordinates, and the values in the column vector are determined as signal strengths, thus obtaining d semantic signals that change with the position t of the word.
[0087] S3-2-2: Perform discrete transformation processing on the semantic signal of the target text description information to obtain the input signal of the target text description information.
[0088] In this embodiment, the position of the word in the target text description information is determined as the time coordinate, and the value of the column vector in the target input embedding matrix is determined as the signal strength. This allows for the rapid acquisition of semantic signals that change with the position of the word. Discretizing the semantic signals yields the input signal of the target text description information, which helps to quickly identify similar images of the target image by processing the input signal using an encoder.
[0089] For example, the transformation process includes one of the one-dimensional discrete Fourier transform and the discrete cosine transform.
[0090] In one embodiment of this disclosure, step S3-2-2 includes: performing a one-dimensional discrete Fourier transform on the semantic signal of the target text description information to obtain the target frequency domain signal, and determining the target frequency domain signal as the target input signal.
[0091] A signal {x} of length L is applied n The formula for the 1D Discrete Fourier Transform (1DDFT) of} is:
[0092]
[0093] Where j is the imaginary unit, i.e., j 2 =-1.
[0094] Applying 1D-DFT to each column of the input embedding matrix can transform d column vectors p i (Column i) is converted to frequency domain representation P i =(P 0,i ,P 1,i ,...,P L-1,i ) T ,in
[0095]
[0096] For ease of training, this expression can retain only the real part of each term. However, training with complex numbers is also feasible. Accordingly, the training parameters need to be extended to the complex domain, making the actual parameter size twice as large.
[0097] Each column of the matrix after being processed by the Discrete Fourier Transform becomes a frequency domain signal, resulting in the matrix P = (P 0 ,P 1 ,...,P d-1 The input embedding matrix is fed into the encoder, which consists of N layers of the input model. During the training and inference processes, the input embedding matrix is output as a d-dimensional vector s through the model's output mapping layer. i .
[0098] s i =encoder(P i ).
[0099] In this embodiment, by performing a one-dimensional discrete Fourier transform on the semantic signal of the target text description information, the target input signal in the frequency domain can be obtained, thereby realizing the fusion of the semantic information of words in the target text description information and the position information in the sentence, which helps to quickly and accurately determine similar images of the target image.
[0100] In another embodiment of this disclosure, step S3-2-2 includes: performing a discrete cosine transform on the semantic signal of the target text description information to obtain a target frequency domain signal, and determining the target frequency domain signal as the target input signal. Compared with 1DDFT, the transformed parameters are real numbers, and the calculation does not need to be extended to the complex domain. The discrete cosine transform can also convert a time-domain signal into a frequency-domain signal; the difference from the Fourier transform lies in the different underlying signals selected.
[0101] In this embodiment, by performing discrete cosine transform on the semantic signal of the target text description information, the target input signal in frequency domain representation can be obtained, thereby realizing the fusion of semantic information of words in the target text description information and position information in the sentence, which helps to quickly and accurately determine similar images of the target image.
[0102] In one embodiment of this disclosure, step S4 includes:
[0103] S4-1: Calculate the cosine similarity between the target feature vector and multiple feature vectors to obtain multiple cosine similarities between the target feature vector and the multiple feature vectors respectively.
[0104] The target text description information t2 is processed by the encoder after undergoing the one-dimensional discrete Fourier transform described above to obtain a d-dimensional complex vector t:
[0105] t = encoder(DFT(t2)).
[0106] The encoder processes the text description of each image in the image library to create a feature vector s. i Calculate the cosine similarity with t:
[0107]
[0108] S4-2: Sort multiple cosine similarities in descending order, and determine a list of similar images to the target image based on the descending order results.
[0109] In this embodiment, a list of similar images to the target image can be quickly determined from the image library based on the cosine similarity between the target feature vector and multiple feature vectors.
[0110] In one embodiment of this disclosure, prior to step S1, the method further includes:
[0111] S0-1: Use the image processing model to process multiple images and obtain text description information for each image.
[0112] The training set of the image processing model includes not only the identifier, image content (img), and meta information for each image, but also the text description information (prompt) for each image.
[0113] Train={e|e=(id,prompt,img,meta)}
[0114] The prompt adds meta information in text format. meta Meta information includes location and time information. Location and time information can be manually annotated and then sent to the image processing model D.
[0115] p meta =###position:Country,Road,###date:year2000,month1,day1.
[0116] The image processing model D is trained on the training set Train. The model input is the image content (img), and the output is the text description information (p). i :
[0117] p i =D(img).
[0118] By using the image processing model D to process each image in the image library Image, text description information p for the corresponding image can be generated. i .
[0119] S0-2: Encode the text description information of each of the multiple images using an encoder to obtain multiple feature vectors. That is, use an encoder to encode the text description information of each of the multiple images in a manner similar to steps S3-1 to S3-3 to obtain the feature vectors of each image.
[0120] In this embodiment, an image information group is stored in advance for each image in the image library. When the target image is determined to be similar to other images, the text description information and corresponding feature vectors of each image can be extracted from the image information group of each image in the image library, thereby improving the efficiency of obtaining similar images.
[0121] Figure 3 This is a structural block diagram of a similar image acquisition device in one embodiment of this disclosure. Figure 3 As shown, the similar image acquisition device includes:
[0122] The text description information acquisition module 100 is used to process the target image using an image processing model to obtain the text description information of the target image, and to acquire the text description information of multiple images in the image library. The text description information of each image in the multiple images includes the location information and time information of the corresponding image.
[0123] The information adding module 200 is used to add location and time information to the text description information of the target image to obtain the target text description information;
[0124] The feature encoding module 300 is used to encode the target text description information using an encoder to obtain the target feature vector, and to obtain multiple feature vectors corresponding to the text description information of multiple images.
[0125] The similarity image determination module 400 is used to calculate the similarity between the target feature vector and multiple feature vectors, and to determine the similar images of the target image based on the similarity calculation results.
[0126] In one embodiment of this disclosure, the feature encoding module 300 is used to determine a target input embedding matrix based on the target text description information; the feature encoding module 300 is also used to perform transformation processing on the target input embedding matrix to obtain the input signal of the target text description information; the feature encoding module 300 is also used to process the input signal of the target text description information using the N-layer structure of the encoder to obtain the target feature vector, wherein N is an integer greater than 1, and each layer in the N-layer structure includes a multi-head attention sublayer, a first normalization sublayer corresponding to the multi-head attention sublayer, a forward feedback sublayer, and a second normalization sublayer corresponding to the forward feedback sublayer.
[0127] In one embodiment of this disclosure, the feature encoding module 300 is used to determine the position of words in the target text description information as time coordinates and to determine the value of the column vector in the target input embedding matrix as signal intensity, thereby obtaining the semantic signal of the target text description information; the feature encoding module 300 is also used to perform discrete transformation processing on the semantic signal of the target text description information to obtain the input signal of the target text description information.
[0128] In one embodiment of this disclosure, the feature encoding module 300 is used to perform a one-dimensional discrete Fourier transform on the semantic signal of the target text description information to obtain the target frequency domain signal, and to determine the target frequency domain signal as the target input signal.
[0129] In one embodiment of this disclosure, the feature encoding module 300 is used to perform discrete cosine transform on the semantic signal of the target text description information to obtain the target frequency domain signal, and to determine the target time domain signal as the target input signal.
[0130] In one embodiment of this disclosure, the similar image determination module 400 is used to calculate the cosine similarity between the target feature vector and multiple feature vectors to obtain multiple cosine similarities between the target feature vector and the multiple feature vectors respectively; the similar image determination module 400 is also used to sort the multiple cosine similarities in descending order and determine a list of similar images of the target image based on the descending order result.
[0131] In one embodiment of this disclosure, the similar image acquisition device further includes:
[0132] The preprocessing module is used to process multiple images using an image-text processing model to obtain text description information for each image. The encoder is then used to encode the text description information for each image to obtain multiple feature vectors.
[0133] It should be noted that the specific implementation of the similar image acquisition device in this disclosure is similar to the specific implementation of the similar image acquisition method in this disclosure. For details, please refer to the description of the similar image acquisition method section. In order to reduce redundancy, it will not be described again.
[0134] In addition, this disclosure also provides an electronic device, including:
[0135] Memory, used to store computer programs;
[0136] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the similar image acquisition method described in any of the above embodiments of the present disclosure.
[0137] Below, for reference Figure 4 To describe an electronic device according to embodiments of this disclosure. For example... Figure 4 As shown, the electronic device includes one or more processors and memory.
[0138] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.
[0139] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the similar image acquisition methods of the various embodiments of this disclosure described above and / or other desired functions.
[0140] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).
[0141] In addition, the input device may also include, for example, a keyboard, a mouse, etc.
[0142] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0143] Of course, for the sake of simplicity, Figure 4 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.
[0144] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the similar image acquisition methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.
[0145] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this disclosure. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0146] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the similar image acquisition method according to various embodiments of this disclosure as described in the foregoing portion of this specification.
[0147] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0148] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0150] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0151] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.
[0152] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.
[0153] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0154] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
Claims
1. A method for obtaining similar images, characterized in that, include: The target image is processed using an image processing model to obtain the text description information of the target image, and the text description information of multiple images in the image library is obtained. The text description information of each image in the multiple images includes the location information and time information of the corresponding image. Location and time information are added to the text description information of the target image to obtain the target text description information; The target text description information is encoded using an encoder to obtain a target feature vector, and multiple feature vectors corresponding to the text description information of each of the multiple images are obtained. Calculate the similarity between the target feature vector and the plurality of feature vectors, and determine the similar images of the target image based on the similarity calculation results; The process of encoding the target text description information using an encoder to obtain the target feature vector includes: Based on the target text description information, determine the target input embedding matrix; The target input embedding matrix is transformed to obtain the input signal of the target text description information; The input signal of the target text description information is processed using the N-layer structure of the encoder to obtain the target feature vector, where N is an integer greater than 1. Each layer of the N-layer structure includes a multi-head attention sublayer, a first normalization sublayer corresponding to the multi-head attention sublayer, a forward feedback sublayer, and a second normalization sublayer corresponding to the forward feedback sublayer. The step of transforming the target input embedding matrix to obtain the input signal of the target text description information includes: The position of the words in the target text description information is determined as the time coordinate, and the value of the column vector in the target input embedding matrix is determined as the signal strength, thereby obtaining the semantic signal of the target text description information; The semantic signal of the target text description information is subjected to discrete transformation processing to obtain the input signal of the target text description information.
2. The method according to claim 1, characterized in that, The step of performing discrete transformation processing on the semantic signal of the target text description information to obtain the input signal of the target text description information includes: A one-dimensional discrete Fourier transform is performed on the semantic signal of the target text description information to obtain the target frequency domain signal, and the target frequency domain signal is determined as the target input signal.
3. The method according to claim 1, characterized in that, The step of performing discrete transformation processing on the semantic signal of the target text description information to obtain the input signal of the target text description information includes: The semantic signal of the target text description information is subjected to discrete cosine transform to obtain the target frequency domain signal, and the target frequency domain signal is determined as the target input signal.
4. The method according to any one of claims 1-3, characterized in that, The step of calculating the similarity between the target feature vector and the plurality of feature vectors, and determining similar images of the target image based on the similarity calculation results, includes: Cosine similarity calculation is performed on the target feature vector and the plurality of feature vectors to obtain multiple cosine similarities between the target feature vector and the plurality of feature vectors respectively; The cosine similarities are sorted in descending order, and a list of similar images to the target image is determined based on the descending order result.
5. The method according to claim 4, characterized in that, Before obtaining the text description information of each of the multiple images in the image library, the method further includes: The image processing model is used to process the multiple images to obtain text description information for each of the multiple images; The encoder is used to encode the text description information of each of the multiple images to obtain the multiple feature vectors.
6. A similar image acquisition device, characterized in that, include: The text description information acquisition module is used to process the target image using an image processing model to obtain the text description information of the target image, and to acquire the text description information of multiple images in the image library. The text description information of each image in the multiple images includes the location information and time information of the corresponding image. The information adding module is used to add location information and time information to the text description information of the target image to obtain the target text description information; The feature encoding module is used to encode the target text description information using an encoder to obtain a target feature vector, and to obtain multiple feature vectors corresponding to the text description information of each of the multiple images. A similar image determination module is used to calculate the similarity between the target feature vector and the plurality of feature vectors, and determine the similar images of the target image based on the similarity calculation results; The feature encoding module is used to determine the target input embedding matrix based on the target text description information; the feature encoding module is also used to transform the target input embedding matrix to obtain the input signal of the target text description information; the feature encoding module is also used to process the input signal of the target text description information using the N-layer structure of the encoder to obtain the target feature vector, where N is an integer greater than 1, and each layer of the N-layer structure includes a multi-head attention sublayer, a first normalization sublayer corresponding to the multi-head attention sublayer, a forward feedback sublayer, and a second normalization sublayer corresponding to the forward feedback sublayer; The feature encoding module is used to determine the position of the words in the target text description information as time coordinates and to determine the value of the column vector in the target input embedding matrix as signal strength, thereby obtaining the semantic signal of the target text description information; the feature encoding module is also used to perform discrete transformation processing on the semantic signal of the target text description information to obtain the input signal of the target text description information.
7. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, wherein, when the computer program product is executed, it implements the method described in any one of claims 1-5.
8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-5.