A picture search processing method and device, a storage medium and an electronic device

Abstract

Embodiments of the present specification disclose a picture search processing method and device, a storage medium and an electronic device, wherein the method comprises: obtaining picture search text input by a user, determining a search text vector corresponding to the picture search text, performing similar text index matching on the search text vector to obtain a similar text index data set, performing picture search processing based on the search text vector and the similar text index data set to obtain at least one reference search picture, and displaying the reference search picture on a search display interface.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to an image search processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the explosive growth of image resources, text-based image retrieval (Text Search) has become an important requirement in the field of information retrieval. However, existing Text Search methods mainly rely on keyword matching or simple text tag association. In practical applications, it is difficult to accurately capture the semantic depth and contextual information of user input text. At the same time, the complexity of image data itself also poses new challenges to retrieval systems. Summary of the Invention

[0003] This specification provides an image search processing method, apparatus, storage medium, and electronic device, the technical solutions of which are as follows:

[0004] Firstly, embodiments of this specification provide an image search processing method, the method comprising:

[0005] Obtain the image search text input by the user, and determine the search text vector corresponding to the image search text;

[0006] The search text vector is subjected to similarity index matching to obtain a similar text index data set;

[0007] Based on the search text vector and the similar text index data set, image search processing is performed to obtain at least one reference search image, which is then displayed on the search display interface.

[0008] In one feasible implementation, the step of performing similarity matching on the search text vector to obtain a similar text index data set includes:

[0009] Obtain multiple text index data from a text index search library, wherein the text search library includes text index data corresponding to multiple source images, and the text index data includes index data between the image description text vector of the source image and the source image vector;

[0010] Using the image description text vector in the text index data as a reference, calculate the text index matching degree between the search text vector and the image description text vector, and determine a first number of similar image description text vectors based on the text index matching degree;

[0011] Obtain the similar text index data corresponding to each of the similar image description text vectors, and generate a similar text index data set based on the similar text index data.

[0012] In one feasible implementation, the method further includes:

[0013] Obtain multiple source images and their corresponding image text information;

[0014] Determine the material image vector corresponding to the material image based on the material image and the image text information;

[0015] Determine the image description text vector corresponding to the material image based on the image text information;

[0016] Establish text index data between the material image vector corresponding to the material image and the image description text vector;

[0017] A text index search library is generated based on the text index data corresponding to all the aforementioned source images.

[0018] In one feasible implementation, determining the material image vector corresponding to the material image based on the material image and the image text information, and determining the image description text vector corresponding to the material image based on the image text information, includes:

[0019] Determine the binary image feature information corresponding to the source image;

[0020] Based on the binary image feature information and the image text information, a vector conversion process is performed to obtain the material image vector;

[0021] Vector segmentation is performed on the image text information to obtain the image description text vector corresponding to the material image.

[0022] In one feasible implementation, the step of performing image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image includes:

[0023] Based on the similar text index data set, at least one candidate search image and a corresponding candidate material image vector are determined.

[0024] Calculate the candidate image similarity between the search text vector and each candidate material image vector;

[0025] At least one reference search image is selected from the candidate search images based on the similarity of the candidate images.

[0026] In one feasible implementation, selecting at least one reference search image from the candidate search images based on the candidate image similarity includes: selecting at least one reference search image from the candidate search images whose similarity to the candidate images is greater than a similarity threshold.

[0027] In one feasible implementation, the method further includes:

[0028] Determine the number of target images in the reference search images;

[0029] If the number of target images is less than the number of images threshold, then the reference material image vector of the reference search image is obtained;

[0030] Based on the reference image vector, similar image vector matching is performed on the image vector library to obtain at least one supplementary image, which is then used as the reference search image.

[0031] In one feasible implementation, the step of performing image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image includes:

[0032] Based on the similar text index data set, at least one candidate search image and a corresponding candidate material image vector are determined.

[0033] A reference search image vector is obtained by extracting the search image vector based on the candidate image vectors described above.

[0034] A comprehensive search vector is determined based on the reference search image vector and the search text vector;

[0035] Calculate the candidate image similarity between the comprehensive search vector and each candidate material image vector, and select at least one reference search image from the candidate search images based on the candidate image similarity; and / or, perform search matching processing on the material image vector library based on the comprehensive search vector to obtain at least one reference search image.

[0036] In one feasible implementation, determining the comprehensive search vector based on the reference search image vector and the search text vector includes:

[0037] The reference search image vector and the search text vector are weighted and combined to obtain the comprehensive search vector.

[0038] Secondly, embodiments of this specification provide an image search processing apparatus, the apparatus comprising:

[0039] The text processing module is used to obtain the image search text input by the user and determine the search text vector corresponding to the image search text;

[0040] The index matching module is used to perform similarity index matching on the search text vector to obtain a similar text index data set;

[0041] The image search module is used to perform image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image, and to display the reference search image on the search display interface.

[0042] In one feasible implementation, the index matching module is configured to:

[0043] Obtain multiple text index data from a text index search library, wherein the text search library includes text index data corresponding to multiple source images, and the text index data includes index data between the image description text vector of the source image and the source image vector;

[0044] Using the image description text vector in the text index data as a reference, calculate the text index matching degree between the search text vector and the image description text vector, and determine a first number of similar image description text vectors based on the text index matching degree;

[0045] Obtain the similar text index data corresponding to each of the similar image description text vectors, and generate a similar text index data set based on the similar text index data.

[0046] In one feasible implementation, the device is further used for:

[0047] Obtain multiple source images and their corresponding image text information;

[0048] Determine the material image vector corresponding to the material image based on the material image and the image text information;

[0049] Determine the image description text vector corresponding to the material image based on the image text information;

[0050] Establish text index data between the material image vector corresponding to the material image and the image description text vector;

[0051] A text index search library is generated based on the text index data corresponding to all the aforementioned source images.

[0052] In one feasible implementation, the device is further used for:

[0053] Determine the binary image feature information corresponding to the source image;

[0054] Based on the binary image feature information and the image text information, a vector conversion process is performed to obtain the material image vector;

[0055] Vector segmentation is performed on the image text information to obtain the image description text vector corresponding to the material image.

[0056] In one feasible implementation, the image search module is used for:

[0057] Based on the similar text index data set, at least one candidate search image and a corresponding candidate material image vector are determined.

[0058] Calculate the candidate image similarity between the search text vector and each candidate material image vector;

[0059] At least one reference search image is selected from the candidate search images based on the similarity of the candidate images.

[0060] In one feasible implementation, the image search module is configured to: select at least one reference search image from the candidate search images whose similarity to the candidate images is greater than a similarity threshold.

[0061] In one feasible implementation, the device is further used for:

[0062] Determine the number of target images in the reference search images;

[0063] If the number of target images is less than the number of images threshold, then the reference material image vector of the reference search image is obtained;

[0064] Based on the reference image vector, similar image vector matching is performed on the image vector library to obtain at least one supplementary image, which is then used as the reference search image.

[0065] In one feasible implementation, the image search module is used for:

[0066] Based on the similar text index data set, at least one candidate search image and a corresponding candidate material image vector are determined.

[0067] A reference search image vector is obtained by extracting the search image vector based on the candidate image vectors described above.

[0068] A comprehensive search vector is determined based on the reference search image vector and the search text vector;

[0069] Calculate the candidate image similarity between the comprehensive search vector and each candidate material image vector, and select at least one reference search image from the candidate search images based on the candidate image similarity; and / or, perform search matching processing on the material image vector library based on the comprehensive search vector to obtain at least one reference search image.

[0070] In one feasible implementation, the base image search module is used for:

[0071] The reference search image vector and the search text vector are weighted and combined to obtain the comprehensive search vector.

[0072] Thirdly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0073] Fourthly, embodiments of this specification provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0074] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0075] In one or more embodiments of this specification, the electronic device determines the search text vector corresponding to the image search text input by the user, performs similarity index matching on the search text vector to obtain a similar text index data set, performs image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image, and displays the reference search image on the search display interface. Through layer-by-layer refined processing, the user-input search text is transformed into a high-dimensional search text vector and similarity index matching is performed to quickly locate relevant similar text index data sets. Based on the search text vector combined with the similar text index data set, a deeper comprehensive search is performed to accurately filter reference search images highly relevant to the user's needs, which can significantly improve the semantic relevance and retrieval accuracy of search results, while also taking into account search efficiency, providing users with a more intelligent and intuitive text-to-image search experience. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is a schematic diagram of a scene of an image search and processing system provided in the embodiments of this specification;

[0078] Figure 2 This is a flowchart illustrating an image search processing method provided in the embodiments of this specification;

[0079] Figure 3 This is a schematic diagram of a similarity matching process provided in the embodiments of this specification;

[0080] Figure 4 This is a schematic diagram illustrating a process for updating a text index search library, as provided in the embodiments of this specification.

[0081] Figure 5 This is a schematic diagram of an image search processing procedure provided in the embodiments of this specification;

[0082] Figure 6 This is a schematic diagram of the structure of an image search processing device provided in the embodiments of this specification;

[0083] Figure 7 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification;

[0084] Figure 8 This is a schematic diagram of the operating system and user space structure provided in the embodiments of this specification;

[0085] Figure 9 yes Figure 8 Architecture diagram of the Android operating system in China;

[0086] Figure 10 yes Figure 8 Architecture diagram of the iOS operating system. Detailed Implementation

[0087] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0088] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0089] In related technologies, traditional text-to-image (TTO) methods mainly rely on keyword matching or simple text tag association, which struggles to accurately capture the semantic depth and contextual information of user-input text. Meanwhile, the inherent complexity of image data itself presents new challenges to retrieval systems: on the one hand, the high-dimensional attributes of image features require efficient indexing calculations; on the other hand, user-input text is often highly abstract or vague descriptions. How to map these descriptions to the image semantic space and achieve accurate matching has become the core bottleneck of current TTO methods. Solving these problems requires a retrieval mechanism that can both understand text semantics and efficiently process image features to meet users' needs for fast and accurate retrieval.

[0090] Please see Figure 1 This is a schematic diagram of a scene for an image search and processing system provided in an embodiment of this application. Figure 1 As shown, an image search and processing system may include at least a client cluster and a service platform 100.

[0091] In some embodiments, the client cluster may include at least one client, such as Figure 1 As shown, it specifically includes client 1 corresponding to user 1, client 2 corresponding to user 2, ..., client n corresponding to user n, where n is an integer greater than 0.

[0092] Each client in a client cluster can be an electronic device with communication capabilities, including but not limited to: wearable devices, handheld devices, personal computers, tablets, in-vehicle devices, smartphones, computing devices, or other processing devices connected to a wireless modem. Electronic devices may have different names in different networks, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), and electronic devices in 5G networks or future evolved networks.

[0093] In some embodiments, the service platform 100 may be a single server device, such as a rack-mounted, blade, tower, or cabinet-type server device, or a workstation, mainframe, or other hardware device with strong computing power; or it may be a server cluster composed of multiple servers, wherein the servers in the service cluster may be composed in a symmetrical manner, wherein each server is functionally and hierarchically equivalent in the transaction link, and each server can provide services to the outside world independently, wherein providing services independently can be understood as not requiring the assistance of other servers.

[0094] In one or more embodiments of this specification, the service platform 100 may establish a communication connection with at least one client in the client cluster, and complete the data interaction during the image search processing based on the communication connection.

[0095] For example, service platform 100 provides search services to external parties. Users can send image search text to service platform 100 through a client. Service platform 100 can determine the search text vector corresponding to the image search text, perform similarity index matching on the search text vector to obtain a similar text index data set, perform image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image, and service platform 100 instructs the client to display the reference search image on the search display interface.

[0096] It should be noted that the service platform 100 establishes a communication connection with at least one client in the client cluster via a network for interactive communication. This network can be a wireless network or a wired network. Wireless networks include, but are not limited to, cellular networks, wireless LANs, infrared networks, or Bluetooth networks. Wired networks include, but are not limited to, Ethernet, universal serial bus (USB), or controller area networks. In one or more embodiments of the specification, technologies and / or formats including HyperText Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network (such as target compressed packets). Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0097] The image search processing system embodiments provided in this specification and the image search processing methods described in one or more embodiments belong to the same concept. The execution entity corresponding to the image search processing methods involved in one or more embodiments of this specification can be an electronic device, which can be the aforementioned service platform 100 or the aforementioned client, depending on the actual application environment. The specific implementation process of the demonstration file generation system embodiment can be found in the following method embodiments, and will not be repeated here.

[0098] The present specification will now be described in detail with reference to specific embodiments.

[0099] In one embodiment, such as Figure 2 As shown, an image search and processing method is proposed. This method can be implemented using a computer program and can run on an image search and processing device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone utility application. The image search and processing device can be an electronic device.

[0100] Specifically, the image search processing method includes:

[0101] S102: Obtain the image search text input by the user, and determine the search text vector corresponding to the image search text;

[0102] User-inputted image search text: refers to the keywords or descriptive text that users enter into the search box of a search engine or electronic device to search for images.

[0103] Search text vectors: Transform user-input natural language text into a high-dimensional vector representation using machine learning models (such as the BERT model) to capture the semantic information of the text.

[0104] For example, a user enters search keywords via keyboard, such as "high-resolution images of blue skies." The electronic device receives this image search text, which can include various expressions, such as keywords ("landscape") and natural language descriptions ("mountains at sunset"). The image search text is then preprocessed (e.g., word segmentation, stop word removal) to obtain preprocessed image search text, ensuring the input data is formatted for subsequent vector processing. A pre-trained natural language processing model (e.g., BERT, RoBERTa) is used to convert the text into a high-dimensional vector, generating the search text vector corresponding to the image search text. The natural language processing model, through training, understands natural language semantics and transforms the input image search text into a semantic vector (e.g., 512-dimensional).

[0105] S104: Perform similarity index matching on the search text vector to obtain a similar text index data set;

[0106] Similarity index matching: By calculating the similarity between the user's input search text vector and the corresponding image description text vectors of pre-stored material images in the text index search library, the most semantically relevant index data is found.

[0107] Similar Text Index Data Set: During the matching process, a set of similar text index data that is semantically related to the user's input text is found. The text index data consists of the index data between the image description text vector of the source image and the source image vector.

[0108] Optionally, the text index search library includes text index data corresponding to multiple source images;

[0109] In one optional implementation, a text index search library can be pre-built. By calculating the similarity between the user-input text vector and the image description text vectors in the text index search library, the set of semantically closest similar text index data can be found. For example, a similarity threshold can be set, or the top-N similar text index data can be selected as the similar text index data set. Through the similarity index, the search scope can be quickly narrowed, providing a high-quality text candidate set for subsequent image searches.

[0110] S106: Perform image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image, and display the reference search image on the search display interface.

[0111] Reference search images: Images from the database that are related to the search text, based on the user's search intent.

[0112] Search display interface: The front-end interface used to display search results to users.

[0113] In one alternative implementation, each text index record in the similar text index data set is used to associate a corresponding image from the image library as a reference search image. Each record in the text index data has been pre-associated with one or more images, thus obtaining at least one reference search image, which is then displayed to the user on the search display interface. The user can save or adjust the reference search image.

[0114] In one feasible implementation, performing the image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image can be done in the following manner:

[0115] Step S2: Based on the similar text index data set, determine at least one candidate search image and the corresponding candidate material image vector;

[0116] Similar Text Index Data Set: A set of semantically related text index data obtained by calculating the similarity of search text vectors.

[0117] Candidate search images: Images associated with each text index in the similar text index dataset.

[0118] Candidate image vector: The visual feature vector of the candidate image, which is a description of the visual semantics of the image obtained in advance by a machine learning model.

[0119] In a schematic way, the material image associated with each piece of similar text index data is determined from the similar text index data set and used as a candidate search image. At the same time, the similar text index data is the index data between the image description text vector and the material image vector for the candidate search image. Based on this, the candidate material image vector corresponding to the candidate search image can be determined from the similar text index data.

[0120] Step S4: Extract the reference search image vector based on the candidate image vectors;

[0121] Search image vector extraction: A comprehensive representative vector is calculated from the candidate image vectors to assist the text retrieval image. A reference search image vector is extracted from the dimension of similar images as the basis for subsequent retrieval.

[0122] Reference search image vector: can be understood as a global feature representing the candidate image set, reflecting the overall semantic and visual features.

[0123] Optionally, the mean vector of all candidate image vectors can be calculated as the reference search image vector;

[0124] Optionally, a reference search image vector can be obtained by weighting all candidate image vectors based on text similarity.

[0125] Step S6: Determine the comprehensive search vector based on the reference search image vector and the search text vector;

[0126] Comprehensive search vector: A joint vector generated by combining the input text vector and the reference search image vector, used for similarity calculation and image matching in subsequent steps.

[0127] This illustrates how the search text vector is fused with the reference search image vector to generate a composite search vector.

[0128] Optionally, determining the comprehensive search vector based on the reference search image vector and the search text vector includes:

[0129] The reference search image vector and the search text vector are weighted and combined to obtain the comprehensive search vector.

[0130] As an illustration, the weighted summation can be performed using the following formula to obtain the comprehensive search vector, as follows:

[0131] F 综合 =α*F 文本 +β*F 参考

[0132] Among them, F 综合 The formula represents the comprehensive search vector, F. 文本 The F represents the search text vector. 参考 This represents the reference search image vector, where α and β are the vector weights used to balance text and image features.

[0133] Optionally, the search text vector and the reference search image vector can be concatenated to obtain a comprehensive search vector.

[0134] Step S8: Calculate the candidate image similarity between the comprehensive search vector and each candidate material image vector, and select at least one reference search image from the candidate search images based on the candidate image similarity; and / or, perform search matching processing on the material image vector library based on the comprehensive search vector to obtain at least one reference search image.

[0135] In one feasible implementation, the similarity between the comprehensive search vector and each candidate material image vector is calculated, and at least one reference search image is selected from the candidate search images based on the candidate image similarity.

[0136] In one feasible implementation, the material image vector library can be searched and matched based on the comprehensive search vector to obtain at least one reference search image;

[0137] In one feasible implementation, the similarity between the comprehensive search vector and each candidate material image vector is calculated, and at least one reference search image is selected from the candidate search images based on the candidate image similarity. Simultaneously, the following can be performed: a search and matching process is applied to the material image vector library based on the comprehensive search vector to obtain at least one reference search image.

[0138] For example, cosine similarity can be used to calculate the similarity between the overall search vector and each candidate image vector. Based on the similarity ranking, a preset number of images with the highest similarity are selected as reference search images.

[0139] For example, if the number of candidate images is insufficient or the results are not relevant enough, the entire material image vector library can be searched based on the comprehensive search vector. That is, the material image vector library can be searched and matched based on the comprehensive search vector to obtain at least one reference search image.

[0140] Through the above steps, the feature information of text and images can be effectively combined to filter out the images that best match the user's needs from a large-scale image library; the search scope can be narrowed layer by layer from text matching to candidate images and then to the final result; and the integrated processing of text and image features enables users to perform a fusion search from both semantic and visual dimensions by simply inputting text, thereby improving the accuracy and relevance of the retrieval.

[0141] In one or more embodiments of this specification, the electronic device determines the search text vector corresponding to the image search text input by the user, performs similarity index matching on the search text vector to obtain a similar text index data set, performs image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image, and displays the reference search image on the search display interface. Through layer-by-layer refined processing, the user-input search text is transformed into a high-dimensional search text vector and similarity index matching is performed to quickly locate relevant similar text index data sets. Based on the search text vector combined with the similar text index data set, a deeper comprehensive search is performed to accurately filter reference search images highly relevant to the user's needs, which can significantly improve the semantic relevance and retrieval accuracy of search results, while also taking into account search efficiency, providing users with a more intelligent and intuitive text-to-image search experience.

[0142] Optional, please see Figure 3 , Figure 3 This is a flowchart illustrating a similarity matching process proposed in this specification. Specifically, the process of performing similarity matching on the search text vector to obtain a similar text index data set can be described as follows:

[0143] S202: Obtain multiple text index data from the text index search library, wherein the text search library includes text index data corresponding to multiple material images, and the text index data includes index data between the image description text vector of the material image and the material image vector;

[0144] Text Index Search Library: An index database that stores information related to source images, including descriptive text vectors for each source image, image feature vectors, and the associated index data between the two, to support efficient semantic search.

[0145] Text index data includes, but is not limited to:

[0146] Image description text vector: A semantic vector generated from the text description of an image (such as title, label, annotation) using a natural language processing model (such as BERT).

[0147] Image vector: A high-dimensional feature vector extracted by deep learning visual models (such as ResNet and EfficientNet) to describe the visual content of an image.

[0148] Indexed data: It establishes a link between image description text vectors and source image vectors, supporting cross-modal retrieval from text descriptions to image content.

[0149] To illustrate, the index data is loaded to extract complete index data for each image from the text index search library. Each index record contains a semantic representation of the image description text, a feature representation of the image itself, and a mapping relationship between the two.

[0150] S204: Using the image description text vector in the text index data as a reference, calculate the text index matching degree between the search text vector and the image description text vector, and determine a first number of similar image description text vectors based on the text index matching degree;

[0151] Text index matching score (similarity): A matching score obtained by comparing the semantic similarity between the user-input search text vector and the image description text vector in the text index. Common similarity measurement methods include cosine similarity.

[0152] First quantity: The upper limit of the number of matching results, for example, returning the top-N most semantically relevant records.

[0153] To illustrate, the process iterates through each text index data in the text index search library, calculates the similarity between the user input text vector and the image description text vector as the text index matching degree, sets a matching degree threshold, sorts the image description text vectors in descending order of similarity, and selects the first number of similar image description text vectors whose text index matching degree is greater than the matching threshold.

[0154] By calculating text similarity, the descriptive text vectors that are most semantically similar to the input text are accurately selected, providing candidate data for the next step of image retrieval.

[0155] S206: Obtain the similar text index data corresponding to each of the similar image description text vectors, and generate a similar text index data set based on the similar text index data.

[0156] Similar text index data: The complete index data associated with the filtered similar image description text vectors, including the image description text vectors, the corresponding source image vectors, and other metadata.

[0157] Similar Text Index Data Set: This set aggregates the index data of all matching results for use in subsequent search processes.

[0158] This example illustrates how, based on the selected similar image description text vectors, corresponding index data is extracted from a text index search library. Each piece of index data contains: an image description text vector and the corresponding source image vector.

[0159] Then, an index data set is generated based on the similar text index data for image matching and comprehensive search in subsequent steps.

[0160] By generating a set of similar text indexes that are semantically related to the search text, an accurate initial candidate range is provided for subsequent image retrieval based on visual features.

[0161] In the embodiments described in this specification, the above method can be used to progressively filter out the descriptive text most relevant to the user's input text and its associated image index data from the text index search library. The entire process narrows the search scope layer by layer, ensuring the semantic relevance of the results, while providing a high-quality candidate data set for subsequent image matching and display.

[0162] Optional, please see Figure 4 , Figure 4 This is a flowchart illustrating a text index search library update process as presented in this specification. For details, please refer to the following:

[0163] S3002: Obtain multiple source images and the corresponding image text information for the source images;

[0164] Image resources: These can be understood as image resources that support text search and image search engines, and can include e-commerce product images, natural landscape images, design material images, etc.

[0165] Image text information: Textual descriptions related to the source image, typically including one or more of the following: title, tags, annotations, user-provided descriptive text, etc.

[0166] Optionally, the source image and its corresponding text information can come from multiple sources, such as: user-uploaded images and their corresponding text descriptions, existing images in the database and their metadata, and automatically generated image descriptions (generated through an image annotation model).

[0167] To illustrate, multiple source images and their corresponding image text information are acquired, and the data is formatted to ensure that the source images and their corresponding text information correspond one-to-one and are stored in a formatted manner.

[0168] S3004: Determine the material image vector corresponding to the material image based on the material image and the image text information, and determine the image description text vector corresponding to the material image based on the image text information;

[0169] Source image vector: A high-dimensional feature vector extracted from source images using a machine learning model, used to describe the visual semantic features of the image.

[0170] Image description text vectors: These are high-dimensional semantic vectors that transform the text description of an image into a text description using natural language processing models, and are used to describe the semantic features of the text.

[0171] To illustrate, visual models (such as ResNet and EfficientNet) are used to process source images, extracting high-dimensional feature representations (usually 512 dimensions or higher) to obtain source image vectors.

[0172] This illustrates how natural language processing models (such as BERT and RoBERTa) are used to process the text descriptions of source images, generating image description text vectors.

[0173] By extracting semantic information from images and text, a basic vector representation is provided for subsequent index building.

[0174] In one feasible implementation, the following approach can be referenced:

[0175] Step A2: Determine the binary image feature information corresponding to the source image;

[0176] Binary image feature information: The underlying data representation of the source image, usually stored in the form of a pixel matrix. The binary data of the file can be read directly. The binary feature information of the source image includes low-level features such as color, texture, and shape. This information is used for subsequent feature extraction and vectorization processing.

[0177] Optionally, the binary image feature information can be grayscale information (such as single-channel data) that converts the source image into a binary image feature representation. Each pixel value is typically between 0 and 255, reflecting the brightness distribution of the image.

[0178] Using black-and-white features as binary graph feature information has the following advantages:

[0179] 1) By removing the color channels, the binary image feature information reduces the data dimension (from RGB 3 channels to a single channel), which facilitates a large amount of calculation and processing in text search images.

[0180] 2) Binary image feature information preserves the structural information of the image (such as contours, boundaries, textures, etc.), making it suitable for basic vision tasks. It achieves faster search calculations and saves search computation while preserving core features.

[0181] 3) Compared to color image processing, black and white image processing is faster, especially in the feature extraction and search stages.

[0182] Step A4: Perform vector conversion processing based on the binary image feature information and the image text information to obtain the material image vector;

[0183] Vector transformation processing: Using deep learning models to transform the binary feature information of an image into a high-dimensional semantic vector (e.g., 512-dimensional), which is used to describe the visual features of the image.

[0184] Source image vectors: Visual feature vectors extracted from source images, capturing the high-level semantics and visual patterns of the images.

[0185] Optionally, the binary image feature information can be input into a pre-trained deep learning model (such as ResNet or EfficientNet), and high-dimensional features of the image can be extracted through the model's convolutional layers. The semantic feature vectors of the source image can then be extracted using the deep learning model, ensuring efficient and accurate representation of the image's visual content.

[0186] Step A6: Perform vector segmentation on the image text information to obtain the image description text vector corresponding to the material image.

[0187] Vector word segmentation: This can be understood as decomposing the text information in an image into basic semantic units (such as words or phrases) and converting them into semantic vector representations.

[0188] Image description text vector: This can be understood as a high-dimensional text vector (e.g., 512-dimensional) generated by word segmentation and semantic embedding models (e.g., BERT) to represent the semantic description of an image.

[0189] This illustration demonstrates how text segmentation or phrase extraction is performed according to language rules to obtain segmented word results. Then, a semantic embedding model (such as BERT or Word2Vec) is used to transform the segmented results into high-dimensional vectors. These vectors accurately reflect the semantic information of the text, providing semantic support for subsequent indexing and retrieval.

[0190] The above method achieves efficient vectorization of source images and their text information, generating source image vectors and image description text vectors, and ensuring that the two have a high semantic correlation. In addition, defining binary image feature information as image features is an effective representation method that can simplify computational complexity while retaining core visual feature information.

[0191] S3006: Establish text index data between the material image vector corresponding to the material image and the image description text vector;

[0192] Text index data: A record used to establish a relationship between the visual vector of a source image and its corresponding image description text vector.

[0193] For example, a mapping is established between the image vector generated from the same source image and the image description text vector to obtain text index data. Each piece of text index data contains at least the following:

[0194] Image identifiers (such as filenames or IDs), source image vectors, and image description text vectors.

[0195] A bidirectional mapping between image features and text descriptions is established based on text index data, laying the foundation for supporting efficient text-to-image retrieval.

[0196] S3008: Store the text index data corresponding to the material image into the text index search library.

[0197] Text Index Search Library: A database or index engine used to store all text index data, supporting efficient cross-modal retrieval between text and images.

[0198] As an example, each generated text index data is written to the index database according to a specified format, and index configuration optimization is performed during storage:

[0199] 1) If an index already exists: check if the index records need to be updated (e.g., duplicate images, modified descriptions), delete the old index data, and insert the new data.

[0200] 2) If it's a new image, add it directly.

[0201] In one or more embodiments of this specification, through the process of S3002-S3008, the electronic device can efficiently complete the updating and maintenance of the text index search library. By establishing a link between the visual features of images and the semantic features of text and storing them as unified index data, integrated data management is achieved. Through vectorized storage and index creation, efficient underlying support is provided for cross-modal retrieval from text to images, enabling rapid retrieval support. Adding, updating, and deleting index data is supported, ensuring the real-time performance and accuracy of the search library.

[0202] Optional, please see Figure 5 , Figure 5 This is a schematic diagram of an image search processing method proposed in this specification. To obtain at least one reference search image by performing image search processing based on the search text vector and the similar text index data set, the following method can be used:

[0203] S4002: Based on the similar text index data set, determine at least one candidate search image and the candidate material image vector corresponding to the candidate search image;

[0204] According to some embodiments:

[0205] Similar Text Index Data Set: A set of semantically related text index data obtained by calculating the similarity of search text vectors.

[0206] Candidate search images: Images associated with each text index in the similar text index dataset.

[0207] Candidate image vector: The visual feature vector of the candidate image, which is a description of the visual semantics of the image obtained in advance by a machine learning model.

[0208] In a schematic way, the material image associated with each piece of similar text index data is determined from the similar text index data set and used as a candidate search image. At the same time, the similar text index data is the index data between the image description text vector and the material image vector for the candidate search image. Based on this, the candidate material image vector corresponding to the candidate search image can be determined from the similar text index data.

[0209] S4004: Calculate the candidate image similarity between the search text vector and each candidate material image vector;

[0210] For example, cosine similarity can be used as a metric to calculate the similarity between the search text vector and each candidate image vector. This is achieved by iterating through the candidate image set and calculating the cosine similarity between the search text vector and each candidate image vector. This yields the candidate image similarity between the search text vector and each candidate image vector.

[0211] S4006: Select at least one reference search image from the candidate search images based on the similarity of the candidate images.

[0212] Reference search images: One or more images selected from the candidate search images that are most semantically relevant to the search text are returned to the user as the final search result.

[0213] For example, candidate images are sorted from highest to lowest similarity, with the most relevant images appearing first. If the similarity of candidate images is below a set threshold (e.g., 0.5), images with poor relevance can be filtered out. Filtering is performed according to a preset number of images, selecting at least one reference search image from the candidate search images. This approach can progressively narrow down the candidate pool and filter out the most relevant images, offering accuracy, efficiency, and scalability.

[0214] In one feasible implementation, selecting at least one reference search image from the candidate search images based on the candidate image similarity can be done in the following way:

[0215] At least one reference search image is selected from the candidate search images whose similarity to the candidate images is greater than a similarity threshold.

[0216] Similarity threshold: The minimum required similarity score. Only candidate images with a similarity score greater than this threshold will be selected as reference search images.

[0217] This is illustrative, and the similarity threshold is either predefined by the system or dynamically adjusted according to the business scenario. For example, for precise retrieval scenarios, the threshold is set to a higher value (e.g., 0.8); for fuzzy retrieval scenarios, the threshold is set to a lower value (e.g., 0.5).

[0218] Furthermore, all candidate search images are traversed, and the similarity between each image and the candidate images is checked to see if the similarity is greater than the similarity threshold. If the similarity is greater than the threshold, the image can be selected as the reference search image after subsequent sorting.

[0219] By setting a similarity threshold, images with low semantic relevance to the search text can be filtered out, ensuring the accuracy of the results. Furthermore, the similarity threshold can be flexibly adjusted according to specific scenarios to balance the quality and coverage of search results. Additionally, images with high semantic relevance can be returned, improving the credibility of search results and user satisfaction.

[0220] Optionally, after selecting at least one reference search image from the candidate search images based on the candidate image similarity, the following method can also be used:

[0221] Step C2: Determine the number of target images for the reference search images;

[0222] Step C4: If the number of target images is less than the number of images threshold, then obtain the reference material image vector of the reference search image;

[0223] Number of target images: The number of reference search images selected, which is usually dynamically determined based on business needs or user requests.

[0224] Image quantity threshold: The minimum number of images required. If the number of images searched is insufficient, more images will be obtained through a supplementary mechanism.

[0225] Reference image vector: The visual feature vector of the selected reference search image, used to match more images with similar visual features.

[0226] For illustration purposes, if the number of target images is greater than or equal to the image number threshold, all reference search images can be returned; if the number of target images is less than the image number threshold, the reference material image vectors of the reference search images are obtained, and these vectors are used as the basis for the query to perform the next step of similar vector matching.

[0227] Step C6: Based on the reference image vector, perform similar image vector matching on the image vector library to obtain at least one supplementary image, and use the supplementary image as the reference search image.

[0228] Image Vector Library: A database that stores the feature vectors of all source images, used to support efficient vector matching and retrieval.

[0229] Supplementary source images: Images similar to the reference source image vector are matched from the source image vector library and used to supplement the search results.

[0230] This is illustrated by calculating the similarity between each image vector in the source image vector library and the reference source image vector for vector matching. Based on the similarity scores, a recommended number of images most similar to the reference source image vectors are selected as supplementary source images. The recommended number is calculated in real-time to ensure a sufficient number of supplementary images, so that the final number of reference search images reaches the image quantity threshold. At this point, the supplementary source images are added to the reference search image set.

[0231] In this specification, the supplementary mechanism based on similar image vector matching can dynamically adjust the result set to ensure that the number of returned images meets business or user needs. Furthermore, by matching similar images, it expands the diversity of the result set while maintaining the visual semantic relevance to the original reference search images. Additionally, when the initial search results are insufficient, the supplementary mechanism provides users with richer reference images, avoiding empty results or insufficient result quantity.

[0232] The following will combine Figure 6 This specification provides a detailed description of the image search and processing apparatus provided in the embodiments. It should be noted that... Figure 6 The image search and processing device shown is used to execute this specification. Figures 1-5 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 1-5 The example shown.

[0233] Please see Figure 6 This diagram illustrates the structure of an image search processing device according to an embodiment of this specification. The image search processing device 1 can be implemented as all or part of a device through software, hardware, or a combination of both. According to some embodiments, the image search processing device 1 includes a text processing module 11, an index matching module 12, and an image search module 13, specifically used for:

[0234] Text processing module 11 is used to obtain the image search text input by the user and determine the search text vector corresponding to the image search text;

[0235] Index matching module 12 is used to perform similarity index matching on the search text vector to obtain a similar text index data set;

[0236] Image search module 13 is used to perform image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image, and to display the reference search image on the search display interface.

[0237] In one feasible implementation, the index matching module is configured to:

[0238] Obtain multiple text index data from a text index search library, wherein the text search library includes text index data corresponding to multiple source images, and the text index data includes index data between the image description text vector of the source image and the source image vector;

[0239] Using the image description text vector in the text index data as a reference, calculate the text index matching degree between the search text vector and the image description text vector, and determine a first number of similar image description text vectors based on the text index matching degree;

[0240] Obtain the similar text index data corresponding to each of the similar image description text vectors, and generate a similar text index data set based on the similar text index data.

[0241] In one feasible implementation, the device is further used for:

[0242] Obtain multiple source images and their corresponding image text information;

[0243] Determine the material image vector corresponding to the material image based on the material image and the image text information;

[0244] Determine the image description text vector corresponding to the material image based on the image text information;

[0245] Establish text index data between the material image vector corresponding to the material image and the image description text vector;

[0246] A text index search library is generated based on the text index data corresponding to all the aforementioned source images.

[0247] In one feasible implementation, the device is further used for:

[0248] Determine the binary image feature information corresponding to the source image;

[0249] Based on the binary image feature information and the image text information, a vector conversion process is performed to obtain the material image vector;

[0250] Vector segmentation is performed on the image text information to obtain the image description text vector corresponding to the material image.

[0251] In one feasible implementation, the image search module is used for:

[0252] Based on the similar text index data set, at least one candidate search image and a corresponding candidate material image vector are determined.

[0253] Calculate the candidate image similarity between the search text vector and each candidate material image vector;

[0254] At least one reference search image is selected from the candidate search images based on the similarity of the candidate images.

[0255] In one feasible implementation, the image search module is configured to: select at least one reference search image from the candidate search images whose similarity to the candidate images is greater than a similarity threshold.

[0256] In one feasible implementation, the device is further used for:

[0257] Determine the number of target images in the reference search images;

[0258] If the number of target images is less than the number of images threshold, then the reference material image vector of the reference search image is obtained;

[0259] Based on the reference image vector, similar image vector matching is performed on the image vector library to obtain at least one supplementary image, which is then used as the reference search image.

[0260] In one feasible implementation, the image search module is used for:

[0261] Based on the similar text index data set, at least one candidate search image and a corresponding candidate material image vector are determined.

[0262] A reference search image vector is obtained by extracting the search image vector based on the candidate image vectors described above.

[0263] A comprehensive search vector is determined based on the reference search image vector and the search text vector;

[0264] Calculate the candidate image similarity between the comprehensive search vector and each candidate material image vector, and select at least one reference search image from the candidate search images based on the candidate image similarity; and / or, perform search matching processing on the material image vector library based on the comprehensive search vector to obtain at least one reference search image.

[0265] In one feasible implementation, the base image search module is used for:

[0266] The reference search image vector and the search text vector are weighted and combined to obtain the comprehensive search vector.

[0267] It should be noted that the image search processing device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the image search processing method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image search processing device and the image search processing method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0268] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0269] In one or more embodiments of this specification, the electronic device determines the search text vector corresponding to the image search text input by the user, performs similarity index matching on the search text vector to obtain a similar text index data set, performs image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image, and displays the reference search image on the search display interface. Through layer-by-layer refined processing, the user-input search text is transformed into a high-dimensional search text vector and similarity index matching is performed to quickly locate relevant similar text index data sets. Based on the search text vector combined with the similar text index data set, a deeper comprehensive search is performed to accurately filter reference search images highly relevant to the user's needs, which can significantly improve the semantic relevance and retrieval accuracy of search results, while also taking into account search efficiency, providing users with a more intelligent and intuitive text-to-image search experience.

[0270] This specification also provides a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1-5 The image search processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-5 The specific details of the illustrated embodiments will not be elaborated here.

[0271] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-5 The image search processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-5 The specific details of the illustrated embodiments will not be elaborated here.

[0272] Please refer to Figure 7This diagram illustrates a structural block diagram of an electronic device provided in an exemplary embodiment of this specification. The electronic device in this specification may include one or more components such as a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, memory 120, input device 130, and output device 140 may be connected via the bus 150.

[0273] Processor 110 may include one or more processing cores. Processor 110 connects to various parts of the electronic device via various interfaces and lines, and performs various functions and processes data of electronic device 100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 110 may integrate one or more of the following: central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0274] The memory 120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 120 may include a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described below, etc. The operating system may be the Android system, including systems deeply developed based on the Android system, the iOS system developed by Apple Inc., including systems deeply developed based on the iOS system, or other systems. The data storage area may also store data created by the electronic device during use, such as phonebook data, audio and video data, chat log data, etc.

[0275] See Figure 8 As shown, the memory 120 can be divided into operating system space and user space. The operating system runs in the operating system space, while native and third-party applications run in the user space. To ensure that different third-party applications can achieve good running performance, the operating system allocates corresponding system resources for each application. However, different application scenarios within the same third-party application have different requirements for system resources. For example, in local resource loading scenarios, third-party applications have high requirements for disk read speed; in animation rendering scenarios, third-party applications have high requirements for GPU performance. Since the operating system and third-party applications are independent of each other, the operating system often cannot promptly perceive the current application scenario of a third-party application, resulting in the operating system's inability to adapt system resources accordingly to the specific application scenario of the third-party application.

[0276] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0277] Taking the Android operating system as an example, the programs and data stored in memory 120 are as follows: Figure 9As shown, the memory 120 can store the Linux kernel layer 320, the system runtime library layer 340, the application framework layer 360, and the application layer 380. The Linux kernel layer 320, system runtime library layer 340, and application framework layer 360 belong to the operating system space, while the application layer 380 belongs to the user space. The Linux kernel layer 320 provides low-level drivers for various hardware components of the electronic device, such as display drivers, audio drivers, camera drivers, Bluetooth drivers, Wi-Fi drivers, and power management. The system runtime library layer 340 provides support for key features of the Android system through several C / C++ libraries. For example, the SQLite library provides database support, the OpenGL / ES library provides 3D graphics support, and the Webkit library provides browser kernel support. The system runtime library layer 340 also provides the Android runtime library, which mainly provides core libraries that allow developers to write Android applications using the Java language. The Application Framework Layer 360 provides various APIs that may be used when building applications. Developers can also use these APIs to build their own applications, such as activity management, window management, view management, notification management, content provider, package management, call management, resource management, and location management. At least one application runs in the Application Layer 380. These applications can be native applications that come with the operating system, such as contacts, SMS, clock, and camera apps; or third-party applications developed by third-party developers, such as games, instant messaging, and photo editing apps.

[0278] Taking the operating system as an example (iOS), the programs and data stored in memory 120 are as follows: Figure 10As shown, the iOS system includes: Core OS layer 420, Core Services layer 440, Media layer 460, and Cocoa Touch layer 480. Core OS layer 420 includes the operating system kernel, drivers, and low-level program frameworks. These low-level program frameworks provide hardware-level functionality for use by the program frameworks located in Core Services layer 440. Core Services layer 440 provides system services and / or program frameworks required by applications, such as Foundation framework, account framework, advertising framework, data storage framework, network connectivity framework, geolocation framework, motion framework, etc. Media layer 460 provides applications with audiovisual interfaces, such as interfaces related to graphics and images, audio technology, video technology, and AirPlay (wireless playback of audio and video transmission technologies). Cocoa Touch layer 480 provides various commonly used interface-related frameworks for application development and is responsible for user touch interaction on electronic devices. Examples include local notification services, remote push services, advertising frameworks, game tool frameworks, message user interface (UI) frameworks, UIKit user interface frameworks, map frameworks, and so on.

[0279] exist Figure 10 The framework shown includes, but is not limited to, the base framework in the core service layer 440 and the UIKit framework in the touchable layer 480. The base framework provides many basic object classes and data types, offering the most basic system services to all applications, and is independent of the UI. The UIKit framework, on the other hand, provides a basic UI class library for creating touch-based user interfaces. iOS applications can use the UIKit framework to provide their UI, thus providing the application's infrastructure for building user interfaces, drawing, handling user interaction events, responding to gestures, and so on.

[0280] The methods and principles for implementing data communication between third-party applications and the operating system in the iOS system can be found in the Android system, and will not be repeated here.

[0281] The input device 130 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 140 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined into a touch screen, which is used to receive touch operations from the user using a finger, stylus, or any suitable object on or near it, and to display the user interface of various applications. The touch screen is usually located on the front panel of the electronic device. The touch screen can be designed as a full-screen, curved screen, or irregularly shaped screen. The touch screen can also be designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen; this specification does not limit this aspect.

[0282] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WiFi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0283] In the embodiments of this specification, the executing entity for each step can be the electronic device described above. Optionally, the executing entity for each step can be the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems; this specification does not limit this.

[0284] The electronic device described in this specification can also be equipped with a display device. This display device can be any device capable of displaying information, such as a cathode ray tube display (CR), a light-emitting diode display (LED), an e-ink screen, a liquid crystal display (LCD), or a plasma display panel (PDP). Users can use the display device on electronic device 101 to view displayed text, images, videos, and other information. The electronic device can be a smartphone, tablet computer, gaming device, AR (Augmented Reality) device, automobile, data storage device, audio playback device, video playback device, laptop, desktop computing device, or wearable device such as an electronic watch, electronic glasses, electronic helmet, electronic bracelet, electronic necklace, or electronic clothing.

[0285] exist Figure 7 In the illustrated electronic device, the processor 110 can be used to call the application program stored in the memory 120 and specifically perform the following operations:

[0286] Obtain the image search text input by the user, and determine the search text vector corresponding to the image search text;

[0287] The search text vector is subjected to similarity index matching to obtain a similar text index data set;

[0288] Based on the search text vector and the similar text index data set, image search processing is performed to obtain at least one reference search image, which is then displayed on the search display interface.

[0289] In one embodiment, the processor 110 performs the following steps when executing the similarity matching of the search text vector to obtain a similar text index data set:

[0290] Obtain multiple text index data from a text index search library, wherein the text search library includes text index data corresponding to multiple source images, and the text index data includes index data between the image description text vector of the source image and the source image vector;

[0291] Using the image description text vector in the text index data as a reference, calculate the text index matching degree between the search text vector and the image description text vector, and determine a first number of similar image description text vectors based on the text index matching degree;

[0292] Obtain the similar text index data corresponding to each of the similar image description text vectors, and generate a similar text index data set based on the similar text index data.

[0293] In one embodiment, the processor 110 further performs the following steps when executing the image search processing method:

[0294] Obtain multiple source images and their corresponding image text information;

[0295] Determine the material image vector corresponding to the material image based on the material image and the image text information;

[0296] Determine the image description text vector corresponding to the material image based on the image text information;

[0297] Establish text index data between the material image vector corresponding to the material image and the image description text vector;

[0298] A text index search library is generated based on the text index data corresponding to all the aforementioned source images.

[0299] In one embodiment, the processor 110 performs the following steps when executing the process of determining the material image vector corresponding to the material image based on the material image and the image text information, and determining the image description text vector corresponding to the material image based on the image text information:

[0300] Determine the binary image feature information corresponding to the source image;

[0301] Based on the binary image feature information and the image text information, a vector conversion process is performed to obtain the material image vector;

[0302] Vector segmentation is performed on the image text information to obtain the image description text vector corresponding to the material image.

[0303] In one embodiment, the processor 110, after performing image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image, performs the following steps:

[0304] Based on the similar text index data set, at least one candidate search image and a corresponding candidate material image vector are determined.

[0305] Calculate the candidate image similarity between the search text vector and each candidate material image vector;

[0306] At least one reference search image is selected from the candidate search images based on the similarity of the candidate images.

[0307] In one embodiment, the processor 110, when performing the step of selecting at least one reference search image from the candidate search images based on the candidate image similarity, includes: selecting at least one reference search image from the candidate search images whose similarity to the candidate images is greater than a similarity threshold.

[0308] In one embodiment, the processor 110 further performs the following steps when executing the image search processing method:

[0309] Determine the number of target images in the reference search images;

[0310] If the number of target images is less than the number of images threshold, then the reference material image vector of the reference search image is obtained;

[0311] Based on the reference image vector, similar image vector matching is performed on the image vector library to obtain at least one supplementary image, which is then used as the reference search image.

[0312] In one embodiment, the processor 110, after performing image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image, performs the following steps:

[0313] Based on the similar text index data set, at least one candidate search image and a corresponding candidate material image vector are determined.

[0314] A reference search image vector is obtained by extracting the search image vector based on the candidate image vectors described above.

[0315] A comprehensive search vector is determined based on the reference search image vector and the search text vector;

[0316] Calculate the candidate image similarity between the comprehensive search vector and each candidate material image vector, and select at least one reference search image from the candidate search images based on the candidate image similarity; and / or, perform search matching processing on the material image vector library based on the comprehensive search vector to obtain at least one reference search image.

[0317] In one embodiment, the processor 110 performs the following steps when determining the comprehensive search vector based on the reference search image vector and the search text vector:

[0318] The reference search image vector and the search text vector are weighted and combined to obtain the comprehensive search vector.

[0319] In one or more embodiments of this specification, the electronic device determines the search text vector corresponding to the image search text input by the user, performs similarity index matching on the search text vector to obtain a similar text index data set, performs image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image, and displays the reference search image on the search display interface. Through layer-by-layer refined processing, the user-input search text is transformed into a high-dimensional search text vector and similarity index matching is performed to quickly locate relevant similar text index data sets. Based on the search text vector combined with the similar text index data set, a deeper comprehensive search is performed to accurately filter reference search images highly relevant to the user's needs, which can significantly improve the semantic relevance and retrieval accuracy of search results, while also taking into account search efficiency, providing users with a more intelligent and intuitive text-to-image search experience.

[0320] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0321] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. An image search processing method, characterized in that, The method includes: Obtain the image search text input by the user, and determine the search text vector corresponding to the image search text; The search text vector is subjected to similarity index matching to obtain a similar text index data set; Based on the search text vector and the similar text index data set, image search processing is performed to obtain at least one reference search image, which is then displayed on the search display interface.

2. The method according to claim 1, characterized in that, The process of performing similarity matching on the search text vector to obtain a similar text index data set includes: Obtain multiple text index data from a text index search library, wherein the text search library includes text index data corresponding to multiple source images, and the text index data includes index data between the image description text vector of the source image and the source image vector; Using the image description text vector in the text index data as a reference, calculate the text index matching degree between the search text vector and the image description text vector, and determine a first number of similar image description text vectors based on the text index matching degree; Obtain the similar text index data corresponding to each of the similar image description text vectors, and generate a similar text index data set based on the similar text index data.

3. The method according to claim 2, characterized in that, The method further includes: Obtain multiple source images and their corresponding image text information; Based on the material image and the image text information, determine the material image vector corresponding to the material image, and based on the image text information, determine the image description text vector corresponding to the material image; Establish text index data between the material image vector corresponding to the material image and the image description text vector; The text index data corresponding to the material image is stored in the text index search library.

4. The method according to claim 3, characterized in that, The step of determining the material image vector corresponding to the material image based on the material image and the image text information, and determining the image description text vector corresponding to the material image based on the image text information, includes: Determine the binary image feature information corresponding to the source image; Based on the binary image feature information and the image text information, a vector conversion process is performed to obtain the material image vector; Vector segmentation is performed on the image text information to obtain the image description text vector corresponding to the material image.

5. The method according to claim 1, characterized in that, The image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image includes: Based on the similar text index data set, at least one candidate search image and a corresponding candidate material image vector are determined. Calculate the candidate image similarity between the search text vector and each candidate material image vector; At least one reference search image is selected from the candidate search images based on the similarity of the candidate images.

6. The method according to claim 5, characterized in that, The step of selecting at least one reference search image from the candidate search images based on the candidate image similarity includes: At least one reference search image is selected from the candidate search images whose similarity to the candidate images is greater than a similarity threshold.

7. The method according to claim 5, characterized in that, The method further includes: Determine the number of target images in the reference search images; If the number of target images is less than the number of images threshold, then the reference material image vector of the reference search image is obtained; Based on the reference image vector, similar image vector matching is performed on the image vector library to obtain at least one supplementary image, which is then used as the reference search image.

8. The method according to claim 1, characterized in that, The image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image includes: Based on the similar text index data set, at least one candidate search image and a corresponding candidate material image vector are determined. A reference search image vector is obtained by extracting the search image vector based on the candidate image vectors described above. A comprehensive search vector is determined based on the reference search image vector and the search text vector; Calculate the candidate image similarity between the comprehensive search vector and each candidate material image vector, and select at least one reference search image from the candidate search images based on the candidate image similarity; and / or, perform search matching processing on the material image vector library based on the comprehensive search vector to obtain at least one reference search image.

9. The method according to claim 8, characterized in that, The step of determining the comprehensive search vector based on the reference search image vector and the search text vector includes: The reference search image vector and the search text vector are weighted and combined to obtain the comprehensive search vector.

10. An image search and processing device, characterized in that, The device includes: The text processing module is used to obtain the image search text input by the user and determine the search text vector corresponding to the image search text; The index matching module is used to perform similarity index matching on the search text vector to obtain a similar text index data set; The image search module is used to perform image search processing based on the search text vector and the similar text index data set to obtain at least one reference search image, and to display the reference search image on the search display interface.

Images