Searching method and device, electronic equipment and storage medium
The search feature vectors are extracted through the multimodal model and the target search area is determined, which solves the problem of low search efficiency in the existing technology, realizes the need to quickly retrieve emergencies, and improves the user experience.
Patent Information
- Application Number
- CN202410869640.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-30
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, the search efficiency of scene events is low, and it is impossible to quickly retrieve emergencies that need attention, and users need to organize specific search requests.
Through the multimodal model, feature extraction of user search requests is obtained to obtain a search feature vector expressing the user's search intention, determine the target scene search area, and search for base database data in this area to reduce the amount of search data and improve search efficiency.
Scenario events can be quickly retrieved without the need for users to organize specific search requests, which improves search efficiency and reduces the user's search request organization time.
Smart Images

Figure CN118939827A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a search method, device, electronic device and storage medium. Background Art
[0002] Scene events generally refer to emergencies in scenes. Since emergencies in various scenes are prone to suddenness and uncontrollability, they are difficult to monitor. For certain specific needs, it is necessary to pay attention to some sudden public events or natural events, such as natural disasters, traffic accidents, traffic congestion, crowd gatherings and other scene events. Traditional text search or image search will be used. However, traditional text search or image search requires users to organize search requests and cannot quickly retrieve scene events that need attention. The search efficiency of scene events is low. Summary of the invention
[0003] The embodiment of the present invention provides a search method, which aims to solve the problem of low search efficiency of existing scene events. Through the multimodal model, the user's search request is feature extracted, and the search feature vector expressing the user's search intention can be obtained without the user organizing a specific search request, which reduces the user's search request organization time. After obtaining the search feature vector, the target scene search area will be determined according to the search feature vector. The base library feature vectors of different scene types will be stored in different scene search areas. Only the base library data of the corresponding scene is searched in the target scene search area, which reduces the amount of search data and improves the search efficiency of scene events.
[0004] In a first aspect, an embodiment of the present invention provides a search method, the method comprising the following steps:
[0005] Obtaining a search request from a user, where the mode of the search request is a text mode or an image mode;
[0006] Extracting features from the search request using a multimodal large model to obtain a feature vector to be searched;
[0007] Based on the vector similarity between the feature vector to be searched and the scene type vector, a target scene search area is determined, each scene type vector corresponds to a scene search area, and the scene search area has an index structure;
[0008] Based on the index structure of the feature vector to be searched in the target scene search area, searching for a target base library feature vector in the base library feature vectors of the scene search area;
[0009] Based on the target base database feature vector, the target base database data is searched and the target data is returned to the user, and each base database feature vector corresponds to one base database data.
[0010] Optionally, before extracting features from the search request text using the multimodal large model to obtain a feature vector to be searched, the method further includes:
[0011] Collect scene pictures of different scene types and description texts of the scene pictures, each scene picture corresponds to one description text;
[0012] Based on the scene image and the description text, a sample matrix is constructed, wherein the matrix units on the diagonal of the sample matrix are positive sample pairs, and the matrix units on the off-diagonal are negative sample pairs;
[0013] The large model to be trained is trained based on the sample matrix, and after the training is completed, a trained multimodal large model is obtained.
[0014] Optionally, the large model to be trained includes a text feature extraction network and an image feature extraction network. The large model to be trained is trained based on the sample matrix. After the training is completed, a trained multimodal large model is obtained, including:
[0015] For a sample pair in the sample matrix, performing text feature extraction on the description text in the sample pair through the text feature extraction network to obtain a text feature vector, and performing image feature extraction on the scene image in the sample pair through the image feature extraction network to obtain an image feature vector;
[0016] Calculating the vector similarity between the text feature vector and the image feature vector, adjusting parameters of the text feature extraction network and the image feature extraction network with the goal of increasing the vector similarity of the positive sample pair and reducing the vector similarity of the negative sample pair, and iterating the parameter adjustment process;
[0017] When the number of iterations reaches the preset number or when the vector similarity of the positive sample pairs reaches the convergence condition, the training is completed and a trained multimodal large model is obtained.
[0018] Optionally, before determining the target scene search area based on the vector similarity between the feature vector to be searched and the scene type vector, the method further includes:
[0019] Get scene events of different scene types;
[0020] Extracting features of the scene event using the multimodal large model to obtain a scene type vector of the scene event;
[0021] Obtain the underlying database data of different scene types;
[0022] Extracting features of the base database data by using the multimodal large model to obtain a base database feature vector of the base database data;
[0023] According to the similarity between the base database feature vector and the scene type vector, the base database feature vector is partitioned and stored in the base database to obtain scene search areas of different scene types, each scene search area corresponding to a scene type vector.
[0024] Optionally, according to the similarity between the base database feature vector and the scene type vector, partitioning and storing the base database feature vector in the base database to obtain scene search areas of different scene types includes:
[0025] According to the number of the scene type vectors, the base library is divided into multiple storage areas, the multiple storage areas include a common storage area and multiple scene storage areas, each scene type vector is associated with one scene storage area, and the common storage area is not associated with the scene type vector;
[0026] For one of the scene type vectors, the base database feature vectors whose similarity is greater than a similarity threshold are added to a scene storage area associated with the scene type vector in order of the similarity;
[0027] For one of the base database feature vectors, if the similarity between the base database feature vector and all the scene type vectors is less than the similarity threshold, the base database feature vector is added to the public storage area in the order of calculating the similarity;
[0028] After all the base database feature vectors are added, an index structure is constructed for each storage area;
[0029] The index structure is associated with the corresponding storage area to obtain scene search areas of different scene types.
[0030] Optionally, the index structure is a hierarchical index structure, and constructing an index structure for each storage area includes:
[0031] For each of the storage areas, an initial hierarchical index structure is constructed according to the amount of data in the storage area, the amount of data in the hierarchical index structure decreases by level, and each level includes a sparse sampler for the next level;
[0032] The base database feature vectors in the storage area are sparsely sampled based on the initial hierarchical index structure to obtain a hierarchical index structure of the storage area.
[0033] Optionally, searching for a target base database feature vector in the scene search area based on the index structure of the feature vector to be searched in the target scene search area includes:
[0034] If the target scene search area corresponds to the public storage area, searching the feature vector to be searched in the hierarchical index structure according to the hierarchical relationship until the target base database feature vector is found in the public storage area;
[0035] If the target scene search area corresponds to the scene storage area, the feature vector to be searched is searched in a hierarchical index structure according to a hierarchical relationship. When a node with a similarity greater than a skip threshold of the current search level is found, the node is determined as a target node. In the hierarchical index structure, each level includes a plurality of nodes, each of which corresponds to a feature vector of the base library, and different levels correspond to different skip thresholds.
[0036] Determine the base database feature vector corresponding to the target node as a search center vector in the scene storage area;
[0037] Determine the search sorting range based on the level of the target node, with different levels corresponding to different sorting ranges;
[0038] The base database feature vectors within the search sorting range are searched with the search center vector as the center, and the base database feature vector with the highest similarity is used as the target base database feature vector.
[0039] In a second aspect, an embodiment of the present invention further provides a search device, the search device comprising:
[0040] A first acquisition module is used to acquire a user's search request, where the mode of the search request is a text mode or an image mode;
[0041] A first processing module is used to extract features from the search request through a multimodal large model to obtain a feature vector to be searched;
[0042] A second processing module, configured to determine a target scene search region based on the vector similarity between the feature vector to be searched and the scene type vector, wherein each scene type vector corresponds to a scene search region, and the scene search region has an index structure;
[0043] A search module, configured to search for a target base library feature vector in the target scene search area based on the index structure of the feature vector to be searched in the scene search area;
[0044] The search module is used to search for target base database data based on the target base database feature vector and return the target data to the user, wherein each base database feature vector corresponds to one base database data.
[0045] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the search method provided in the embodiment of the present invention when executing the computer program.
[0046] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the search method provided in the embodiment of the invention are implemented.
[0047] In an embodiment of the present invention, a user's search request is obtained, and the mode of the search request is a text mode or an image mode; a feature extraction is performed on the search request through a multimodal large model to obtain a feature vector to be searched; based on the vector similarity between the feature vector to be searched and the scene type vector, a target scene search area is determined, each scene type vector corresponds to a scene search area, and the upper scene search area has an index structure; based on the index structure of the feature vector to be searched in the target scene search area, a target base library feature vector is searched out in the base library feature vector of the scene search area; based on the target base library feature vector, the target base library data is found and returned to the user, and each base library feature vector corresponds to a base library data. The user's search request is feature extracted through a multimodal model, and a search feature vector expressing the user's search intention can be obtained without the user organizing a specific search request, which reduces the user's search request organization time. After obtaining the search feature vector, the target scene search area will be determined according to the search feature vector, and the base library feature vectors of different scene types will be stored in different scene search areas. Only the base library data of the corresponding scene is searched in the target scene search area, which reduces the amount of search data, thereby improving the search efficiency of scene events. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0049] Figure 1 is a flow chart of a search method provided by an embodiment of the present invention;
[0050] Figure 2is a structural schematic diagram of a search device provided in an embodiment of the present invention;
[0051] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] like Figure 1 As shown, Figure 1 is a flow chart of a search method provided by an embodiment of the present invention, including:
[0054] 101. Obtain the user's search request.
[0055] In an embodiment of the present invention, the above-mentioned search method can be applied to a search platform. The above-mentioned search platform can be built based on a server or a distributed server. The above-mentioned search platform is deployed with functional applications such as search applications, databases, large models or interfaces of large models that support data processing and data search. The search application calls the multimodal large model through the interface to perform feature extraction processing. The above-mentioned search platform is connected to the user terminal for communication, and the user can submit a search request to the search platform through the user terminal. The above-mentioned search request can be a multimodal search request. Furthermore, the modality of the above-mentioned search request is a text modality or an image modality.
[0056] The user enters a search request on the user terminal. The search request does not need to be logically organized according to the database rules, and can be a sentence that conforms to the user's language habits or a picture. When the user completes the search request, the user's search request is submitted to the search platform through the user terminal.
[0057] The target of the above search is a scene event, and the scene type of the above scene event may be a natural disaster, a traffic accident, a traffic jam, a crowd gathering, etc. The carrier of the above scene event may be multimodal data, such as text, pictures, audio, etc. The embodiment of the present invention is described by taking text and pictures as examples, and other modal data may be added or replaced.
[0058] 102. Perform feature extraction on the search request through a multimodal large model to obtain a feature vector to be searched.
[0059] In the embodiment of the present invention, the above-mentioned multi-modal large model can be understood as a large model that can receive multiple modal data as input data.
[0060] It should be noted that the multimodal large model of the embodiment of the present invention can be used only for feature extraction of multiple modal data, representing data of different modalities in the same vector space, without the need for classification output.
[0061] The multimodal large model can be a trained special large model, which is trained using sample data of multiple modalities. The multimodal large model can also be an open source large model that complies with relevant regulations.
[0062] The above-mentioned multimodal large model can be deployed in the search platform, or the interface can be set in the search platform for the search platform to call.
[0063] After obtaining the user's search request, the search request is input into the multimodal large model, and the corresponding feature vector is output by the multimodal large model as the feature vector to be searched.
[0064] 103. Determine a target scene search area based on the vector similarity between the feature vector to be searched and the scene type vector.
[0065] In the embodiment of the present invention, after obtaining the feature vector to be searched, the feature vector to be searched can be used to perform scene search in the base library (database) to determine the corresponding scene type, and different scenes correspond to different scene search areas.
[0066] Specifically, the base database has multiple scene search areas, each of which corresponds to a scene type vector, which can be understood as a representative vector of the scene search area. The scene search area includes a number of base database feature vectors, and all base database feature vectors in the same scene search area belong to the same scene type.
[0067] Before conducting a specific search, vector similarity calculation can be performed between the feature vector to be searched and each scene type vector to obtain the vector similarity between the feature vector to be searched and each scene type vector, and the scene search area corresponding to the n scene type vectors with the highest vector similarity is determined as the target scene search area, where n is a number greater than or equal to 1, that is, the number of target scene search areas can be one or more.
[0068] The above scene search area has an index structure, which may be an index structure for general data or a specific index structure for a scene type. The specific index structure will be described in detail later and will not be described here.
[0069] In a possible embodiment, the multiple scene search areas further include a search area without a scene type, and the base database feature vector in the search area without a scene type does not belong to any existing scene type.
[0070] 104. Based on the index structure of the feature vector to be searched in the target scene search area, search for a target base database feature vector from base database feature vectors in the target scene search area.
[0071] In an embodiment of the present invention, after determining the target scene search area, the index structure of the target scene search area is used to search for the feature vector to be searched, and then the base library feature vector with the highest similarity to the feature vector to be searched can be searched in the base library feature vector of the target scene search area as the target base library feature vector. The number of the above target base library feature vectors can be m, where m is an integer greater than or equal to 1. When there are n target scene search areas, each target scene search area will determine m target base library feature vectors, and the total number of target base library feature vectors is n*m.
[0072] 105. Based on the target base database feature vector, the target base database data is searched and obtained, and the target base database data is returned to the user.
[0073] In an embodiment of the present invention, each base database feature vector corresponds to a base database data. After obtaining the target base database feature vector, the base database data corresponding to the target base database feature vector is found as the target base database data according to the mapping relationship between the base database feature vector and the base database data.
[0074] After obtaining the target database data, the search platform can return the target database data to the user terminal, and the user can search for the returned target database data through the user terminal. The above-mentioned database data can be multimodal data of scene events, such as text data, picture data, audio data, etc.
[0075] In an embodiment of the present invention, a user's search request is obtained, and the mode of the search request is a text mode or an image mode; a feature extraction is performed on the search request through a multimodal large model to obtain a feature vector to be searched; based on the vector similarity between the feature vector to be searched and the scene type vector, a target scene search area is determined, each scene type vector corresponds to a scene search area, and the upper scene search area has an index structure; based on the index structure of the feature vector to be searched in the target scene search area, a target base library feature vector is searched out in the base library feature vector of the scene search area; based on the target base library feature vector, the target base library data is found and returned to the user, and each base library feature vector corresponds to a base library data. The user's search request is feature extracted through a multimodal model, and a search feature vector expressing the user's search intention can be obtained without the user organizing a specific search request, which reduces the user's search request organization time. After obtaining the search feature vector, the target scene search area will be determined according to the search feature vector, and the base library feature vectors of different scene types will be stored in different scene search areas. Only the base library data of the corresponding scene is searched in the target scene search area, which reduces the amount of search data, thereby improving the search efficiency of scene events.
[0076] It should be noted that the search method provided in the embodiment of the present invention can be applied to electronic devices such as mobile phones, tablets, computers, servers, etc. that can perform searches.
[0077] It can be understood that in the specific implementation of this application, it involves monitoring data, text data, image data, user data and other related data. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data, as well as the training, deployment and calling of multimodal large models, must comply with the relevant laws, regulations and standards of relevant countries and regions.
[0078] Optionally, before extracting features from the search request text through the multimodal large model to obtain the feature vector to be searched, scene images of different scene types and description texts of the scene images can be collected, with each scene image corresponding to a description text; based on the scene images and the description texts, a sample matrix is constructed, in which the matrix units on the diagonal of the sample matrix are positive sample pairs, and the matrix units on the off-diagonal are negative sample pairs; the large model to be trained is trained based on the sample matrix, and after the training is completed, a trained multimodal large model is obtained.
[0079] In an embodiment of the present invention, the scene type may be a natural disaster, a traffic accident, a traffic jam, a crowd gathering, etc. The scene picture is a picture recording the scene event, and the description text is a text describing the scene event. Of course, in a possible embodiment, other modal data in different scenes may also be collected.
[0080] The scene images are arranged in columns and the description texts are arranged in rows. For a pair of scene images and description texts, the order of the scene images in the columns is the same as the order of the description texts in the rows, or the scene images are arranged in rows and the description texts are arranged in columns, and a sample matrix can be obtained. Specifically, the sample matrix is shown in Table 1 below:
[0081] Table 1
[0082] B1 B2 …… BN A1 (A1, B1) (A1, B2) …… (A1, BN) A2 (A2, B1) (A2, B2) …… (A2, BN) …… …… …… …… AN (AN, B1) (AN, B2) …… (AN, BN)
[0083] In the above sample matrix, each matrix unit corresponds to a sample pair, and the sample pairs on the diagonal such as (A1, B1), (A2, B2), (AN, BN) are positive sample pairs, and the other sample pairs not on the diagonal are negative sample pairs.
[0084] After obtaining the sample matrix, the sample pairs in the sample matrix are used as training samples of the large model to be trained, and the large model to be trained is trained. The large model to be trained can be a pre-trained large model or an untrained large model. The multimodal large model can be a CLIP model.
[0085] The goal of training is to make the large model extract features from positive sample pairs, so that the similarity of feature vector pairs corresponding to positive sample pairs is as large as possible, and the similarity of feature vector pairs corresponding to negative sample pairs is as small as possible. In this way, the same scene event of different modalities can be represented in the same vector space, and no matter what modality the input data is, feature vectors represented by the same vector space will be obtained.
[0086] After the training is completed, a trained multimodal large model is obtained. The trained multimodal large model can be deployed in the search platform, or the trained multimodal large model can be deployed in the server and the calling interface can be set in the search platform.
[0087] Optionally, the large model to be trained includes a text feature extraction network and an image feature extraction network. When the large model to be trained is trained based on the sample matrix, after the training is completed, in the step of obtaining the trained multimodal large model, for a sample pair in the sample matrix, the text feature extraction network can be used to extract text features from the description text in the sample pair to obtain a text feature vector, and the image feature extraction network can be used to extract image features from the scene picture in the sample pair to obtain an image feature vector; the vector similarity between the text feature vector and the image feature vector is calculated, and with the goal of increasing the vector similarity of the positive sample pairs and reducing the vector similarity of the negative sample pairs, the text feature extraction network and the image feature extraction network are adjusted in parameters, and the parameter adjustment process is iterated; when the number of iterations reaches a preset number or when the vector similarity of the positive sample pairs reaches a convergence condition, the training is completed and the trained multimodal large model is obtained.
[0088] The text feature extraction network can be a neural network that can extract text features based on a recurrent neural network RNN, a long short-term memory network LSTM, BERT, etc. The image feature extraction network can be a neural network that can extract image features based on a convolutional neural network CNN, YOLOV, etc.
[0089] By extracting features from the description text through the text feature extraction network, a text feature vector of the description text can be obtained. By extracting features from the scene image through the image feature extraction network, an image feature vector of the scene image can be obtained.
[0090] The goal of training is to maximize the vector similarity between the text feature vector and the image feature vector of the positive sample pair, and at the same time, minimize the vector similarity between the text feature vector and the image feature vector of the negative sample pair. In this way, the trained multimodal large model can represent different modal data of the same scene event through highly similar feature vectors.
[0091] In a possible embodiment, before the description text is input into the text feature extraction network, the description text can be constructed using a prompt template corresponding to the scene type, and the constructed description text is input into the text feature extraction network for feature extraction processing. Each scene type corresponds to a prompt template, and by constructing the description text using prompt templates of different scene types, the scene features can be more prominent.
[0092] By continuously inputting sample pairs for similarity calculation, back propagation is used to continuously adjust the network parameters of the text feature extraction network and the image feature extraction network, thereby realizing the iteration of the parameter adjustment process.
[0093] When the number of iterations reaches a preset number, or the vector similarity of the positive sample pair reaches the convergence condition, the training can be stopped to obtain a trained multimodal large model. The above-mentioned convergence condition can be that the vector similarity distribution of the positive sample approaches the maximum value. For example, if the vector similarity is cosine similarity, the maximum value is 1. When the vector similarity distribution of the positive sample converges (approaches) to 1, it can be considered that the vector similarity of the positive sample pair reaches the convergence condition.
[0094] Optionally, before the step of determining the target scene search area based on the vector similarity between the feature vector to be searched and the scene type vector, scene events of different scene types can also be obtained; feature extraction is performed on the scene events through a multimodal large model to obtain a scene type vector of the scene event; base library data of different scene types is obtained; feature extraction is performed on the base library data through a multimodal large model to obtain a base library feature vector of the base library data; based on the similarity between the base library feature vector and the scene type vector, the base library feature vector is partitioned and stored in the base library to obtain scene search areas of different scene types, each scene search area corresponds to a scene type vector.
[0095] In an embodiment of the present invention, corresponding scene events may be prepared in advance for each scene type, and each scene type corresponds to a scene event. The scene event may be text data, or image data or data in other modalities. The embodiment of the present invention is described with text data. The scene event is a scene description text.
[0096] After the scene event is obtained, the scene event is subjected to feature extraction processing through the above-mentioned trained multimodal large model to obtain the corresponding scene feature vector. Each scene type can correspond to a scene feature vector.
[0097] After the scene feature vectors corresponding to each scene type are obtained, a storage area may be allocated to each scene feature vector in the base database, and the scene feature vectors may be stored in the corresponding storage area.
[0098] At the same time, base database data of different scene types are obtained. The base database data may be data recording scene events occurring in key areas. The key areas may be understood as areas that users need to pay attention to.
[0099] For example, pictures captured by the front-end camera in the key area can be collected. If the pictures contain scene events, the pictures are used as the background data.
[0100] The feature extraction of the base database data is performed through the multimodal large model to obtain the base database feature vector corresponding to the base database data.
[0101] After obtaining the base database feature vector corresponding to the base database data, it is necessary to allocate the base database feature vector to the corresponding storage area for storage. The principle of allocation is to allocate according to the scene type. Specifically, the scene type of the base database feature vector can be determined according to the similarity between the base database feature vector and the scene type vector, so that the base database feature vector of the same scene type can be allocated to the storage area of the corresponding scene type for storage.
[0102] By partitioning and storing the base database feature vectors in the base database according to the scene type, when searching for scene events, the corresponding partitions can also be searched, thereby reducing the amount of search data.
[0103] Optionally, in the step of partitioning and storing the base database feature vector in the base database according to the similarity between the base database feature vector and the scene type vector to obtain scene search areas of different scene types, the base database can be divided into multiple storage areas according to the number of scene type vectors, the multiple storage areas include a common storage area and multiple scene storage areas, each scene type vector is associated with a scene storage area, and the common storage area is not associated with the scene type vector; for a scene type vector, the base database feature vectors whose similarity is greater than a similarity threshold are added to the scene storage area associated with the scene type vector in order of similarity; for a base database feature vector, if the similarity between it and all scene type vectors is less than the similarity threshold, the base database feature vector is added to the common storage area in the order of similarity calculation; after all base database feature vectors are added, an index structure is constructed for each storage area; the index structure is associated with the corresponding storage area to obtain scene search areas of different scene types.
[0104] In the embodiment of the present invention, in the base database, according to the number of scene type vectors, multiple storage areas are divided, the number of storage areas is the number of scene type vectors plus 1, and the multiple storage areas are public storage areas for storing base database feature vectors whose scene types cannot be determined. Each scene type vector is assigned a storage area and stored in the corresponding storage area, the scene storage area.
[0105] The similarity between the base database feature vector to be stored and each scene type vector is traversed and calculated to obtain the similarity between the base database feature vector and each scene type vector. If the similarity between the base database feature vector and a scene type vector is greater than or equal to the similarity threshold, the base database feature vector is added to the scene storage area corresponding to the scene type vector. If there are multiple base database feature vectors in a scene storage area, the multiple base database feature vectors are sorted according to the similarity and stored in the sorted order. In the storage area, each base database feature vector can correspond to a storage node.
[0106] If the similarities between the base database feature vector and all scene type vectors are less than the similarity threshold, the base database feature vector is added to the public storage area, and multiple base database feature vectors in the public storage area can be arranged in the order of addition.
[0107] After all base library feature vectors are added, an index structure is constructed for each storage area. The index structure may be an index structure for general data or a specific index structure for a scene type.
[0108] Optionally, the index structure is a hierarchical index structure. In the step of building an index structure for each storage area, an initial hierarchical index structure can be built for each storage area according to the amount of data in the storage area. The amount of data in the hierarchical index structure decreases by level, and each level includes a sparse sampler for the next level. Based on the initial hierarchical index structure, sparse sampling is performed on the bottom library feature vectors in the storage area to obtain the hierarchical index structure of the storage area.
[0109] In the embodiment of the present invention, each storage area corresponds to a hierarchical index, and the hierarchical index structure includes multiple levels, and the number of nodes included in the lowest level is second only to the number of storage nodes in the storage area. The higher the level, the fewer the number of nodes included. The number of levels of the hierarchical index structure can be determined according to the amount of data in the corresponding storage area. The smaller the amount of data in the storage area, the fewer the number of levels of the corresponding hierarchical index structure. The larger the amount of data in the storage area, the more the number of levels of the corresponding hierarchical index structure.
[0110] The above sparse sampler may be a sampler based on an arithmetic progression, that is, sampling is performed at intervals of a constant. Preferably, the sparse sampler in the embodiment of the present invention may be a sampler based on a Fibonacci sequence.
[0111] Specifically, the upper-level data is obtained by sampling the lower-level data, and specific data can be selected and constructed according to the Fibonacci sequence. For a scene storage area, the bottom library feature vector is sorted according to similarity, and the Fibonacci sequence is used to select nodes. Each time, the next number in the Fibonacci sequence is selected as the node interval (for example, there are 10 nodes from 1 to 10 in the bottom layer, and the second layer will get five nodes 1, 2, 3, 5, and 8). Then, the upper-level index is constructed by obtaining the nodes. The construction rules are the same as the bottom-level index. A dynamic hierarchical algorithm is established, which can automatically construct the number of layers according to the amount of data, and the data of each layer is selected from the next layer. The hierarchical index structure constructed in this way can reduce the amount of data for the search index. For example, in the above example, when there is only the bottom-level index, the bottom-level index needs to traverse the entire bottom-level nodes to search for the number 9. In the case of multi-layer indexing, according to the traversal of the upper-level index, it can be known that in the interval of 8-10 nodes, the upper-level index data volume is less, and the bottom-level data can be quickly obtained according to the upper-level index, which improves the search efficiency.
[0112] After constructing the hierarchical index structure of each storage area, the hierarchical index structure is associated with the corresponding storage area to form a scene search area, and the scene search area includes a search area corresponding to the common storage area.
[0113] Optionally, in the step of searching for a target bottom library feature vector in the scene search area based on the index structure of the feature vector to be searched in the target scene search area, if the target scene search area corresponds to a public storage area, the feature vector to be searched can be searched in the hierarchical index structure according to a hierarchical relationship until the target bottom library feature vector is searched out in the public storage area; if the target scene search area corresponds to the scene storage area, the feature vector to be searched is searched in the hierarchical index structure according to a hierarchical relationship, and when a node whose similarity is greater than the jump threshold of the current search level is searched, the node is determined as the target node, and in the hierarchical index structure, each level includes multiple nodes, each node corresponds to a bottom library feature vector, and different levels correspond to different jump thresholds; in the scene storage area, the bottom library feature vector corresponding to the target node is determined as the search center vector; the search sorting range is determined based on the level where the target node is located, and different levels correspond to different sorting ranges; with the search center vector as the center, the bottom library feature vectors within the search sorting range are searched, and the bottom library feature vector with the highest similarity is taken as the target bottom library feature vector.
[0114] In the embodiment of the present invention, when searching for the feature vector to be searched in the target scene search area, it can be determined whether the target scene search area corresponds to the public storage area. If it corresponds to the public storage area, the layer-skipping search will not be triggered. This is mainly because the feature vectors of the base library in the public storage area are not sorted according to the similarity. It can also be understood that the hierarchical index structure corresponding to the public storage area does not have a layer-skipping mechanism.
[0115] If it corresponds to the scene storage area, a jump-level search will be triggered. By jumping-level search, the search of subsequent levels can be avoided, saving search time. The jump-level search mainly relies on the sorting of the bottom library feature vectors in the scene storage area according to similarity, so that more similar features will be enriched in the corresponding sequence segment. For example, if the similarity between the feature vector to be searched and node 53 is higher than 95, then it is highly likely that the similarity with node 52 is also high. Therefore, the search can be performed in the range before and after node 53. In this way, at worst, node 53 can be used as the target node, and the bottom library feature vector corresponding to node 53 can be used as the target bottom library feature vector.
[0116] Specifically, in the hierarchical index structure corresponding to the scene storage area, each level corresponds to a skip threshold. When searching for the feature vector to be searched, the similarity between each node of the current level and the feature vector to be searched is calculated. When there is a node with a similarity greater than the skip threshold of the level, the node is determined as the target node, and the bottom-level scene storage area is directly searched for the bottom-level feature vector corresponding to the target node. With the bottom-level feature vector as the center, a local search is performed in the sorting range. When there is no node with a similarity greater than the skip threshold of the level, the node with the largest similarity is used as the entry node to enter the next level, and the next level is searched through the entry node.
[0117] It should be noted that the higher the level, the smaller the layer jump threshold and the larger the sorting range, and the lower the level, the lower the layer jump threshold and the smaller the sorting range. Through the layer jump mechanism, when the target scene search area corresponds to the scene storage area, the time required for the search can be effectively reduced, further improving the search efficiency of scene events.
[0118] In a possible embodiment, if the target scene search area corresponds to the scene storage area, the layer-by-layer search may not be triggered.
[0119] In a possible embodiment, after obtaining the target base database data, if the user finds that the target base database data is obviously wrong, the target base database data can be collected for training and adjustment of the multimodal large model to improve the accuracy of the model.
[0120] like Figure 2As shown, an embodiment of the present invention further provides a search device, including:
[0121] A first acquisition module 201 is used to acquire a user's search request, where the mode of the search request is a text mode or an image mode;
[0122] A first processing module 202 is used to extract features from the search request using a multimodal large model to obtain a feature vector to be searched;
[0123] The second processing module 203 is used to determine a target scene search area based on the vector similarity between the feature vector to be searched and the scene type vector, each scene type vector corresponds to a scene search area, and the scene search area has an index structure;
[0124] A search module 204, configured to search for a target base database feature vector in the target scene search area based on the index structure of the feature vector to be searched in the scene search area;
[0125] The search module 205 is used to search for target base database data based on the target base database feature vector and return the target data to the user. Each base database feature vector corresponds to one base database data.
[0126] Optionally, the device further comprises:
[0127] A collection module, used to collect scene pictures of different scene types and description texts of the scene pictures, each scene picture corresponds to one description text;
[0128] A construction module, used to construct a sample matrix based on the scene image and the description text, wherein the matrix units on the diagonal of the sample matrix are positive sample pairs, and the matrix units on the off-diagonal are negative sample pairs;
[0129] The training module is used to train the large model to be trained based on the sample matrix. After the training is completed, a trained multimodal large model is obtained.
[0130] Optionally, the large model to be trained includes a text feature extraction network and an image feature extraction network, and the training module is also used to, for a sample pair in the sample matrix, perform text feature extraction on the description text in the sample pair through the text feature extraction network to obtain a text feature vector, and perform image feature extraction on the scene picture in the sample pair through the image feature extraction network to obtain an image feature vector; calculate the vector similarity between the text feature vector and the image feature vector, and adjust the parameters of the text feature extraction network and the image feature extraction network with the goal of increasing the vector similarity of the positive sample pairs and reducing the vector similarity of the negative sample pairs, and iterate the parameter adjustment process; when the number of iterations reaches a preset number or when the vector similarity of the positive sample pairs reaches a convergence condition, the training is completed to obtain a trained multimodal large model.
[0131] Optionally, the device further comprises:
[0132] The second acquisition module is used to acquire scene events of different scene types;
[0133] A third processing module, configured to extract features of the scene event using the multimodal large model to obtain a scene type vector of the scene event;
[0134] The third acquisition module is used to acquire the base database data of different scene types;
[0135] A fourth processing module is used to extract features from the base database data by using the multimodal large model to obtain a base database feature vector of the base database data;
[0136] The fifth processing module is used to partition and store the base database feature vector in the base database according to the similarity between the base database feature vector and the scene type vector to obtain scene search areas of different scene types, each scene search area corresponding to a scene type vector.
[0137] Optionally, the fifth processing module is also used to divide the base library into multiple storage areas according to the number of the scene type vectors, the multiple storage areas including a common storage area and multiple scene storage areas, each scene type vector is associated with one scene storage area, and the common storage area is not associated with the scene type vector; for one scene type vector, the base library feature vector whose similarity is greater than a similarity threshold is added to the scene storage area associated with the scene type vector in order of similarity; for one base library feature vector, if the similarity between it and all the scene type vectors is less than the similarity threshold, the base library feature vector is added to the common storage area in the order of calculation of the similarity; after all the base library feature vectors are added, an index structure is constructed for each storage area; the index structure is associated with the corresponding storage area to obtain scene search areas of different scene types.
[0138] Optionally, the index structure is a hierarchical index structure, and the fifth processing module is further used to construct an initial hierarchical index structure for each storage area according to the amount of data in the storage area, the amount of data in the hierarchical index structure decreases by level, and each level includes a sparse sampler for the next level; based on the initial hierarchical index structure, the bottom library feature vector in the storage area is sparsely sampled to obtain the hierarchical index structure of the storage area.
[0139] Optionally, the search module 204 is also used to search the feature vector to be searched in the hierarchical index structure according to the hierarchical relationship if the target scene search area corresponds to the public storage area, until the target base library feature vector is searched out in the public storage area; if the target scene search area corresponds to the scene storage area, search the feature vector to be searched in the hierarchical index structure according to the hierarchical relationship, and when a node whose similarity is greater than the jump threshold of the current search level is found, determine the node as the target node, in the hierarchical index structure, each level includes a plurality of nodes, each node corresponds to a base library feature vector, and different levels correspond to different jump thresholds; determine the base library feature vector corresponding to the target node in the scene storage area as the search center vector; determine the search sorting range based on the level where the target node is located, and different levels correspond to different sorting ranges; search the base library feature vectors within the search sorting range with the search center vector as the center, and take the base library feature vector with the highest similarity as the target base library feature vector.
[0140] It should be noted that the search device provided in the embodiment of the present invention can be applied to devices such as smart phones, computers, servers, etc. that can perform the search method.
[0141] The search device provided in the embodiment of the present invention can implement each process implemented by the regional retention method in the above method embodiment, and can achieve the same beneficial effect. To avoid repetition, it will not be described here.
[0142] like Figure 3 As shown, an embodiment of the present invention further provides an electronic device, including a processor, and the processor can execute any one of the above-mentioned search methods.
[0143] Specifically, it includes a processor 301 and a memory 302, and a computer program for executing the search method stored in the memory 302 and capable of running on the processor 301, wherein:
[0144] The processor 301 runs the computer program of the search method stored in the memory 302 to perform the following steps:
[0145] Obtaining a search request from a user, where the mode of the search request is a text mode or an image mode;
[0146] Extracting features from the search request using a multimodal large model to obtain a feature vector to be searched;
[0147] Based on the vector similarity between the feature vector to be searched and the scene type vector, a target scene search area is determined, each scene type vector corresponds to a scene search area, and the scene search area has an index structure;
[0148] Based on the index structure of the feature vector to be searched in the target scene search area, searching for a target base library feature vector in the base library feature vectors of the scene search area;
[0149] Based on the target base database feature vector, the target base database data is searched and the target data is returned to the user, and each base database feature vector corresponds to one base database data.
[0150] Optionally, before extracting features from the search request text using the multimodal large model to obtain a feature vector to be searched, the method executed by the processor 301 further includes:
[0151] Collect scene pictures of different scene types and description texts of the scene pictures, each scene picture corresponds to one description text;
[0152] Based on the scene image and the description text, a sample matrix is constructed, wherein the matrix units on the diagonal of the sample matrix are positive sample pairs, and the matrix units on the off-diagonal are negative sample pairs;
[0153] The large model to be trained is trained based on the sample matrix, and after the training is completed, a trained multimodal large model is obtained.
[0154] Optionally, the large model to be trained includes a text feature extraction network and an image feature extraction network. The processor 301 performs the training of the large model to be trained based on the sample matrix. After the training is completed, a trained multimodal large model is obtained, including:
[0155] For a sample pair in the sample matrix, performing text feature extraction on the description text in the sample pair through the text feature extraction network to obtain a text feature vector, and performing image feature extraction on the scene image in the sample pair through the image feature extraction network to obtain an image feature vector;
[0156] Calculating the vector similarity between the text feature vector and the image feature vector, adjusting parameters of the text feature extraction network and the image feature extraction network with the goal of increasing the vector similarity of the positive sample pair and reducing the vector similarity of the negative sample pair, and iterating the parameter adjustment process;
[0157] When the number of iterations reaches the preset number or when the vector similarity of the positive sample pairs reaches the convergence condition, the training is completed and a trained multimodal large model is obtained.
[0158] Optionally, before determining the target scene search area based on the vector similarity between the feature vector to be searched and the scene type vector, the method executed by the processor 301 further includes:
[0159] Get scene events of different scene types;
[0160] Extracting features of the scene event using the multimodal large model to obtain a scene type vector of the scene event;
[0161] Obtain the underlying database data of different scene types;
[0162] Extracting features of the base database data by using the multimodal large model to obtain a base database feature vector of the base database data;
[0163] According to the similarity between the base database feature vector and the scene type vector, the base database feature vector is partitioned and stored in the base database to obtain scene search areas of different scene types, each scene search area corresponding to a scene type vector.
[0164] Optionally, the processor 301 performs partitioning and storing the base database feature vector in the base database according to the similarity between the base database feature vector and the scene type vector to obtain scene search areas of different scene types, including:
[0165] According to the number of the scene type vectors, the base library is divided into multiple storage areas, the multiple storage areas include a common storage area and multiple scene storage areas, each scene type vector is associated with one scene storage area, and the common storage area is not associated with the scene type vector;
[0166] For one of the scene type vectors, the base database feature vectors whose similarity is greater than a similarity threshold are added to a scene storage area associated with the scene type vector in order of the similarity;
[0167] For one of the base database feature vectors, if the similarity between the base database feature vector and all the scene type vectors is less than the similarity threshold, the base database feature vector is added to the public storage area in the order of calculating the similarity;
[0168] After all the base database feature vectors are added, an index structure is constructed for each storage area;
[0169] The index structure is associated with the corresponding storage area to obtain scene search areas of different scene types.
[0170] Optionally, the index structure is a hierarchical index structure, and the step of constructing an index structure for each storage area executed by the processor 301 includes:
[0171] For each of the storage areas, an initial hierarchical index structure is constructed according to the amount of data in the storage area, the amount of data in the hierarchical index structure decreases by level, and each level includes a sparse sampler for the next level;
[0172] The base database feature vectors in the storage area are sparsely sampled based on the initial hierarchical index structure to obtain a hierarchical index structure of the storage area.
[0173] Optionally, the step of searching for a target base database feature vector in the scene search area based on the index structure of the feature vector to be searched in the target scene search area performed by the processor 301 includes:
[0174] If the target scene search area corresponds to the public storage area, searching the feature vector to be searched in the hierarchical index structure according to the hierarchical relationship until the target base database feature vector is found in the public storage area;
[0175] If the target scene search area corresponds to the scene storage area, the feature vector to be searched is searched in a hierarchical index structure according to a hierarchical relationship. When a node with a similarity greater than a skip threshold of the current search level is found, the node is determined as a target node. In the hierarchical index structure, each level includes a plurality of nodes, each of which corresponds to a feature vector of the base library, and different levels correspond to different skip thresholds.
[0176] Determine the base database feature vector corresponding to the target node as a search center vector in the scene storage area;
[0177] Determine the search sorting range based on the level of the target node, with different levels corresponding to different sorting ranges;
[0178] The base database feature vectors within the search sorting range are searched with the search center vector as the center, and the base database feature vector with the highest similarity is used as the target base database feature vector.
[0179] It should be noted that the electronic device provided in the embodiment of the present invention can be applied to devices such as smart phones, computers, servers, etc. that can perform the search method.
[0180] The electronic device provided in the embodiment of the present invention can implement each process implemented by the regional retention method in the above method embodiment and can achieve the same beneficial effect. To avoid repetition, it will not be described here.
[0181] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the search method provided by the embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0182] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the above-mentioned computer program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned computer-readable storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0183] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
Claims
1. A search method, characterized in that: The method comprises the following steps: Obtaining a search request from a user, where the mode of the search request is a text mode or an image mode; Extracting features from the search request using a multimodal large model to obtain a feature vector to be searched; Based on the vector similarity between the feature vector to be searched and the scene type vector, a target scene search area is determined, each scene type vector corresponds to a scene search area, and the scene search area has an index structure; Based on the index structure of the feature vector to be searched in the target scene search area, searching for a target base database feature vector in the base database feature vectors in the target scene search area; Based on the target base database feature vector, the target base database data is searched and returned to the user, and each base database feature vector corresponds to one base database data.
2. The search method according to claim 1, characterized in that: Before extracting features from the search request text using the multimodal large model to obtain a feature vector to be searched, the method further includes: Collect scene pictures of different scene types and description texts of the scene pictures, each scene picture corresponds to one description text; Based on the scene image and the description text, a sample matrix is constructed, wherein the matrix units on the diagonal of the sample matrix are positive sample pairs, and the matrix units on the off-diagonal are negative sample pairs; The large model to be trained is trained based on the sample matrix, and after the training is completed, a trained multimodal large model is obtained.
3. The search method according to claim 1 or 2, characterized in that: The large model to be trained includes a text feature extraction network and an image feature extraction network. The large model to be trained is trained based on the sample matrix. After the training is completed, a trained multimodal large model is obtained, including: For a sample pair in the sample matrix, performing text feature extraction on the description text in the sample pair through the text feature extraction network to obtain a text feature vector, and performing image feature extraction on the scene image in the sample pair through the image feature extraction network to obtain an image feature vector; Calculating the vector similarity between the text feature vector and the image feature vector, adjusting parameters of the text feature extraction network and the image feature extraction network with the goal of increasing the vector similarity of the positive sample pair and reducing the vector similarity of the negative sample pair, and iterating the parameter adjustment process; When the number of iterations reaches the preset number or when the vector similarity of the positive sample pairs reaches the convergence condition, the training is completed and a trained multimodal large model is obtained.
4. The search method according to any one of claims 1 to 3, characterized in that: Before determining the target scene search area based on the vector similarity between the feature vector to be searched and the scene type vector, the method further includes: Get scene events of different scene types; Extracting features of the scene event using the multimodal large model to obtain a scene type vector of the scene event; Obtain the underlying database data of different scene types; Extracting features of the base database data by using the multimodal large model to obtain a base database feature vector of the base database data; According to the similarity between the base database feature vector and the scene type vector, the base database feature vector is partitioned and stored in the base database to obtain scene search areas of different scene types, each scene search area corresponding to a scene type vector.
5. The search method according to claim 4, characterized in that: According to the similarity between the base database feature vector and the scene type vector, the base database feature vector is partitioned and stored in the base database to obtain scene search areas of different scene types, including: According to the number of the scene type vectors, the base library is divided into a plurality of storage areas, the plurality of storage areas including a common storage area and a plurality of scene storage areas, each of the scene type vectors is associated with one of the scene storage areas, and the common storage area is not associated with the scene type vector; For one of the scene type vectors, the base database feature vectors whose similarity is greater than a similarity threshold are added to a scene storage area associated with the scene type vector in order of the similarity; For one of the base database feature vectors, if the similarity between the base database feature vector and all the scene type vectors is less than the similarity threshold, the base database feature vector is added to the public storage area in the order of calculating the similarity; After all the base database feature vectors are added, an index structure is constructed for each storage area; The index structure is associated with the corresponding storage area to obtain scene search areas of different scene types.
6. The search method according to claim 5, characterized in that: The index structure is a hierarchical index structure, and constructing an index structure for each storage area includes: For each of the storage areas, an initial hierarchical index structure is constructed according to the amount of data in the storage area, the amount of data in the hierarchical index structure decreases by level, and each level includes a sparse sampler for the next level; The base database feature vectors in the storage area are sparsely sampled based on the initial hierarchical index structure to obtain a hierarchical index structure of the storage area.
7. The search method according to claim 6, characterized in that: The step of searching for a target base database feature vector in the target scene search area based on the index structure of the feature vector to be searched in the target scene search area comprises: If the target scene search area corresponds to the public storage area, searching the feature vector to be searched in the hierarchical index structure according to the hierarchical relationship until the target base database feature vector is found in the public storage area; If the target scene search area corresponds to the scene storage area, the feature vector to be searched is searched in a hierarchical index structure according to a hierarchical relationship. When a node with a similarity greater than a skip threshold of the current search level is found, the node is determined as a target node. In the hierarchical index structure, each level includes a plurality of nodes, each of which corresponds to a feature vector of the base library, and different levels correspond to different skip thresholds. Determine the base database feature vector corresponding to the target node as a search center vector in the scene storage area; Determine the search sorting range based on the level of the target node, with different levels corresponding to different sorting ranges; The base database feature vectors within the search sorting range are searched with the search center vector as the center, and the base database feature vector with the highest similarity is used as the target base database feature vector.
8. A search device, characterized in that: The search device comprises: A first acquisition module is used to acquire a user's search request, where the mode of the search request is a text mode or an image mode; A first processing module is used to extract features from the search request through a multimodal large model to obtain a feature vector to be searched; A second processing module, configured to determine a target scene search region based on the vector similarity between the feature vector to be searched and the scene type vector, wherein each scene type vector corresponds to a scene search region, and the scene search region has an index structure; A search module, configured to search for a target base database feature vector in the target scene search area based on the index structure of the feature vector to be searched in the target scene search area; The search module is used to search for target base database data based on the target base database feature vector and return the target base database data to the user, wherein each base database feature vector corresponds to one base database data.
9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the search method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the search method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Feature base library construction method, feature retrieval method and related device
CN113255610A
Vehicle intelligent identification method and device based on AI vision and electronic equipment
CN115424217A
Target domain information search method and model training method and device
CN115827732A
Target object retrieval method and device, computer equipment and storage medium
CN117668275A
Electronic device and search method thereof, and medium
WO2021180109A1
Cited By
Railway geological disaster early warning method, device, equipment, medium and product
CN119600782A
Image retrieval method and device and storage medium
CN120448574A
Image retrieval method, device, and storage medium
CN120448574B
Vector retrieval method and device for multiple service scenes, storage medium and program product
CN121301445A
Vector retrieval method, device, storage medium and program product for multi-service scenario
CN121301445B
