Processing method and device and electronic equipment

By extracting and mapping different forms of input information and generating unified search requests, the one-sided problem of traditional search engines in multimodal information retrieval is solved, and more accurate and rich search results are achieved to meet the complex needs of users.

CN118916526APending Publication Date: 2024-11-08LENOVO (BEIJING) LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411026069.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional search engines find it difficult to achieve unified search when facing multimodal information, resulting in users being unable to obtain search results that fully meet their needs.

Method used

Feature extraction and mapping are performed by obtaining different forms of input information (such as text, pictures, audio, video), and using a cross-modal attention model based on deep learning to map different modal data to a shared semantic space, and using weighted averages Method fusion feature representation generates a unified search request.

Benefits of technology

It achieves more accurate and rich search results, meets the complex and diverse information needs of users, improves the accuracy and relevance of search results, and reduces the time cost of information retrieval.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916526A_ABST
    Figure CN118916526A_ABST
Patent Text Reader

Abstract

The invention discloses a processing method and device and electronic device.The method comprises the steps that input information is obtained, the input information comprises first information in a first form and second information in a second form, and the first form is different from the second form; obtaining first processing information according to the first information, and obtaining second processing information according to the second information; and generating a search request corresponding to the input information according to the first processing information, the first form, the second processing information and the second form.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of information processing, and relate to but are not limited to a processing method, device, and electronic device. Background Art

[0002] Traditional search engines face a series of challenges when facing multimodal information. These challenges stem from the diversity and complexity of different types of data (such as text, images, audio, and video). Traditional search engines often need to rely on different engines or process data separately, and cannot achieve unified retrieval of multimodal information well. In addition, due to the one-sidedness of search results for single-modal information, users often cannot obtain search results that fully meet their needs. Summary of the invention

[0003] In view of this, embodiments of the present application provide a processing method, device, and electronic device.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a processing method, the method comprising:

[0006] Obtain input information, the input information including first information in a first form and second information in a second form, the first form being different from the second form; obtain first processing information based on the first information, and obtain second processing information based on the second information; generate a search request corresponding to the input information based on the first processing information and the first form, and the second processing information and the second form.

[0007] In a second aspect, an embodiment of the present application provides a processing device, the device comprising:

[0008] A first obtaining module is used to obtain input information, the input information includes first information in a first form and second information in a second form, the first form is different from the second form; a second obtaining module is used to obtain first processing information based on the first information, and obtain second processing information based on the second information; a generating module is used to generate a search request corresponding to the input information based on the first processing information and the first form, and the second processing information and the second form.

[0009] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and the processor implements the above method when executing the program.

[0010] In a fourth aspect, an embodiment of the present application provides a storage medium storing executable instructions for implementing the above method when executed by a processor.

[0011] In a fifth aspect, a computer program product includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the steps in the above method are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of an implementation flow of a processing method provided in an embodiment of the present application;

[0013] Figure 2 A schematic diagram of an implementation flow of generating a search request provided in an embodiment of the present application;

[0014] Figure 3 A schematic diagram of a multimodal search process provided in an embodiment of the present application;

[0015] Figure 4 A flowchart of document retrieval based on a large language model provided in an embodiment of the present application;

[0016] Figure 5 A schematic diagram of the structure of a processing device provided in an embodiment of the present application;

[0017] Figure 6 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the embodiments of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0019] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0020] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0022] The present application embodiment provides a processing method, such as Figure 1 As shown, the method includes:

[0023] Step S110: obtaining input information, the input information including first information in a first form and second information in a second form, the first form being different from the second form;

[0024] Here, the input information may be information in different forms, including but not limited to the following forms: text, picture, audio and video.

[0025] In some embodiments, the first information in the first form may be text information in a text form, and the second information in the second form may be picture information in a picture form.

[0026] In some embodiments, the first information in the first form may be audio information in an audio form, and the second information in the second form may be video information in a video form.

[0027] The different forms of information provided above can be arbitrarily combined into the first information and the second information.

[0028] In some embodiments, the input information may also include first information in a first form, second information in a second form, and third information in a third form. For example, the input information may include text information, picture information, and audio information. In the implementation process, information in different forms may be arbitrarily combined to obtain the first information, the second information, and the third information.

[0029] In some embodiments, the input information may further include first information in a first form, second information in a second form, third information in a third form, and fourth information in a fourth form. For example, the input information may include text information, picture information, audio information, and video information.

[0030] During the implementation, there is no restriction on the form of information, and there is no restriction on the form of input information.

[0031] Step S120: obtaining first processing information according to the first information, and obtaining second processing information according to the second information;

[0032] Here, the first processed information may be first feature information corresponding to the first information obtained by extracting features from the first information; and the second processed information may be second feature information corresponding to the second information obtained by extracting features from the second information.

[0033] For example, we can extract features from different types of data, such as text, images, audio, and video, and obtain their own feature representations, that is, converting the original multimodal data into a feature representation that can be understood and processed by computers.

[0034] Step S130: Generate a search request corresponding to the input information according to the first processing information and the first form, and the second processing information and the second form.

[0035] Here, the search request is request information for searching in a database.

[0036] During the implementation process, information from different forms (different modal data) can first be mapped into a shared semantic space to facilitate subsequent fusion and analysis. At this stage, a cross-modal attention model based on deep learning can be used to realize the mapping relationship between modalities. This can learn the complex relationship between different modal data and automatically discover the corresponding relationship between modalities during the learning process. By introducing the attention mechanism, these models can better handle the heterogeneity and correlation between different modal data, thereby achieving more accurate modal mapping.

[0037] After the mapping between forms (modalities) is completed, the next step is to fuse the information of different modalities into a unified feature representation for subsequent analysis and processing. In the feature fusion process, considering the importance and contribution of different modal features, the weighted average method can be used to convert the data of different modalities into a unified semantic space and fuse them into a unified feature representation, namely, a unified multimodal vector (Embedding).

[0038] During implementation, the fused multimodal vector can be used as the search request corresponding to the input information.

[0039] In an embodiment of the present application, input information is first obtained, and the input information includes first information in a first form and second information in a second form, and the first form is different from the second form; then, first processing information is obtained according to the first information, and second processing information is obtained according to the second information; finally, a search request corresponding to the input information is generated according to the first processing information and the first form, and the second processing information and the second form. In this way, it is possible to generate search requests according to information in different forms and enrich the information forms included in the search requests; further, by using multimodal information as a search request, more comprehensive and diverse search results can be provided to meet the user's more extensive and complex information needs. The fusion and collaboration between information in different forms make the search results richer and more diverse, so as to better meet the actual needs of users. Through the fusion of information in different forms, the accuracy and relevance of search results are improved, and the search experience is improved. Users can use search requests to find the required information more quickly, reduce the time cost of information retrieval, and improve search efficiency and satisfaction.

[0040] In some embodiments, the above processing method further comprises the following steps:

[0041] converting the first information into third information in a third form;

[0042] In some embodiments, the third form is the same as the second form. For example, the first information may be picture information, the second information may be text information, and the picture may be identified to obtain text information describing the picture, and the text information may be used as the third information.

[0043] In some embodiments, the third form is different from the second form. For example, the first information may be text information: "Search for an introduction video of a scenic spot", the second information may be a photo of the scenic spot, and the third information may be related video information of the scenic spot.

[0044] The above step S120 "obtaining first processing information according to the first information, and obtaining second processing information according to the second information" can be implemented by the following process:

[0045] Obtain the first processing information according to the first information, obtain the second processing information according to the second information, and obtain the third processing information according to the third information;

[0046] For example, the first information may be "search for dog pictures", the second information may be pictures including dogs, and the third information may be pictures of dogs generated based on the first information. In the implementation process, a first feature vector corresponding to "search for dog pictures" may be obtained based on the first information; the second information may be a second feature vector corresponding to the pictures of dogs; and the third information may be a third feature vector corresponding to the generated pictures of dogs.

[0047] Correspondingly, the step S130 of "generating a search request corresponding to the input information according to the first processing information and the first form, and the second processing information and the second form" can be implemented by the following process:

[0048] A search request corresponding to the input information is generated based on the first processing information and the first form, the second processing information and the second form, and the third processing information and the third form, wherein the third form is the same as the second form, or the third form is different from the second form.

[0049] During implementation, a search request may be generated in the first form, the second form, and the third form based on a first feature vector generated by the first information, a second feature vector generated by the second information, and a third feature vector generated by the third information.

[0050] In the embodiment of the present application, the first information is converted into third information in a third form; the first processed information is obtained according to the first information, the second processed information is obtained according to the second information, and the third processed information is obtained according to the third information; a search request corresponding to the input information is generated according to the first processed information and the first form, the second processed information and the second form, and the third processed information and the third form. In this way, a richer information form of the search request can be achieved, and more search results matching the search request can be retrieved.

[0051] In some embodiments, the above step S120 "obtaining first processing information according to the first information, and obtaining second processing information according to the second information" can be implemented by the following steps:

[0052] Step 123: convert the first information into a first feature vector;

[0053] Step 123: convert the second information into a second feature vector;

[0054] During the implementation process, the information forms include text, pictures, audio, video, etc. For different types of data such as text, pictures, audio, video, etc., feature extraction is performed as follows:

[0055] Text feature extraction: Text data usually exists in the form of word sequences, so a common method is to use pre-trained word embedding models (such as Word2Vec, GloVe, FastText) to convert text into fixed-dimensional word vector representations. These word vectors can capture the semantic and contextual information between words and establish a representation for text data in a continuous vector space.

[0056] Image feature extraction: For image data, convolutional neural networks (CNN) (such as VGG, ResNet and other models) can extract high-level feature representations from images through multi-layer convolution and pooling operations. These features usually have abstract representations of information such as objects, textures, shapes, etc. in the image, providing a basis for semantic understanding of image data.

[0057] Audio feature extraction: Mel-Frequency Spectrum (MFC) and Mel Frequency Cepstrum Coefficient (MFCC) are used to extract the spectral information and acoustic features of the audio. The linear predictive coding method is used to decompose the audio signal into a linear filter and a residual signal, and the vocal tract information of the speech is extracted. The spectrogram method is used to convert the audio signal into a time-frequency image, which reflects the changes in the audio in time and frequency.

[0058] Video feature extraction: Use 3D convolutional neural network to perform three-dimensional convolution operation on video data to extract the spatiotemporal features of the video. Use optical flow method to estimate the motion information between video frames and capture the motion trajectory of objects in the video. Use frame difference method: calculate the difference between adjacent video frames to extract dynamic change information in the video. Use video histogram features: Count the pixel distribution in the video frame and extract the color and brightness features of the video.

[0059] Correspondingly, the step S130 of "generating a search request corresponding to the input information according to the first processing information and the first form, and the second processing information and the second form" can be implemented by the following process:

[0060] The search request is generated according to the first feature vector, a first form corresponding to the first feature vector, the second feature vector, and a second form corresponding to the second feature vector.

[0061] In the embodiment of the present application, the first information is converted into a first feature vector; the second information is converted into a second feature vector; and the search request is generated according to the first feature vector, the first form corresponding to the first feature vector, the second feature vector, and the second form corresponding to the second feature vector. In this way, information in different forms can be fused into a search request, so that the search request can reflect information in multiple forms input.

[0062] In some embodiments, Figure 2As shown, the above step of "generating the search request according to the first feature vector, the first form corresponding to the first feature vector, the second feature vector and the second form corresponding to the second feature vector" can be implemented by the following steps:

[0063] Step S210: obtaining an association relationship between the first information and the second information;

[0064] For example, the first information may be "dog" and the second information may be a picture of a dog, then the first information and the second information have a higher correlation. The first information may be "search for dog breeds in pictures" and the second information may be a picture of a dog, then the weight of the second information may be greater than the weight of the first information.

[0065] Step S220: determining a first weighting coefficient of the first eigenvector and a second weighting coefficient of the second eigenvector based on the association relationship;

[0066] For example, when the first information is “dog” and the second information is a picture of a dog, it can be determined that both the first weighting coefficient and the second weighting coefficient are 0.5.

[0067] When the first information is "search for dog breeds in pictures" and the second information is pictures including dogs, it can be determined that the first weighting coefficient is 0.25 and the second weighting coefficient is 0.75.

[0068] Step S230: Based on the first weighting coefficient and the second weighting coefficient, the first form corresponding to the first feature vector and the second form corresponding to the second feature vector, perform weighted processing on the first feature vector and the second feature vector to generate the search request.

[0069] In the embodiment of the present application, the association relationship between the first information and the second information is first obtained; then, based on the association relationship, a first weighting coefficient of the first feature vector and a second weighting coefficient of the second feature vector are determined; finally, based on the first weighting coefficient and the second weighting coefficient, a first form corresponding to the first feature vector and a second form corresponding to the second feature vector, the first feature vector and the second feature vector are weighted to generate the search request. In this way, the obtained search request is more likely to match a suitable search result.

[0070] In some embodiments, the above step S220 "determining a first weighting coefficient of the first feature vector and a second weighting coefficient of the second feature vector based on the association relationship" can be implemented by the following steps:

[0071] Step 221: Determine a search form of search results based on the input information, wherein the search form is the first form or the second form;

[0072] Here, the input information may be parsed to determine the search form of the search results that the user wishes to obtain. For example, when the text information of the input information is "search for pictures of dogs of the French bulldog breed", it may be determined that the search form of the search results is pictures.

[0073] Step 222: Determine the first weighting coefficient and the second weighting coefficient based on the search form and the association relationship.

[0074] When it is determined that the search form is the same as the first form, the first weighting coefficient can be adjusted and increased. Similarly, when it is determined that the search form is the same as the second form, the second weighting coefficient can be adjusted and increased to effectively improve the accuracy of the search.

[0075] During implementation, in addition to considering the search form, the association relationship between the first information and the second information may also be considered simultaneously, so as to determine the first weighting coefficient and the second weighting coefficient based on the search form and the association relationship.

[0076] In the embodiment of the present application, the search form of the search results is first determined based on the input information; then the first weighting coefficient and the second weighting coefficient are determined based on the search form and the association relationship. In this way, the search request generated by considering the first weighting coefficient and the second weighting coefficient determined by the search form can improve the accuracy of the search.

[0077] Figure 3 A schematic diagram of a multimodal search process provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, it includes the following four stages: stage one, input of multimodal information; stage two, extraction of multimodal information; stage three, mapping and fusion of multimodal information; stage four, retrieval of multimodal information, among which,

[0078] Stage 1: Input of multimodal information;

[0079] First, you need to determine the type of multimodal information that needs to be retrieved, such as text, pictures, audio, video, etc. Determine the type of multimodal information that needs to be retrieved based on the specific application scenario and user needs. The format of text information includes but is not limited to: WORD, PDF, TXT, PPT; the format of picture information includes but is not limited to: JPG, PNG, TIFF; the format of audio information includes but is not limited to: MP3, WA, AAC; the format of video information includes but is not limited to: MP4, MOV, MKV.

[0080] Stage 2: Multimodal information extraction;

[0081] For different types of data such as text, pictures, audio, and video, feature extraction is performed to obtain their respective feature representations. The work at this stage is to convert the original multimodal data into a feature representation that can be understood and processed by computers.

[0082] For example, we can extract features from different modal data such as text T, image P, audio A, and video V in the input information and convert them into feature vector Tokens of corresponding types, where:

[0083] The text feature vector is denoted as T, which is expressed as T = [t1, ..., t m ];

[0084] The image feature vector is denoted as P, which is expressed as P = [p1, ..., p n ];

[0085] The audio feature vector is recorded as A, which is expressed as A = [a1, ..., a k ];

[0086] The video feature vector is denoted as V, which is expressed as V = [v1, ..., v l ].

[0087] Stage 3: Multimodal mapping and fusion;

[0088] Multimodal mapping stage: Map information from different modal data into a shared semantic space for subsequent fusion and analysis. In this stage, a cross-modal attention model based on deep learning can be used to realize the mapping relationship between modalities. This can learn the complex relationship between different modal data and automatically discover the corresponding relationship between modalities during the learning process. By introducing the attention mechanism, these models can better handle the heterogeneity and correlation between different modal data, thereby achieving more accurate modal mapping.

[0089] Multimodal fusion stage: After the mapping between modalities is completed, the next step is to fuse the information of different modalities into a unified feature representation for subsequent analysis and processing. In the feature fusion process, the importance and contribution of different modal features are considered, and the weighted average method is used to convert the data of different modalities into a unified semantic space and fuse them into a unified feature representation, namely the unified multimodal vector (Embedding).

[0090] Unified multimodal storage: After generating unified multimodal vectors (Embeddings) for each text, image, audio, and video, these Embeddings are saved in the database together with the corresponding file path components.

[0091] In the implementation process, the information from different modal data can be mapped into a shared semantic space to make full use of the information of each different modal data. To this end, all possible modal feature vector Tokens can be combined individually or in pairs. There are 10 combinations, such as T, P, A, V, TP, TA, TV, PA, PV and AV. For example, TP represents the mapping that will generate text and picture feature vector Tokens, that is, TP = [t1, ..., t m , p1,…,p n ], PV represents the mapping of image and video feature vector Tokens, that is, PV = [p1, ..., p n , v1,…,v l ].

[0092] After completing the mapping between different modalities, the information of different modal data can be fused into a unified multimodal vector, denoted as TPAV. This vector contains four-dimensional Tokens vectors of text T, image P, audio A, and video V. Considering the importance and contribution of different modal features, the system defines a four-dimensional fusion function to perform weighted average of the corresponding modal data dimensions on the 10 mapped feature vector Tokens:

[0093] F(T)=a1T+a2TP+a3TA+a4TV;

[0094] F(P)=b1P+b2TP+b3PA+b4PV;

[0095] F(A)=c1A+c2TA+c3PA+c4AV;

[0096] F(V)=d1V+d2TV+d3PV+d4AV;

[0097] Among them, a1, a2, a3, and a4 are the weighted average coefficients of T, TP, TA, and TV related to the text dimension, b1, b2, b3, and b4 are the weighted coefficients of P, TP, PA, and PV related to the image dimension, c1, c2, c3, and c4 are the weighted average coefficients of A, TA, PA, and AV related to the audio dimension, and d1, d2, d3, and d4 are the weighted coefficients of V, TV, PV, and AV related to the video dimension. In this way, the feature vector Tokens of different modal data can be converted into a four-dimensional unified multimodal vector TPAV. The above four groups of weighting coefficients can be set according to actual needs, or determined according to the contribution of different forms of input information to the search results.

[0098] Stage 4: Retrieval of multimodal information;

[0099] Feature extraction and query conversion: After a query is given, the above-mentioned stages 2 and 3 are used to extract and convert its features, so as to convert the query into a unified multimodal vector form of the query instruction (QueryEmbeddings), namely the query vector.

[0100] Retrieve candidate set: After obtaining the query vector, search the database for candidate sets related to the query. The candidate set contains all documents, pictures, audio or video information related to the query vector.

[0101] After generating unified multimodal vectors for text, images, audio, and video, the system saves these vectors together with the metadata consisting of the corresponding file paths into the database. When a user submits a query, the system uses the above steps to extract and transform its features, and converts the query into a unified multimodal vector Query TPAV form of the query instruction. Then, the system calculates the cosine similarity between the query instruction Query TPAV vector and some other TPAV vectors stored in the database in four dimensions, and performs weighted average summation to obtain the final similarity value. Based on the similarity value, the system returns the file path information corresponding to some of the TOP N TPAVs before the value to obtain the final retrieval results (i.e., related text, images, audio, and video). The calculation formula (1) is as follows:

[0102] Match(TPAV)=M1Cos(F(QueryT),F(T))+M2Cos(F(QueryP),F(P))+M3Cos(F(QueryA),F(A))+M4Cos(F(QueryV),F(V)) (1);

[0103] Among them, M1 to M4 are weighting coefficients, F(QueryT) is a text query vector, F(T) is a text vector stored in the database, F(QueryP) is an image query vector, F(P) is an image vector existing in the database, F(QueryA) is an audio query vector, F(A) is an audio vector stored in the database, F(QueryV) is a video query vector, and F(V) is a video vector existing in the database.

[0104] Semantic matching: For each candidate, calculate its semantic similarity with the query vector. This can be achieved by calculating the cosine similarity between the query features and the candidate features.

[0105] Comprehensive sorting: The calculated semantic similarities are comprehensively sorted.

[0106] Return results: Based on the results of the comprehensive ranking, the top candidates are returned to the user as the final search results. Usually, a threshold is set to determine the number of results returned based on user preferences and system requirements.

[0107] The above weighted correlation coefficient is determined as follows:

[0108] All weighted correlation coefficients (a1 to d4, M1 to M4) can be trained and confirmed by minimizing the loss function. For example, a triplet loss function can be used to drive model learning to ensure that similar samples are closer in the embedding space and dissimilar samples are farther away. The system constructs a triplet, including an anchor sample, a positive sample, and a negative sample, where the anchor sample and the positive sample are from the same category, while the negative sample is from a different category. The goal of the loss function is to make the distance between the anchor sample and the positive sample as small as possible, and at the same time make the distance between the anchor sample and the negative sample as large as possible. The specific loss function (2) is defined as follows:

[0109] L(Xa,Xp,Xn)=max(0,d(Xa,Xp)-d(Xa,Xn)+α) (2);

[0110] Among them, Xa is the multimodal vector representation of the anchor sample, Xp is the multimodal vector representation of the positive sample, Xn is the multimodal vector representation of the negative sample, d(,) represents the distance metric function between samples, and cosine similarity can be used. α is the edge value, which is used to control the distance interval between positive and negative samples to ensure that they have sufficient separation in the embedding space.

[0111] Anchor samples are fixed samples during the training process and used to calculate the loss function. Positive samples are samples similar to anchor samples, while negative samples are samples that are dissimilar to anchor samples. Through training, the model is required to minimize the distance between anchor samples and positive samples and maximize the distance between anchor samples and negative samples, so that the model can learn better feature representation.

[0112] For example, an anchor sample can be a specific book, a positive sample can be other books similar to the book, and a negative sample can be other books unrelated to the book. Through such sample pairs, the system can learn how to classify similar books into one category and how to distinguish dissimilar books, thereby improving the accuracy and effectiveness of retrieval.

[0113] In the embodiments of the present application, multimodal information extraction is carried out for different modal information such as text, pictures, audio, and video, and a series of feature extraction methods are used to convert them into a unified feature representation. For text information, not only the semantic meaning of words is considered, but also the syntactic structure and contextual information are combined for feature extraction; for picture information, a convolutional neural network (CNN) is used to extract visual features to capture the high-level representation of the image; for audio information, acoustic features are extracted using methods such as Mel spectrum features; and for video information, a 3D convolutional neural network (3D CNN) is used for feature extraction in combination with spatiotemporal information. It can fully integrate different modal information such as text, pictures, audio, and video to achieve cross-modal intelligent retrieval. No longer limited to a single modal information, users can obtain more comprehensive and rich search results by inputting data in various forms.

[0114] The mapping and fusion between modalities uses a cross-modal attention network model to map different modal information into a unified semantic space and achieve efficient fusion of cross-modal information. This process not only takes into account the heterogeneity between different modalities, but also makes full use of the commonalities and correlations between modalities, thereby achieving an organic combination and deep fusion of information. Through the learned shared representation, the semantic associations between different modal information are better captured and expressed, providing richer and more accurate semantic information for subsequent semantic matching and search.

[0115] Semantic matching and comprehensive sorting realize cross-modal intelligent search by calculating the similarity between different modal information in a unified semantic space. The fusion and semantic matching of multimodal information can improve the accuracy and relevance of search results and improve the search experience. Users can find the information they need more quickly, reduce the time cost of information retrieval, and improve search efficiency and satisfaction. Taking into account the similarity of multimodal information and user needs, the search results are comprehensively sorted through a series of algorithms and models. This process not only fully considers the correlation between each modal information, but also fully combines the user's personalized needs and search history, thereby providing more accurate, personalized and diversified search results.

[0116] In some embodiments, the database corresponding to the search request includes at least one information block obtained by dividing the stored information based on different forms and preset information amounts, and each information block is corresponding to a first index and a second index, wherein the first index is summary information obtained by extracting all information in the information block, and the second index is all the information in the information block.

[0117] During the implementation process, all document contents stored in the database can be divided into information blocks of appropriate size according to certain rules. This division can be based on the length of the document, paragraph structure, different forms of the document, or specific tags, to ensure that each information block (document block) contains an appropriate amount of information, neither too large to reduce processing efficiency nor too small to cause information loss. The divided document blocks are used as the basic units in the system for subsequent processing and retrieval.

[0118] After completing the document content segmentation and vectorization, you can also perform hierarchical index design on these document blocks, that is, set the first index and the second index, as follows:

[0119] Set the first index: The first index is composed of the summary of the document block and several questions generated by the Large Language Model (LLM), which is used to quickly filter out relevant document blocks. First, for each document block, extract its summary information, including key sentences or paragraphs, and some important information related to the document content. Then, use LLM to generate some questions that may be related to the document content. These questions can be keywords extracted from the summary or some common questions related to the document content. Combine the summary information of the document block and the generated questions together to build an index entry and store it in the first index.

[0120] Set up a second index: This second index consists of actual document blocks and is used to further retrieve related document blocks. All document blocks are directly stored in the second index to form a library of document blocks. This second index is used to perform a deeper document block search when the first index cannot meet the retrieval requirements.

[0121] In the embodiment of the present application, the database includes at least one information block obtained by dividing the stored information based on different forms and preset information amounts, and each of the information blocks is correspondingly provided with a first index and a second index. In this way, the amount of information contained in each information block can be moderate, neither too large to reduce processing efficiency nor too small to cause information loss. The hierarchical index design improves retrieval efficiency and reduces retrieval time.

[0122] In some embodiments, the above processing method further comprises the following steps:

[0123] Step S140: Match the first index in the database based on the search request to obtain search results;

[0124] During implementation, the vector corresponding to the search request may be matched with the vector corresponding to the first index in the data to obtain an information block corresponding to the first index as the search result.

[0125] or,

[0126] Step S150: Match the first index in the database based on the search request to obtain at least one information block;

[0127] During the implementation process, the vector corresponding to the search request can be matched with the vector corresponding to the first index in the data to obtain at least one information block corresponding to the first index, that is, the entire content of the actual information block. When it is determined that the entire content cannot satisfy the search request, the following step S160 can be executed to continue a deeper search in the second index.

[0128] Step S160: Match at least one piece of information in the at least one information block based on the search request to obtain the search result.

[0129] During implementation, the search request may be used to further perform an in-depth search in at least one information block to obtain at least one piece of information as a search result.

[0130] In the embodiment of the present application, the search result is obtained based on the search request matching the first index in the database; or the at least one information block is obtained based on the search request matching the first index in the database; the search result is obtained based on the search request matching at least one piece of information in the at least one information block. In this way, different retrieval requirements can be met, and the required information block can be quickly retrieved, or a specific piece of information can be further retrieved.

[0131] In some embodiments, the above processing method further comprises the following steps:

[0132] Step S170: In response to the information amount of the search result being less than an information amount threshold, determining at least one target information block associated with the second index based on the second index corresponding to the search result;

[0133] When it is determined that the amount of information in the search results is small, the most relevant second index and at least one target information block of the context of the second index may be expanded to form richer information content.

[0134] Step S180: Update the search results based on the at least one target information block.

[0135] In the embodiment of the present application, when the amount of information of the search result is less than the information amount threshold, at least one target information block associated with the second index is determined based on the second index corresponding to the search result; and the search result is updated based on the at least one target information block. In this way, when the amount of information retrieved is small, at least one target information can be expanded to provide users with richer information content as search results.

[0136] In some embodiments, the above processing method further comprises the following steps:

[0137] Step S190: sort and output information blocks corresponding to different forms in the search results based on the first form and the second form, wherein the search results are determined in the database based on the search request.

[0138] In the implementation process, the information blocks corresponding to different forms in the search results can be sorted and output based on the first form and the second form in the input information. For example, when the first form is text and the second form is image, the information blocks in the text and image forms in the search results can be sorted and output first.

[0139] In the embodiment of the present application, information blocks corresponding to different forms in the search results are sorted and output based on the first form and the second form. In this way, since the input information includes the first form and the second form, information blocks corresponding to different forms in the search results can be sorted and output based on the form in the input information, which more accurately matches the sorting requirements expected by the user.

[0140] In some embodiments, the above step S190 of “sorting and outputting information blocks corresponding to different forms in the search results based on the first form and the second form” can be implemented by the following steps:

[0141] Step 191: Match the information form of each information block in the search result with the first form and the second form to determine a first correlation coefficient of each information block;

[0142] Step 192: perform keyword matching on the summary information corresponding to each information block in the search result and the search request to obtain a second correlation coefficient of each information block;

[0143] Here, the keyword-based retrieval algorithm can match the search request with the summary of the document block by keywords and calculate the second correlation coefficient. This method is suitable for simple semantic matching and is fast.

[0144] Step 193: semantically match all information of each information block in the search result with the search request to obtain a third correlation coefficient of each information block;

[0145] Here, based on the semantic retrieval algorithm, LLM can be used to semantically match the search request and the information block and calculate the third correlation coefficient. This method can more accurately capture the semantic information of the document content and improve the accuracy and relevance of the search results.

[0146] Step 194: performing weighted averaging on the first correlation coefficient, the second correlation coefficient and the third correlation coefficient to obtain a target correlation coefficient of each information block;

[0147] Step 195: sort all the target correlation coefficients to obtain a sort result of all the information blocks in the search result.

[0148] In the embodiment of the present application, all information blocks in the search results are sorted based on the form of the input information, keyword matching and semantic matching. In this way, when sorting all information blocks in the search results, the information blocks corresponding to different forms in the search results are sorted based on the form in the input information, and the keyword information and semantic information of the information blocks in the search results can be captured more accurately for sorting, thereby improving the accuracy and relevance of the search result sorting.

[0149] Figure 4 A document retrieval flow chart based on a large language model provided in an embodiment of the present application is as follows: Figure 4 As shown, the following steps are included:

[0150] Step S410: obtaining a search request;

[0151] Step S420: determining a conversion route according to the search request;

[0152] First, the conversion router takes different processing methods according to the complexity of the search request to improve the search efficiency and accuracy. An example is as follows:

[0153] When the search request is determined to be a simple query, the routing does not need to be converted. The user's search request is directly submitted to the document retrieval system to perform subsequent corresponding retrieval operations. Among them, a simple query can be a single-step query with clear query information.

[0154] When it is determined that the search request consists of multiple steps, the system uses sub-problem decomposition to break down the search request into multiple separate sub-queries. The purpose of this is to break down complex search request tasks into multiple simple sub-tasks, each of which can perform retrieval operations independently, and then merge or further combine the results of each sub-query to finally obtain a complete retrieval result. For example, if the user's search request is "how to learn programming and master data structures and algorithms", the system can break it down into two sub-queries, "learn programming" and "master data structures and algorithms", perform retrieval operations on each, and then merge the results.

[0155] Determine that the search request is multiple unrelated queries, that is, when the search request contains multiple sub-queries that are not related to each other, the system converts the search request into multiple multi-step queries. In this case, the system will treat each sub-query as an independent query task and process them separately. For example, if the user's query is "When is the latest model of Apple mobile phone released" and "How to make a delicious pizza", the system will process these two queries separately instead of combining them into a compound query.

[0156] Before the retrieval system executes subsequent search request operations, it first completes the following operations on the database:

[0157] Operation 1: Document content is stored in the database in blocks and vectorized;

[0158] Document content segmentation: First, the system divides all document contents stored in the database into information blocks of appropriate size according to certain rules. This segmentation can be based on the length of the document, paragraph structure, or specific tags to ensure that each document block contains the right amount of information, neither too large to reduce processing efficiency nor too small to cause information loss. The divided document blocks are used as the basic units in the system for subsequent processing and retrieval.

[0159] Vectorization: Each document block is vectorized to convert text information into numerical form, which is convenient for semantic matching and similarity calculation. Each document block is vectorized using pre-trained large language models such as BERT and GPT. By inputting the text into LLM, its corresponding semantic vector representation is obtained.

[0160] Operation 2: Design hierarchical indexes for the database;

[0161] After completing the document content segmentation and vectorization, you can also perform hierarchical indexing on these document blocks, as follows:

[0162] Set the first index: The first index is composed of the summary of the document block and several questions generated by LLM, which is used to quickly filter out relevant document blocks. First, for each document block, extract its summary information, including key sentences or paragraphs, and some important information related to the document content. Then, use LLM to generate some questions that may be related to the document content. These questions can be keywords extracted from the summary or some common questions related to the document content. Combine the summary information of the document block and the generated questions together to construct an index entry and store it in the first index.

[0163] Set up a second index: This second index consists of actual document blocks and is used to further retrieve related document blocks. All document blocks are directly stored in the second index to form a library of document blocks. This second index is used for more in-depth document block retrieval when the first index cannot satisfy the search request.

[0164] Step S430: Search in the database based on the conversion route;

[0165] Here, the search phase includes information filtering and information enhancement. The search phase refers to filtering out information related to the search request from a large number of document blocks and enhancing the search results to provide more accurate and rich retrieval results. It includes the following two steps of search:

[0166] In the first step of searching, the candidate document blocks are quickly filtered using the summary and hypothetical questions of the first index;

[0167] In the implementation process, the system first uses the summary information of the document blocks stored in the first index and the hypothetical questions associated with it to quickly filter the candidate document blocks. The summary information can help the system quickly understand the topic and key points of the document, while the hypothetical questions can guide the system to more accurately match the user's query intention.

[0168] The second step is to search: according to the size of the document, it can be divided into the following two situations for processing;

[0169] Scenario 1: Search operation when the document block is large: When the candidate document block is large, the system adopts a method of embedding each sentence separately, and inputs each sentence in the document block into the language model for vectorization. Combined with the search request, the system further retrieves the sentences most relevant to the query. These sentences may contain key information of the search request, which helps to improve the accuracy of the search results. Then, the system expands the most relevant single sentence and its surrounding context by a certain number of sentences to form richer information content, and sends them together to the LLM for further processing and analysis.

[0170] Scenario 1: Search operation when the document block is small: When the candidate document block is small, the system expands the previous and next document blocks and sends several previous and next document blocks of the candidate document block to LLM for processing. The purpose of this is to provide more context information to help LLM better understand and analyze the document content, thereby improving the quality and accuracy of search results.

[0171] Step S440, filtering and sorting the search results;

[0172] Here, the search results may be summarized, including re-filtering and re-ranking the search results.

[0173] The multiple document blocks searched out in step S430 need to be further filtered and sorted to ensure the quality of the search results and improve the search efficiency. The following two steps are included:

[0174] 1. Duplicate content removal and spam content filtering: The system compares the document content in the search results and removes duplicate document blocks to ensure the diversity and effectiveness of the results. At the same time, some spam content filtering technologies, such as rule-based filtering and classification filtering based on machine learning models, are used to exclude document blocks that are irrelevant to the search request or are of low quality from the search results.

[0175] 2. Reordering phase: includes the following two steps:

[0176] (1) Calculating relevance values: The system uses two methods to calculate the relevance value of each document block and the search request, namely, a keyword-based retrieval algorithm and a semantic-based retrieval algorithm.

[0177] (A) Keyword-based retrieval algorithm: The system matches the search request with the summary of the document block by keywords and calculates the relevance value. This method is suitable for simple semantic matching and is fast, but it may ignore the semantic information of the document content.

[0178] (B) Based on semantic retrieval algorithm: The system uses LLM to semantically match search requests and document blocks and calculate the relevance value. This method can more accurately capture the semantic information of document content and improve the accuracy and relevance of search results.

[0179] (2) Weighted average to obtain the final document block relevance value: The system performs a weighted average of the two calculated relevance values ​​to obtain the final document block relevance value. By adjusting the weights, the impact of keyword matching and semantic matching on search results can be balanced, further improving the quality and accuracy of search results.

[0180] Step S450: Processing search results using a large language model based on historical chat records;

[0181] During implementation, chat history data can be embedded into LLM, and search results can be processed based on LLM.

[0182] The chat history system can retrieve the historical context related to the search request and send this historical information to LLM together with the final retrieved document block and search request, so as to provide a more coherent and personalized answer. The specific operation process is as follows:

[0183] 1. Retrieve chat history context: When a user initiates a search request, the system first retrieves the user's chat history from the chat history vector database, including previous conversation content, query history, etc. This historical information can provide valuable contextual information to help the system better understand the user's intentions and background.

[0184] 2. Integrate search requests and search document blocks: The system integrates the user's current search request with the searched related document blocks to form a complete query context. This query context includes the user's specific query content and the relevant information searched by the system, providing richer input for LLM.

[0185] 3. Send to LLM for processing: The integrated query context and search results are sent to LLM for processing. LLM can use this information for semantic understanding and reasoning to generate more coherent and personalized answers.

[0186] 4. Re-update of chat history data: All responses made by LLM to the search request together with the search request will be re-embedded into the chat history vector database as new historical records so as to better understand and respond to the search request next time.

[0187] The embodiment of the present application provides a document retrieval enhancement solution based on a large language model. The solution includes a series of retrieval enhancement processes including document content segmentation and vectorization, hierarchical index design, search request conversion routing, search information filtering and information enhancement, re-filtering and re-ranking of search summaries, and chat history data embedding.

[0188] The advantages of adopting this solution are as follows:

[0189] 1. Semantic understanding and matching: Achieve more accurate semantic understanding and matching through a large language model, thereby improving the relevance and accuracy of search results.

[0190] 2. Multi-layer indexing improves retrieval efficiency: Layered index design and multi-step search strategy improve retrieval efficiency and reduce retrieval time.

[0191] 3. Personalization and history engine: Combines search request history and personalized needs to provide search results and answers that are more in line with user needs.

[0192] 4. Comprehensive filtering and sorting: Multiple strategies are used to remove duplicate content and filter junk content. At the same time, dual sorting based on keywords and semantics improves the quality of results.

[0193] 5. System optimization and learning: The embedding of LLM history records and query conversion routing provide the system with opportunities for optimization and learning, continuously improving search results and user experience.

[0194] Based on the foregoing embodiments, the embodiments of the present application provide a processing device, which includes the modules included, each module includes each sub-module, each sub-module includes a unit, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0195] Figure 5 A schematic diagram of the structure of the processing device provided in the embodiment of the present application is shown in FIG. Figure 5 As shown, the device 500 includes:

[0196] A first obtaining module 510, configured to obtain input information, the input information including first information in a first form and second information in a second form, the first form being different from the second form;

[0197] A second obtaining module 520, configured to obtain first processing information according to the first information, and obtain second processing information according to the second information;

[0198] The generating module 530 is configured to generate a search request corresponding to the input information according to the first processing information and the first form, and the second processing information and the second form.

[0199] In some embodiments, the processing method also includes a conversion module for converting the first information into third information in a third form; the second acquisition module 520 is also used to obtain the first processing information based on the first information, obtain the second processing information based on the second information, and obtain the third processing information based on the third information; correspondingly, the generation module 530 is also used to generate a search request corresponding to the input information based on the first processing information and the first form, the second processing information and the second form, and the third processing information and the third form, wherein the third form is the same as the second form, or the third form is different from the second form.

[0200] In some embodiments, the second acquisition module 520 includes a first conversion submodule and a second conversion submodule, wherein the first conversion submodule is used to convert the first information into a first feature vector; the second conversion submodule is used to convert the second information into a second feature vector; correspondingly, the generation module 530 is also used to generate the search request based on the first feature vector, the first form corresponding to the first feature vector, the second feature vector and the second form corresponding to the second feature vector.

[0201] In some embodiments, the generation module 530 includes an acquisition submodule, a determination submodule and a generation submodule, wherein the acquisition submodule is used to acquire the association relationship between the first information and the second information; the determination submodule is used to determine the first weighting coefficient of the first feature vector and the second weighting coefficient of the second feature vector based on the association relationship; the generation submodule is used to perform weighted processing on the first feature vector and the second feature vector based on the first weighting coefficient and the second weighting coefficient, the first form corresponding to the first feature vector and the second form corresponding to the second feature vector, to generate the search request.

[0202] In some embodiments, the determination submodule includes a first determination unit and a second determination unit, wherein the first determination unit is used to determine a search form of search results based on the input information, wherein the search form is the first form or the second form; and the second determination unit is used to determine the first weighting coefficient and the second weighting coefficient based on the search form and the association relationship.

[0203] In some embodiments, the database corresponding to the search request includes at least one information block obtained by dividing the stored information based on different forms and preset information amounts, and each information block is corresponding to a first index and a second index, wherein the first index is summary information obtained by extracting all information in the information block, and the second index is all the information in the information block.

[0204] In some embodiments, the processing device also includes a first matching module, or a second matching module and a third matching module, wherein the first matching module is used to match the first index in the database based on the search request to obtain search results; the second matching module is used to match the first index in the database based on the search request to obtain at least one information block; and the third matching module is used to match at least one piece of information in the at least one information block based on the search request to obtain the search results.

[0205] In some embodiments, the processing device also includes a determination module and an update module, wherein the determination module is used to determine at least one target information block associated with the second index corresponding to the search result based on the second index corresponding to the search result when the amount of information of the search result is less than an information amount threshold; and the update module is used to update the search result based on the at least one target information block.

[0206] In some embodiments, the processing device further comprises a sorting module for sorting and outputting information blocks corresponding to different forms in the search results based on the first form and the second form, wherein the search results are determined in a database based on the search request.

[0207] In some embodiments, the sorting module includes a first matching submodule, a second matching submodule, a third matching submodule, a weighted average submodule and a sorting submodule, wherein the first matching submodule is used to match the information form of each information block in the search results with the first form and the second form to determine the first correlation coefficient of each information block; the second matching submodule is used to match the summary information corresponding to each information block in the search results with the search request by keywords to obtain the second correlation coefficient of each information block; the third matching submodule is used to semantically match all the information of each information block in the search results with the search request to obtain the third correlation coefficient of each information block; the weighted average submodule is used to weighted average the first correlation coefficient, the second correlation coefficient and the third correlation coefficient to obtain the target correlation coefficient of each information block; the sorting submodule is used to sort all the target correlation coefficients to obtain the sorting results of all the information blocks in the search results.

[0208] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0209] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0210] Correspondingly, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the processing method provided in the above embodiment are implemented.

[0211] Correspondingly, an embodiment of the present application provides an electronic device, Figure 6 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the hardware entity of the device 600 includes: a memory 601 and a processor 602, wherein the memory 601 stores a computer program that can be run on the processor 602, and the processor 602 implements the steps in the processing method provided in the above embodiment when executing the program.

[0212] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or processed by the processor 602 and various modules in the electronic device 600 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0213] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0214] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0215] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0216] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0217] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0218] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0219] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0220] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0221] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0222] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0223] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0224] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A processing method, the method comprising: obtaining input information, the input information comprising first information in a first form and second information in a second form, the first form being different from the second form; Obtain first processing information according to the first information, and obtain second processing information according to the second information; A search request corresponding to the input information is generated based on the first processing information and the first form, and the second processing information and the second form.

2. The method of claim 1, further comprising: converting the first information into third information in a third form; Correspondingly, obtaining first processing information according to the first information, and obtaining second processing information according to the second information, includes: Obtain the first processing information according to the first information, obtain the second processing information according to the second information, and obtain the third processing information according to the third information; Correspondingly, generating a search request corresponding to the input information according to the first processing information and the first form, and the second processing information and the second form, includes: A search request corresponding to the input information is generated based on the first processing information and the first form, the second processing information and the second form, and the third processing information and the third form, wherein the third form is the same as the second form, or the third form is different from the second form.

3. The method according to claim 1, wherein obtaining first processing information according to the first information and obtaining second processing information according to the second information comprises: Converting the first information into a first feature vector; converting the second information into a second feature vector; Correspondingly, generating a search request corresponding to the input information according to the first processing information and the first form, and the second processing information and the second form, includes: The search request is generated according to the first feature vector, a first form corresponding to the first feature vector, the second feature vector, and a second form corresponding to the second feature vector.

4. The method according to claim 3, wherein generating the search request according to the first feature vector, the first form corresponding to the first feature vector, the second feature vector and the second form corresponding to the second feature vector comprises: Acquire an association relationship between the first information and the second information; Determining a first weighting coefficient of the first eigenvector and a second weighting coefficient of the second eigenvector based on the association relationship; Based on the first weighting coefficient and the second weighting coefficient, a first form corresponding to the first eigenvector and a second form corresponding to the second eigenvector, weighted processing is performed on the first eigenvector and the second eigenvector to generate the search request.

5. The method according to claim 4, wherein determining a first weighting coefficient of the first feature vector and a second weighting coefficient of the second feature vector based on the association relationship comprises: Determining a search form of search results based on the input information, wherein the search form is the first form or the second form; The first weighting coefficient and the second weighting coefficient are determined based on the search form and the association relationship.

6. The method according to claim 1, wherein the database corresponding to the search request includes at least one information block obtained by dividing the stored information based on different forms and preset information amounts, and each of the information blocks is correspondingly provided with a first index and a second index, wherein: The first index is summary information obtained by extracting all information in the information block, and the second index is all information in the information block.

7. The method of claim 6, further comprising: Obtain search results based on matching the search request to the first index in the database; or, Obtaining at least one of the information blocks based on matching the search request to the first index in the database; The search result is obtained based on matching the search request with at least one piece of information in the at least one information block.

8. The method of claim 7, further comprising: In response to the information amount of the search result being less than an information amount threshold, determining at least one target information block associated with the second index based on the second index corresponding to the search result; The search results are updated based on the at least one target information block.

9. The method according to any one of claims 1 to 8, further comprising: Based on the first form and the second form, information blocks corresponding to different forms in the search results are sorted and output, wherein the search results are determined in the database based on the search request.

10. The method according to claim 9, wherein the step of sorting and outputting information blocks in different forms in the search results based on the first form and the second form comprises: Matching the information form of each information block in the search result with the first form and the second form, and determining a first correlation coefficient of each information block; Perform keyword matching on the summary information corresponding to each information block in the search result and the search request to obtain a second correlation coefficient of each information block; semantically matching all information of each information block in the search result with the search request to obtain a third correlation coefficient of each information block; Taking a weighted average of the first correlation coefficient, the second correlation coefficient and the third correlation coefficient to obtain a target correlation coefficient of each information block; All the target correlation coefficients are sorted to obtain a sorting result of all the information blocks in the search result.