Information query method and device, electronic equipment and medium

Abstract

The invention relates to an information query method and device, electronic equipment and a medium. The information query method comprises the following steps: in response to query information input by a user, inputting the query information into a target model to obtain first intention information, content information and a first text vector; according to the first intention information, a corresponding target database is determined, and the target database is an audio-video database and / or a text database; determining a target vector matched with the first text vector in a target database according to the content information; and outputting target information corresponding to the target vector. The thought of the user is predicted through the target model, and the target information is queried from various types of databases, so that various types of query tasks can be realized, and the use efficiency of the model is improved. Meanwhile, due to the fact that the target model can achieve multiple types of query tasks, the number of the models needing to be deployed is reduced, and therefore the storage space occupied by the models is reduced.

Description

Technical Field

[0001] This disclosure relates to the field of information retrieval technology, and in particular to an information retrieval method, apparatus, electronic device, and medium. Background Technology

[0002] With the development of artificial intelligence technology, multimodal models, as a type of model capable of processing information from multiple modalities, have been widely used to provide users with convenient query services. However, because multimodal models can only perform a single type of query task during use, their efficiency is low. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an information query method, apparatus, electronic device, and medium.

[0004] According to a first aspect of the present disclosure, an information query method is provided, the information query method comprising:

[0005] In response to the query information input by the user, the query information is input into the target model to obtain first intent information, content information and first text vector;

[0006] Based on the first intent information, a corresponding target database is determined, wherein the target database is an audio-visual database and / or a text database;

[0007] Based on the content information, determine the target vector in the target database that matches the first text vector;

[0008] Output the target information corresponding to the target vector.

[0009] In some embodiments of this disclosure, inputting the query information into the target model to obtain first intent information, content information, and a first text vector includes:

[0010] The query information is input into the text encoder in the target model to obtain the first text vector;

[0011] The first text vector is input into the prediction network in the target model to obtain the first intent information and the content information.

[0012] In some embodiments of this disclosure, determining the corresponding target database based on the first intent information includes:

[0013] If the first intent information points to audio-visual information, the audio-visual database is determined to be the target database;

[0014] If the first intent information points to text information, the text database is determined to be the target database;

[0015] If the first intent information points to audio-visual information and text information, the audio-visual database and the text database are determined to be the target database.

[0016] In some embodiments of this disclosure, before determining the corresponding target database based on the first intent information, the information query method further includes:

[0017] Acquire the first audio-visual information and the first text information stored in the electronic device;

[0018] Based on the first audio-visual information, construct the audio-visual database;

[0019] Based on the first text information, construct the text database.

[0020] In some embodiments of this disclosure, constructing the audio-visual database based on the first audio-visual information includes:

[0021] The first audio-visual information is input into the audio-visual encoder in the target model to obtain the first audio-visual vector;

[0022] The audio-visual database is constructed using the first audio-visual vector; and / or,

[0023] The step of constructing the text database based on the first text information includes:

[0024] The first text information is input into the text encoder in the target model to obtain the second text vector;

[0025] The text database is constructed using the second text vector.

[0026] In some embodiments of this disclosure, the first text information includes long text information and / or short text information; and / or, the first text information is monolingual text information or multilingual text information.

[0027] In some embodiments of this disclosure, before inputting the query information into the target model to obtain the first intent information, content information, and first text vector, the information query method further includes:

[0028] Acquire the first audio-visual text pair information, the first long and short text pair information, and the second text information containing the second intent information;

[0029] Based on the first audio-visual text pair information and the first long and short text pair information, the audio-visual encoder and text encoder in the first model are trained to obtain the second model;

[0030] Based on the second text information, the prediction network in the second model is trained to obtain the target model.

[0031] In some embodiments of this disclosure, the step of training the audio-visual encoder and text encoder in the first model based on the first audio-visual text pair information and the first long-short text pair information to obtain the second model includes:

[0032] Construct a first loss function to match audio / video vectors and text vectors;

[0033] Construct a second loss function to match long text vectors and short text vectors;

[0034] Based on the first loss function and the second loss function, a third loss function is constructed;

[0035] Based on the first audio-visual text pair information, the first long and short text pair information, and the third loss function, the audio-visual encoder and the text encoder in the first model are trained to obtain the second model.

[0036] In some embodiments of this disclosure, training the audio-visual encoder and the text encoder in the first model based on the first audio-visual text pair information, the first long-short text pair information, and the third loss function to obtain the second model includes:

[0037] The audio-visual information and text information in the first audio-visual text pair information are respectively input into the audio-visual encoder and the text encoder in the first model to obtain the second audio-visual vector and the third text vector;

[0038] The first long and short text pair information is input into the text encoder in the first model to obtain the fourth text vector corresponding to the long text information and the fifth text vector corresponding to the short text information.

[0039] The second model is obtained by training the audio-visual encoder and the text encoder in the first model based on the second audio-visual vector, the third text vector, the fourth text vector, the fifth text vector, and the third loss function.

[0040] In some embodiments of this disclosure, training the prediction network in the second model based on the second text information to obtain the target model includes:

[0041] Construct a fourth loss function to match predicted information with real information;

[0042] Based on the text encoder in the second model, the prediction network in the second model is trained according to the second text information and the fourth loss function to obtain the target model.

[0043] In some embodiments of this disclosure, the text encoder in the second model is a monolingual text encoder; before training the prediction network in the second model based on the second text information to obtain the target model, the information query method further includes:

[0044] Obtain the first multilingual text pair information;

[0045] Construct a fifth loss function to match text vectors from different languages;

[0046] Based on the first multilingual text pair information and the fifth loss function, the translation model is trained using the text encoder in the second model to obtain a multilingual text encoder;

[0047] The multilingual text encoder is used instead of the monolingual text encoder as the text encoder in the second model.

[0048] In some embodiments of this disclosure, after training the prediction network in the second model based on the second text information to obtain the target model, the information query method further includes:

[0049] Acquire the second audio-visual text pair information, the second long and short text pair information, and the third text information containing the third intent information;

[0050] A sixth loss function is constructed based on the third loss function used to train the first model and the fourth loss function used to train the second model;

[0051] Based on the second audio-visual text pair information, the second long and short text pair information, the third text information, and the sixth loss function, the audio-visual encoder, the text encoder, and the prediction network in the target model are trained to obtain the trained target model.

[0052] According to a second aspect of the present disclosure, an information query device is provided, the information query device comprising:

[0053] An input module is configured to respond to user-inputted query information by inputting the query information into a target model to obtain first intent information, content information, and a first text vector.

[0054] A first determining module is configured to determine a corresponding target database based on the first intent information, wherein the target database is an audio-visual database and / or a text database.

[0055] The second determining module is configured to determine, based on the content information, a target vector in the target database that matches the first text vector;

[0056] An output module is configured to output target information corresponding to the target vector.

[0057] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0058] processor;

[0059] Memory used to store the processor's executable instructions;

[0060] The processor is configured to execute the information query method described above.

[0061] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when instructions in the storage medium are executed by a processor of a terminal, enables the terminal to perform the information query method as described above.

[0062] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0063] In response to user-inputted query information, the user needs to retrieve target information. Since the target model can understand natural language to predict the user's thoughts, the query information is input into the target model, yielding first intent information, content information, and a first text vector. Based on the first intent information, the corresponding audio-visual database and / or text database is determined as the target database to identify the database containing the target information. Because multiple vectors in the target database may be related to the first text vector, the vectors are filtered based on the content information to determine the target vector in the target database that matches the first text vector, and the target information corresponding to the target vector is output. By using the target model to predict user thoughts and retrieve target information from multiple types of databases, various query tasks can be implemented, thereby improving the model's efficiency. Simultaneously, because the target model can perform multiple query tasks, the number of models required for deployment is reduced, thus reducing the storage space occupied by the model.

[0064] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0066] Figure 1 This is a flowchart illustrating an information query method according to an exemplary embodiment;

[0067] Figure 2 This is a flowchart illustrating an information query method according to another exemplary embodiment;

[0068] Figure 3 This is a flowchart illustrating an information query method according to another exemplary embodiment;

[0069] Figure 4 This is a schematic diagram illustrating a first model training process according to an exemplary embodiment;

[0070] Figure 5 This is a flowchart illustrating an information query method according to another exemplary embodiment;

[0071] Figure 6 This is a schematic diagram illustrating the translation model training process according to an exemplary embodiment;

[0072] Figure 7 This is a flowchart illustrating an information query method according to another exemplary embodiment;

[0073] Figure 8 This is a schematic diagram illustrating the target model training process according to an exemplary embodiment;

[0074] Figure 9 This is a schematic diagram illustrating an information query process according to an exemplary embodiment;

[0075] Figure 10 This is a flowchart illustrating an information query method according to another exemplary embodiment;

[0076] Figure 11 This is a block diagram illustrating an information query device according to an exemplary embodiment;

[0077] Figure 12 This is a block diagram of an electronic device according to an exemplary embodiment.

[0078] In the picture:

[0079] 100 - Input module; 150 - First determining module; 200 - Second determining module; 250 - Output module; 400 - Electronic device; 402 - Processing component; 404 - Memory; 406 - Power supply component; 408 - Multimedia component; 410 - Audio component; 412 - Input / output interface; 414 - Sensor component; 416 - Communication component; 420 - Processor. Detailed Implementation

[0080] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims. It should also be understood that the term “and / or” as used in this disclosure refers to and includes any or all possible combinations of one or more of the associated listed items.

[0081] With the development of artificial intelligence technology, multimodal models, as a type of model capable of processing information from multiple modalities, have been widely used to provide users with convenient query services. For example, multimodal models can process image information, text information, video information, audio information, and so on.

[0082] In related technologies, an information query method is provided, including: responding to user-inputted text information, inputting the text information into a multimodal model to obtain a text vector; determining the image vector corresponding to the text vector in an image database; and outputting the image information corresponding to the image vector. Since the multimodal model can output corresponding image information based on user-inputted text information, it allows users to query the image information they need, providing convenience. However, because the multimodal model can only perform the single task of image information query during use, its efficiency is low. Furthermore, increasing the tasks that the multimodal model can handle requires setting up additional multimodal models, resulting in a large storage space occupied by the model.

[0083] To address the aforementioned technical issues, this disclosure provides an information query method. This method uses a target model to understand the natural language input by the user and retrieves one or more matching target vectors from an audio-visual database and a text database, outputting the corresponding target information. This enables various tasks such as natural language understanding, audio-visual querying, and text querying, thereby improving the model's efficiency. Furthermore, since a single target model can perform multiple types of query tasks, there is no need to set up additional models to implement the same type of task, thus reducing the storage space occupied by the model.

[0084] This disclosure provides an information query method, such as...Figure 1 As shown, the method includes:

[0085] S100: In response to the query information input by the user, input the query information into the target model to obtain the first intent information, content information and the first text vector.

[0086] S200. Based on the first intent information, determine the corresponding target database, which is an audio-visual database and / or a text database.

[0087] S300. Based on the content information, determine the target vector in the target database that matches the first text vector.

[0088] S400: Output the target information corresponding to the target vector.

[0089] In this embodiment, in response to user-input query information, the user needs to query target information. Since the target model can understand natural language to predict the user's thoughts, the query information is input into the target model to obtain first intent information, content information, and a first text vector. Based on the first intent information, the corresponding audio-visual database and / or text database is determined as the target database to identify the database containing the target information. Since multiple vectors in the target database may be related to the first text vector, the vectors are filtered based on the content information to determine the target vector in the target database that matches the first text vector, and the target information corresponding to the target vector is output. By using the target model to predict the user's thoughts and query target information from multiple types of databases, various types of query tasks can be achieved, thereby improving the model's efficiency. Simultaneously, because the target model can achieve multiple types of query tasks, the number of models required for deployment is reduced, thus reducing the storage space occupied by the model.

[0090] For example, the target model can be a multimodal model. The query information can be text, speech, gestures, etc. If the query information is not text, it needs to be converted to text before being input into the target model. The audio-visual database can be an image database, audio database, and / or video database, etc. The audio-visual database stores the audio-visual vectors corresponding to the audio-visual information. The text database stores the text vectors corresponding to the text information. The prediction network can be a prediction head, and the structure of the prediction head can be a multi-layer fully connected network or a capsule network.

[0091] For example, the first intent information reflects the matter the user intends to process and may include action information, file type information, etc. The content information reflects the specific content of the matter and may include time information, semantic information, etc. The first text vector may reflect all or part of the query information. For instance, when the query information is "photos of the baby eating lunch today," the first intent information is "image information," the content information is "lunch today" and "baby eating," and the first text vector is the text vector corresponding to "photos of the baby eating lunch today." When the query information is "definition of an intelligent agent," the first intent information is "text information," the content information is "intelligent agent" and "definition," and the first text vector is the text vector corresponding to "definition of an intelligent agent."

[0092] In one embodiment, such as Figure 2 As shown, in step S100, the query information is input into the target model to obtain the first intent information, content information, and first text vector, which are determined in the following way:

[0093] S110. Input the query information into the text encoder in the target model to obtain the first text vector.

[0094] S120. Input the first text vector into the prediction network in the target model to obtain the first intent information and content information.

[0095] In this embodiment, since it is necessary to determine the matching target vector based on the first text vector, the query information is input into the text encoder in the target model to obtain the first text vector. Because the prediction network cannot directly determine the user's intent and expressed content based on the query information, the first text vector is input into the prediction network in the target model to obtain first intent information and content information. By sequentially obtaining the first text vector, first intent information, and content information, the target model can understand the user's thoughts to query the matching target vector, thereby improving the reliability of information retrieval.

[0096] For example, after inputting the first text vector into the prediction network of the target model in step S120, the prediction network outputs sequence tags. The intent tag in the sequence tag contains the first intent information, and the slot tag contains the content information.

[0097] For example, after inputting the first text vector into the prediction network of the target model in step S120 to obtain the first intent information and content information, step S100, which involves inputting the query information into the target model to obtain the first intent information, content information, and the first text vector, further includes:

[0098] Based on the first intent information and content information, the first text vector is filtered to retain only the semantic information portion.

[0099] In one embodiment, the determination of the corresponding target database based on the first intent information in step S200 is made in the following manner:

[0100] If the initial intent information points to audio-visual information, the audio-visual database is determined as the target database.

[0101] If the initial intent information points to text information, the text database is determined as the target database.

[0102] If the initial intent information points to audio-visual information and text information, the audio-visual database and the text database are identified as the target databases.

[0103] In this embodiment, when the first intent information points to audio-visual information, the user needs to query the audio-visual information to determine the audio-visual database as the target database. When the first intent information points to text information, the user needs to query the text information to determine the text database as the target database. When the first intent information points to both audio-visual information and text information, the user needs to query both audio-visual information and text information to determine both the audio-visual database and the text database as target databases. By determining the corresponding target database based on the direction of the first intent information, the target vector corresponding to the target information can be queried from various types of databases, thereby improving the efficiency of the model.

[0104] For example, in step S300, determining the target vectors in the target database that match the first text vector based on the content information can be done by selecting at least a portion of the vectors in the target database based on the content information. The similarity between each selected vector and the first text vector is then determined. Vectors with high similarity to the first text vector are selected as target vectors. That is, the number of target vectors can be one or more.

[0105] For example, the target information output in step S400 corresponding to the target vector can be: determining the corresponding target information based on the target vector; displaying the target information on the screen; or playing the target information aloud.

[0106] In one embodiment, before determining the corresponding target database based on the first intent information in step S200, the information query method further includes:

[0107] Obtain the first audio-visual information and the first text information stored in the electronic device.

[0108] Based on the first audio-visual information, construct an audio-visual database.

[0109] Construct a text database based on the first text information.

[0110] In this embodiment, when a user performs information queries locally, they need to query information stored on the electronic device, retrieving the first audio-visual information and the first text information stored on the electronic device. Based on the first audio-visual information, an audio-visual database is constructed to form a local audio-visual database. Based on the first text information, a text database is constructed to form a local text database. By constructing the database using information stored on the electronic device, the target model can be deployed on the electronic device and quickly query target information, thereby improving the efficiency of information retrieval.

[0111] In one embodiment, the step of constructing the audio-visual database based on the first audio-visual information is determined in the following way:

[0112] The first audio-visual information is input into the audio-visual encoder in the target model to obtain the first audio-visual vector.

[0113] Construct an audio-visual database using the first audio-visual vector.

[0114] In this embodiment, an audio-visual database is constructed using the first audio-visual vector output by the audio-visual encoder in the target model. Target information can be queried simply by vector matching, thereby improving the efficiency of information retrieval.

[0115] In one embodiment, the step of constructing the text database based on the first text information is determined in the following way:

[0116] The first text information is input into the text encoder in the target model to obtain the second text vector.

[0117] A text database is constructed using the second text vector.

[0118] In this embodiment, a text database is constructed using the second text vector output by the text encoder in the target model. Target information can be queried simply by vector matching, thereby improving the efficiency of information retrieval.

[0119] In one embodiment, the first text information includes long text information.

[0120] In this embodiment, since the first text information includes long text information, the text database stores the second text vector corresponding to the long text information. During the information query process, the target model can perform the task of querying long text information, thereby improving the efficiency of model usage.

[0121] In one embodiment, the first text information includes short text information.

[0122] In this embodiment, since the first text information includes short text information, and the text database stores the second text vector corresponding to the short text information, the target model can perform the task of querying short text information during the information query process, thereby improving the efficiency of model usage.

[0123] For example, long text information can be text information with a word count exceeding a certain number, while short text information can be text information with a word count not exceeding a certain number. Long text information can be text information that explains nouns, phrases, etc., or text information that describes facts in detail, etc. Short text information can be text information such as nouns, phrases, etc., or text information that simply describes facts, etc.

[0124] In one embodiment, the first text information is monolingual text information.

[0125] In this embodiment, since the first text information is monolingual, the second text vector stored in the text database corresponds to only one language. Because the number of text vectors in the text database is small, the time required for information retrieval can be reduced, thereby improving the efficiency of information retrieval.

[0126] In one embodiment, the first text information is multilingual text information.

[0127] In this embodiment, since the first text information is multilingual, the second text vector stored in the text database corresponds to multiple languages. Because the text database contains a large number of text vectors, it is possible to query the corresponding target vector from text vectors in multiple languages, thereby improving the accuracy of information retrieval.

[0128] For example, monolingual text information refers to text information in one language, such as Chinese text information or English text information. Multilingual text information refers to text information in multiple languages, such as Chinese text information and English text information.

[0129] In one embodiment, such as Figure 3 As shown, before inputting the query information into the target model in step S100 to obtain the first intent information, content information, and first text vector, the information query method further includes:

[0130] S500: Obtain first audio-visual text pair information, first long and short text pair information, and second text information containing second intent information.

[0131] S600. Based on the first audio-visual text pair information and the first long and short text pair information, train the audio-visual encoder and text encoder in the first model to obtain the second model.

[0132] S700. Based on the second text information, train the prediction network in the second model to obtain the target model.

[0133] In this embodiment, first audio-visual text pairs, first long-short text pairs, and second text information containing second intent information are acquired to obtain first audio-visual text pairs matching audio-visual information and text information, first long-short text pairs matching long text information and short text information, and second text information expressed in natural language. Based on the first audio-visual text pairs and the first long-short text pairs, the audio-visual encoder and text encoder in the first model are trained to obtain a second model capable of performing audio-visual query tasks and long-short text query tasks. Based on the second text information, the prediction network in the second model is trained to obtain a target model capable of performing natural language understanding tasks. By training the audio-visual encoder, text encoder, and prediction network separately, the target model can perform various types of tasks such as natural language understanding, audio-visual query, and long-short text query, thereby improving the efficiency of model usage.

[0134] For example, the first model is the initial model, with a dual-tower structure, including a video encoder and a text encoder. The video encoder can be a structure-optimized VIT (Vision Transformer) model. The text encoder can be a structure-optimized BERT (Bidirectional Encoder Representation from Transformers) model. The first model may not include a prediction network; the prediction network can be added after the text encoder is obtained from the second model.

[0135] For example, the text information in the first audio-visual text pair may include long text information and / or short text information. When the text information in the first audio-visual text pair is long text information, a certain amount of abstract text information can be introduced, thereby improving the generalization of the text encoder.

[0136] For example, audio-visual text pair information refers to information containing paired audio-visual information and text information. For instance, image information of a dog walking and text information of "dog walking". Long-short text pair information refers to information containing paired long text information and short text information. For instance, long text information detailing the agent's definition and short text information of "agent definition". Second text information containing second intent information refers to text information with a directional intent. For instance, text information of "open the photo of yesterday's meal".

[0137] In one embodiment, the second model is determined by training the audio-visual encoder and text encoder in the first model based on the first audio-visual text pair information and the first long-short text pair information in step S600, as follows:

[0138] Construct a first loss function to match audio / video vectors and text vectors.

[0139] Construct a second loss function to match long text vectors with short text vectors.

[0140] Based on the first and second loss functions, construct the third loss function.

[0141] Based on the first audio-visual text pair information, the first long and short text pair information, and the third loss function, the audio-visual encoder and text encoder in the first model are trained to obtain the second model.

[0142] In this embodiment, since video / audio query and long / short text query are two independent tasks, separate loss functions need to be constructed. A first loss function is constructed to match video / audio vectors with text vectors, and a second loss function is constructed to match long text vectors with short text vectors, enabling the second model to find matching video / audio vectors and text vectors, as well as long text vectors and short text vectors. Based on the first and second loss functions, a third loss function is constructed to jointly train the first model. Based on the first video / audio text pair information, the first long / short text pair information, and the third loss function, the video / audio encoder and text encoder in the first model are jointly trained to obtain the second model. By constructing the third loss function to jointly train the first model, only one text encoder needs to be set in the first model, and the trained second model can achieve matching of multiple types of vectors, thereby improving the efficiency of model training.

[0143] For example, the first loss function constructed in the above steps to match the audio / video vectors and the text vectors is expressed by the following formula:

[0144]

[0145] in, Let f represent the first loss function, I represent the set of video / audio vectors, T represent the set of text vectors corresponding to the set of video / audio vectors, i represent the video / audio vectors in the set of video / audio vectors, t represent the text vectors corresponding to the video / audio vectors, and f θ (i,t) represents the similarity between an audio / video vector and a positive sample text vector. This represents the similarity between an audio / video vector and a negative sample text vector. This represents the similarity between a text vector and a negative sample video / audio vector. The first loss function includes losses in two directions: from video / audio vector to text vector and from text vector to video / audio vector.

[0146] For example, the second loss function constructed in the above steps to match long text vectors and short text vectors is expressed by the following formula:

[0147]

[0148] in, Let f represent the second loss function, Q represent the set of short text vectors, P represent the set of long text vectors corresponding to the short text vector set, q represent the short text vectors in the short text vector set, p represent the long text vectors corresponding to the short text vectors, and f represent the second loss function. θ (q,p) represents the similarity between a long text vector and a short text vector of a positive sample. This represents the similarity between a short text vector and a negative sample long text vector. The first loss function represents the similarity between a long text vector and a negative sample short text vector. The second loss function includes losses in two directions: from short text vector to long text vector and from long text vector to short text vector.

[0149] For example, the third loss function, constructed based on the first and second loss functions in the above steps, is determined by the following formula:

[0150]

[0151] in, Let α represent the third loss function, and let α represent the first balancing coefficient that balances the loss between audio-visual text and long and short text, where 0 < α < 1.

[0152] In one embodiment, such as Figure 4 As shown, in the above steps, the audio-visual encoder and text encoder in the first model are trained based on the first audio-visual text pair information, the first long and short text pair information, and the third loss function. The second model is determined in the following way:

[0153] The audio and text information in the first audio-visual text pair are input into the audio encoder and text encoder in the first model, respectively, to obtain the second audio-visual vector and the third text vector.

[0154] The first long and short text pair information is input into the text encoder in the first model to obtain the fourth text vector corresponding to the long text information and the fifth text vector corresponding to the short text information.

[0155] The second model is obtained by training the audio encoder and text encoder in the first model based on the second audio-visual vector, the third text vector, the fourth text vector, the fifth text vector, and the third loss function.

[0156] In this embodiment, since the first loss function is constructed based on audio-visual vectors and text vectors, the audio-visual information and text information in the first audio-visual-text pair are respectively input into the audio-visual encoder and text encoder in the first model to obtain the second audio-visual vector and the third text vector. Since the second loss function is constructed based on long text vectors and short text vectors, the first long-short text pair information is input into the text encoder in the first model to obtain the fourth text vector and the fifth text vector. Based on the second audio-visual vector, the third text vector, the fourth text vector, the fifth text vector, and the third loss function, the audio-visual encoder and text encoder in the first model are jointly trained in a contrastive learning manner to obtain the second model. By converting the first audio-visual-text pair information and the first long-short text pair information into vectors and then jointly training the first model to obtain the second model, the second model can perform matching of audio-visual vectors and text vectors, as well as matching of long text vectors and short text vectors, thereby improving the efficiency of model training.

[0157] For example, I1~I N Represents the second audio-visual vector, T1~T N T represents the third and fifth text vectors. s1 ~T sN This represents the fourth text vector. The audio / video vector output by the audio / video encoder and the text vector output by the text encoder can both be 1*512 matrices.

[0158] For example, the above steps of training the audio encoder and text encoder in the first model based on the second audio-visual vector, the third text vector, the fourth text vector, the fifth text vector, and the third loss function to obtain the second model can be performed by training the audio encoder and text encoder in the first model based on the second audio-visual vector, the third text vector, and the third loss function, and by training the text encoder in the first model based on the fourth text vector, the fifth text vector, and the third loss function.

[0159] In one embodiment, the target model is determined by training the prediction network in the second model based on the second text information in step S700 as follows:

[0160] Construct a fourth loss function to match predicted information with true information.

[0161] Based on the text encoder in the second model, the prediction network in the second model is trained according to the second text information and the fourth loss function to obtain the target model.

[0162] In this embodiment, a fourth loss function is constructed to match predicted information with real information, enabling the target model to predict the matching real information. Based on the text encoder in the second model, the prediction network in the second model is trained according to the second text information and the fourth loss function to obtain the target model. By constructing the fourth loss function to train the prediction network in the second model, the target model can understand natural language to perform video and audio queries and text queries, thereby improving the model training effect.

[0163] For example, the fourth loss function constructed in the above steps to match the predicted information with the true information is expressed by the following formula:

[0164]

[0165] in, Let g represent the fourth loss function. i y represents the true intent of the i-th second text message. i This represents the predicted intent of the i-th second text message. This represents the actual value of the j-th slot in the i-th second text message. Let N represent the predicted value of the i-th second text information in the j-th slot, N represent the total number of second text information, and M represent the total number of slots.

[0166] For example, in the above steps, the text encoder in the second model is used to train the prediction network in the second model based on the second text information and the fourth loss function, and the target model is determined in the following way:

[0167] The second text information is input into the text encoder in the second model to obtain the sixth text vector.

[0168] Using the text encoder in the second model as the backbone network, the prediction network in the second model is trained based on the sixth text vector and the fourth loss function to obtain the target model.

[0169] For example, in the above steps, the text encoder in the second model is used as the backbone network, and the prediction network in the second model is trained according to the sixth text vector and the fourth loss function to obtain the target model. In this process, the parameters of the text encoder in the second model are kept unchanged, and only the prediction network in the second model is trained.

[0170] For example, after inputting the sixth text vector into the prediction network of the second model, the prediction network outputs sequence labels. For instance, if the second text information is "Find a photo of walking the dog yesterday," inputting the converted sixth text vector into the prediction network yields the sequence labels [C, O, O, O, B_T, I_T, B_S, I_S, O, O, O]. Here, C in the intent label represents the second intent information, B_T in the slot label represents the first character of the time slot in the time information, I_T represents the middle character of the time slot, B_S represents the first character of the semantic slot in the semantic information, I_S represents the middle character of the semantic slot, and 0 represents an empty label. The second intent information in the intent label is "photo," the time information in the time slot is "yesterday," and the semantic information in the semantic slot is "walking the dog."

[0171] In one embodiment, such as Figure 5 As shown, the text encoder in the second model is a monolingual text encoder. Before training the prediction network in the second model based on the second text information in step S700 to obtain the target model, the information query method further includes:

[0172] S610, Obtain the first multilingual text pair information.

[0173] S620. Construct a fifth loss function to match text vectors from different languages.

[0174] S630. Based on the information of the first multilingual text pair and the fifth loss function, the translation model is trained using the text encoder in the second model to obtain the multilingual text encoder.

[0175] S640, a multilingual text encoder is used instead of a monolingual text encoder as the text encoder in the second model.

[0176] In this embodiment, since the text encoder in the second model can only match monolingual text information and cannot match multilingual text information, some text information in other languages cannot be effectively queried. The first multilingual text pair information is obtained to obtain text pair information in which two or more languages are matched. A fifth loss function for matching text vectors in different languages is constructed, so that the trained second model can find and match text vectors in different languages. Based on the first multilingual text pair information and the fifth loss function, the translation model is trained with the text encoder in the second model as the basis, and the trained translation model is used as the multilingual text encoder. The multilingual text encoder is used to replace the monolingual text encoder as the text encoder in the second model, so that the second model can query information in multiple languages. By training the monolingual text encoder in the second model into a multilingual text encoder, the second model can query information in multiple languages, thereby improving the effect of model training. At the same time, since the text vectors of the multilingual text encoder are aligned to the vector space of the monolingual text encoder, the alignment of the vector spaces of the multilingual text encoder and the audio-visual encoder is indirectly achieved without retraining, thereby improving the efficiency of model training.

[0177] Exemplarily, the multilingual text pair information refers to text pair information in which text information in one language and text information in another language are paired and matched. For example, the Chinese text information of "The dog walks on the grass" and the English text information of "Dog walks on the grass". One language in the first multilingual text pair information is a language that the monolingual text encoder can recognize (such as Chinese), and the other or other languages that are matched are languages that the monolingual text encoder cannot recognize (such as English). The first multilingual text pair information can be information on the Internet, or the text information in the first audio-visual text pair information and the first long and short text pair information and their translated text information.

[0178] Exemplarily, the construction of the fifth loss function for matching text vectors in different languages in step S620 is determined by the following formula:

[0179]

[0180] where denotes the fifth loss function, T Mi denotes the text vector corresponding to the i-th text information in one language in the first multilingual text pair information, T i denotes the text vector corresponding to the i-th text information in another language in the first multilingual text pair information, N denotes the total number of the first multilingual text pair information, and the training loss is the mean square error (MSE) loss.

[0181] Exemplarily, such asFigure 6 As shown, in step S630, the translation model is trained using the text encoder in the second model based on the information from the first multilingual text pair and the fifth loss function. The multilingual text encoder is determined in the following way:

[0182] The first language text information and the second language text information in the first multilingual text pair information are respectively input into the monolingual text encoder in the second model and the translation model that serves as the multilingual text encoder, to obtain the seventh text vector and the eighth text vector.

[0183] Based on the seventh text vector, the eighth text vector, and the fifth loss function, the translation model is trained using the text encoder in the second model to obtain a multilingual text encoder.

[0184] For example, T S1 ~T SN T represents the seventh text vector. M1 ~T MN This represents the eighth text vector.

[0185] For example, the step above, which involves training the translation model using the text encoder in the second model based on the seventh text vector, the eighth text vector, and the fifth loss function to obtain a multilingual text encoder, can be achieved by using the text encoder in the second model as the teacher model and keeping the parameters unchanged, and training the translation model with a Bert structure as the student model based on the seventh text vector, the eighth text vector, and the fifth loss function to obtain a multilingual text encoder.

[0186] In one embodiment, such as Figure 7 As shown, after training the prediction network in the second model based on the second text information in step S700 to obtain the target model, the information query method further includes:

[0187] S800: Obtain the second audio-visual text pair information, the second long and short text pair information, and the third text information containing the third intent information.

[0188] S810. Construct a sixth loss function based on the third loss function used to train the first model and the fourth loss function used to train the second model.

[0189] S820. Based on the second audio-visual text pair information, the second long and short text pair information, the third text information, and the sixth loss function, train the audio-visual encoder, text encoder, and prediction network in the target model to obtain the trained target model.

[0190] In this embodiment, after training the audio / video encoder, text encoder, and prediction network separately, parameter fine-tuning is required to improve the query performance of the target model. Second audio / video text pair information, second long / short text pair information, and third text information containing third intent information are acquired to calibrate for functions such as natural language understanding, audio / video query, and text query. A sixth loss function is constructed based on the third loss function used to train the first model and the fourth loss function used to train the second model, making the sixth loss function correspond to functions such as natural language understanding, audio / video query, and text query. The audio / video encoder, text encoder, and prediction network in the target model are trained based on the second audio / video text pair information, the second long / short text pair information, the third text information, and the sixth loss function to obtain the trained target model. By fine-tuning the target model after obtaining it, the target model can accurately identify the user's intent and query the corresponding target information, thereby improving the reliability of model training.

[0191] For example, the amount of the second audio-visual text pair, the second long-short text pair, and the third text information can be the same or different. If the audio-visual information in the second audio-visual text pair is image information, the image information can be long image information and short image information, and the text information in the second audio-visual text pair can be text information in multiple languages. The long and short text information in the second long-short text pair can be text information in multiple languages, and may include text information as negative samples. The third text information can be text information in multiple languages. In addition to using the second audio-visual text pair, the second long-short text pair, and the third text information containing the third intent information for training the target model, the second multilingual text pair can also be used for training the target model.

[0192] For example, in step S810, the sixth loss function is constructed based on the third loss function used to train the first model and the fourth loss function used to train the second model, using the following formula:

[0193]

[0194] in, Let represent the sixth loss function, and β represent the second balance coefficient, where 0 < β < 1.

[0195] For example, such as Figure 8 As shown, in step S820, the audio-visual encoder, text encoder, and prediction network in the target model are trained based on the second audio-visual text pair information, the second long-short text pair information, the third text information, and the sixth loss function. The trained target model is determined in the following way:

[0196] The audio and text information in the second audio-visual-text pair are input into the audio encoder and text encoder in the target model, respectively, to obtain the third audio-visual vector and the ninth text vector.

[0197] The second long and short text pair information is input into the text encoder in the target model to obtain the tenth and eleventh text vectors.

[0198] The third text information is input into the text encoder in the target model to obtain the twelfth text vector.

[0199] Based on the third audio-visual vector, the ninth text vector, the tenth text vector, the eleventh text vector, the twelfth text vector, and the sixth loss function, the audio-visual encoder, the text encoder, and the prediction network in the target model are trained to obtain the trained target model.

[0200] For example, T I1 ~T IN T represents the third audio-visual vector. m1 ~T mN This represents the ninth text vector.

[0201] For example, the above steps of training the audio-visual encoder, text encoder, and prediction network in the target model based on the third audio-visual vector, the ninth text vector, the tenth text vector, the eleventh text vector, the twelfth text vector, and the sixth loss function to obtain the trained target model can be: training the audio-visual encoder and text encoder in the target model based on the third audio-visual vector, the ninth text vector, and the sixth loss function; training the text encoder in the target model based on the tenth text vector, the eleventh text vector, and the sixth loss function; and training the prediction network in the target model based on the twelfth text vector and the sixth loss function.

[0202] For example, such as Figure 9As shown, the electronic device obtains stored preset image information and first text information through clients such as photo albums, notes, recording, document management, and SMS. The preset image information is input into the image encoder of the target model to obtain preset image vectors. An image database is constructed using these preset image vectors. The first text information is input into the text encoder of the target model to obtain first text vectors. A text database is constructed using these first text vectors. When a user needs to query target information, they interact with clients such as global search, photo albums, notes, recording, document management, and SMS on the electronic device, inputting query information. The client inputs the query information into the target model, and the target model outputs the first text vector, first intent information, and content information. Based on the first intent information, one or more of the image database and text database are identified as the target database. Based on the content information, the target vector matching the first text vector in the target database is determined, and the target information corresponding to the target vector is output.

[0203] This disclosure provides an information query method, such as... Figure 10 As shown, the method includes:

[0204] S900: Obtain first image-text pair information, second image-text pair information, first long-short text pair information, second long-short text pair information, second text information, third text information, and first multilingual text pair information.

[0205] S910. Construct the first loss function and the second loss function.

[0206] S920. Based on the first loss function and the second loss function, construct the third loss function.

[0207] S930. Based on the first image-text pair information, the first long-short text pair information, and the third loss function, train the image encoder and text encoder in the first model to obtain the second model.

[0208] S940. Construct the fifth loss function.

[0209] S950. Based on the information from the first multilingual text pair and the fifth loss function, the translation model is trained using the text encoder in the second model to obtain the multilingual text encoder.

[0210] S960 replaces the monolingual text encoder with a multilingual text encoder as the text encoder in the second model.

[0211] S970. Construct the fourth loss function.

[0212] S980. Based on the text encoder in the second model, the prediction network in the second model is trained according to the second text information and the fourth loss function to obtain the target model.

[0213] S990. Based on the third and fourth loss functions, construct the sixth loss function.

[0214] S1000. Based on the second image-text pair information, the second long-short text pair information, the third text information, and the sixth loss function, train the image encoder, text encoder, and prediction network in the target model to obtain the trained target model.

[0215] S1010: Obtain preset image information and first text information stored in the electronic device.

[0216] S1020. Construct an image database and a text database based on preset image information and first text information, respectively.

[0217] S1030. In response to the query information input by the user, the query information is input into the text encoder in the target model to obtain the first text vector.

[0218] S1040. Input the first text vector into the prediction network in the target model to obtain the first intent information and content information.

[0219] S1050. Based on the first intent information, determine the corresponding target database.

[0220] S1060. Based on the content information, determine the target vector in the target database that matches the first text vector.

[0221] S1070, Output the target information corresponding to the target vector.

[0222] In this embodiment, first image-text pair information, second image-text pair information, first long-short text pair information, second long-short text pair information, second text information containing second intent information, third text information containing third intent information, and first multilingual text pair information are acquired to obtain various information for model training. A first loss function for matching image vectors and text vectors and a second loss function for matching long text vectors and short text vectors are constructed, and a third loss function for joint training is constructed based on the first and second loss functions. Based on the first image-text pair information, the first long-short text pair information, and the third loss function, the image encoder and text encoder in the first model are trained to obtain a second model capable of image and text queries. A fifth loss function for matching text vectors of different languages ​​is constructed to achieve multilingual information queries. Based on the first multilingual text pair information and the fifth loss function, the translation model is trained using the text encoder in the second model to obtain a multilingual text encoder. The multilingual text encoder replaces the monolingual text encoder as the text encoder in the second model to obtain a second model capable of multilingual queries. A fourth loss function is constructed to match predicted and real information, thereby achieving natural language understanding. Based on the text encoder in the second model, the prediction network in the second model is trained according to the second text information and the fourth loss function to obtain a target model capable of achieving natural language understanding. A sixth loss function is constructed based on the third and fourth loss functions to fine-tune the target model. The image encoder, text encoder, and prediction network in the target model are trained according to the second image-text pair information, the second long-short text pair information, the third text information, and the sixth loss function to obtain the trained target model. Preset image information and first text information stored in the electronic device are acquired, and an image database and a text database are constructed according to the preset image information and the first text information, respectively. In response to the user's query information, the query information is input into the text encoder in the target model to obtain a first text vector. The first text vector is input into the prediction network in the target model to obtain first intent information and content information to reflect the user's thoughts. The corresponding target database is determined according to the direction of the intent information. According to the content information, the target vector matching the first text vector in the target database is determined, and the target information corresponding to the target vector is output. By using a target model to predict user thoughts and query target information from various types of databases, diverse query tasks can be performed, thereby improving model efficiency. Furthermore, because the target model can handle multiple query tasks, the number of models required for deployment is reduced, thus decreasing the storage space occupied by the models.

[0223] In one exemplary embodiment, an information query device is provided for implementing the above-described method. (Reference) Figure 11 As shown, the information query device may include an input module 100, a first determining module 150, a second determining module 200, and an output module 250. During the implementation of the above method,

[0224] The input module 100 is configured to respond to user input query information by inputting the query information into the target model to obtain first intent information, content information and first text vector.

[0225] The first determining module 150 is configured to determine the corresponding target database based on the first intent information, wherein the target database is an audio-visual database and / or a text database.

[0226] The second determining module 200 is configured to determine, based on content information, a target vector in the target database that matches the first text vector.

[0227] Output module 250 is configured to output target information corresponding to the target vector.

[0228] In one exemplary embodiment, an information query device is provided, wherein an input module 100 is configured to:

[0229] The query information is input into the text encoder in the target model to obtain the first text vector.

[0230] The first text vector is input into the prediction network of the target model to obtain the first intent information and content information.

[0231] In one exemplary embodiment, an information query apparatus is provided, wherein a first determining module 150 is configured to:

[0232] If the initial intent information points to audio-visual information, the audio-visual database is determined as the target database.

[0233] If the initial intent information points to text information, the text database is determined as the target database.

[0234] If the initial intent information points to audio-visual information and text information, the audio-visual database and the text database are identified as the target databases.

[0235] In one exemplary embodiment, an information query device is provided, the device further comprising:

[0236] The acquisition module is configured to acquire first audio-visual information and first text information stored in the electronic device.

[0237] The building module is configured to build an audio-visual database based on the first audio-visual information.

[0238] Construct a text database based on the first text information.

[0239] In one exemplary embodiment, an information query apparatus is provided, wherein a construction module is configured to:

[0240] The first audio-visual information is input into the audio-visual encoder in the target model to obtain the first audio-visual vector.

[0241] Construct an audio-visual database using the first audio-visual vector.

[0242] In one exemplary embodiment, an information query apparatus is provided, wherein a construction module is configured to:

[0243] The first text information is input into the text encoder in the target model to obtain the second text vector.

[0244] A text database is constructed using the second text vector.

[0245] In one exemplary embodiment, an information query apparatus is provided, wherein the acquisition module is configured to:

[0246] Obtain the first audio-visual text pair information, the first long and short text pair information, and the second text information containing the second intent information.

[0247] In one exemplary embodiment, an information query device is provided, the device further comprising:

[0248] The training module is configured to train the audio encoder and text encoder in the first model based on the first audio-visual text pair information and the first long and short text pair information, so as to obtain the second model.

[0249] Based on the second text information, the prediction network in the second model is trained to obtain the target model.

[0250] In one exemplary embodiment, an information query apparatus is provided, wherein a training module is configured to:

[0251] Construct a first loss function to match audio / video vectors and text vectors.

[0252] Construct a second loss function to match long text vectors with short text vectors.

[0253] Based on the first and second loss functions, construct the third loss function.

[0254] Based on the first audio-visual text pair information, the first long and short text pair information, and the third loss function, the audio-visual encoder and text encoder in the first model are trained to obtain the second model.

[0255] In one exemplary embodiment, an information query apparatus is provided, wherein a training module is configured to:

[0256] The audio and text information in the first audio-visual text pair are input into the audio encoder and text encoder in the first model, respectively, to obtain the second audio-visual vector and the third text vector.

[0257] The first long and short text pair information is input into the text encoder in the first model to obtain the fourth text vector corresponding to the long text information and the fifth text vector corresponding to the short text information.

[0258] The second model is obtained by training the audio encoder and text encoder in the first model based on the second audio-visual vector, the third text vector, the fourth text vector, the fifth text vector, and the third loss function.

[0259] In one exemplary embodiment, an information query apparatus is provided, wherein a training module is configured to:

[0260] Construct a fourth loss function to match predicted information with true information.

[0261] Based on the text encoder in the second model, the prediction network in the second model is trained according to the second text information and the fourth loss function to obtain the target model.

[0262] In one exemplary embodiment, an information query apparatus is provided, wherein the acquisition module is configured to:

[0263] Obtain the first multilingual text pair information.

[0264] In one exemplary embodiment, an information query apparatus is provided, wherein a training module is configured to:

[0265] Construct a fifth loss function to match text vectors from different languages.

[0266] Based on the information from the first multilingual text pair and the fifth loss function, the translation model is trained using the text encoder in the second model to obtain the multilingual text encoder.

[0267] A multilingual text encoder is used instead of a monolingual text encoder as the text encoder in the second model.

[0268] In one exemplary embodiment, an information query apparatus is provided, wherein the acquisition module is configured to:

[0269] Acquire the second audio-visual text pair information, the second long and short text pair information, and the third text information containing the third intent information.

[0270] In one exemplary embodiment, an information query apparatus is provided, wherein a training module is configured to:

[0271] A sixth loss function is constructed based on the third loss function used to train the first model and the fourth loss function used to train the second model.

[0272] Based on the second audio-visual text pair information, the second long and short text pair information, the third text information, and the sixth loss function, the audio-visual encoder, text encoder, and prediction network in the target model are trained to obtain the trained target model.

[0273] In one exemplary embodiment, an electronic device is provided, such as a mobile phone, a laptop computer, a tablet computer, and a wearable device.

[0274] refer to Figure 12 As shown, the electronic device 400 may include one or more of the following components: processing component 402, memory 404, power supply component 406, multimedia component 408, audio component 410, input / output (I / O) interface 412, sensor component 414, and communication component 416.

[0275] Processing component 402 typically controls the overall operation of electronic device 400, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 402 may include one or more processors 420 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 402 may include one or more modules to facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402.

[0276] Memory 404 is configured to store various types of data to support the operation of electronic device 400. Examples of this data include instructions for any application or method operating on electronic device 400, contact data, phonebook data, messages, pictures, videos, etc. Memory 404 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0277] Power supply component 406 provides power to various components of electronic device 400. Power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 400.

[0278] Multimedia component 408 includes a screen that provides an output interface between electronic device 400 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 408 includes a front-facing camera module and / or a rear-facing camera module. When electronic device 400 is in an operating mode, such as shooting mode or video mode, the front-facing camera module and / or rear-facing camera module may receive external multimedia data. Each front-facing camera module and rear-facing camera module may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0279] Audio component 410 is configured to output and / or input audio signals. For example, audio component 410 includes a microphone (MIC) configured to receive external audio signals when electronic device 400 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 404 or transmitted via communication component 416. In some embodiments, audio component 410 also includes a speaker for outputting audio signals.

[0280] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0281] Sensor assembly 414 includes one or more sensors for providing state assessments of various aspects of electronic device 400. For example, sensor assembly 414 may detect the on / off state of electronic device 400, the relative positioning of components such as the display and keypad of electronic device 400, changes in position of electronic device 400 or a component of electronic device 400, the presence or absence of user contact with electronic device 400, orientation or acceleration / deceleration of electronic device 400, and temperature changes of electronic device 400. Sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 414 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0282] Communication component 416 is configured to facilitate wired or wireless communication between electronic device 400 and other terminals. Electronic device 400 can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 416 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 416 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0283] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing terminals (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods shown in the above embodiments or combinations thereof.

[0284] In one exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, which can be executed by a processor 420 of an electronic device 400 to perform the methods shown in the embodiments or combinations thereof. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage terminal, etc. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is able to perform the methods shown in the embodiments or combinations thereof.

[0285] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0286] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0287] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0288] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0289] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An information retrieval method, characterized in that, The information query method includes: In response to the query information input by the user, the query information is input into the target model to obtain first intent information, content information and first text vector; Based on the first intent information, a corresponding target database is determined, wherein the target database is an audio-visual database and / or a text database; Based on the content information, determine the target vector in the target database that matches the first text vector; Output the target information corresponding to the target vector.

2. The information query method according to claim 1, characterized in that, The step of inputting the query information into the target model to obtain first intent information, content information, and a first text vector includes: The query information is input into the text encoder in the target model to obtain the first text vector; The first text vector is input into the prediction network in the target model to obtain the first intent information and the content information.

3. The information query method according to claim 1, characterized in that, The step of determining the corresponding target database based on the first intent information includes: If the first intent information points to audio-visual information, the audio-visual database is determined to be the target database; If the first intent information points to text information, the text database is determined to be the target database; If the first intent information points to audio-visual information and text information, the audio-visual database and the text database are determined to be the target database.

4. The information query method according to claim 1, characterized in that, Before determining the corresponding target database based on the first intent information, the information query method further includes: Acquire the first audio-visual information and the first text information stored in the electronic device; Based on the first audio-visual information, construct the audio-visual database; Based on the first text information, construct the text database.

5. The information query method according to claim 4, characterized in that, The step of constructing the audio-visual database based on the first audio-visual information includes: The first audio-visual information is input into the audio-visual encoder in the target model to obtain the first audio-visual vector; The audio-visual database is constructed using the first audio-visual vector; and / or, The step of constructing the text database based on the first text information includes: The first text information is input into the text encoder in the target model to obtain the second text vector; The text database is constructed using the second text vector.

6. The information query method according to claim 4, characterized in that, The first text information includes long text information and / or short text information; and / or, the first text information is monolingual text information or multilingual text information.

7. The information query method according to any one of claims 1 to 6, characterized in that, Before inputting the query information into the target model to obtain the first intent information, content information, and first text vector, the information query method further includes: Acquire the first audio-visual text pair information, the first long and short text pair information, and the second text information containing the second intent information; Based on the first audio-visual text pair information and the first long and short text pair information, the audio-visual encoder and text encoder in the first model are trained to obtain the second model; Based on the second text information, the prediction network in the second model is trained to obtain the target model.

8. The information query method according to claim 7, characterized in that, The step of training the audio-visual encoder and text encoder in the first model based on the first audio-visual text pair information and the first long-short text pair information to obtain the second model includes: Construct a first loss function to match audio / video vectors and text vectors; Construct a second loss function to match long text vectors and short text vectors; Based on the first loss function and the second loss function, a third loss function is constructed; Based on the first audio-visual text pair information, the first long and short text pair information, and the third loss function, the audio-visual encoder and the text encoder in the first model are trained to obtain the second model.

9. The information query method according to claim 8, characterized in that, The step of training the audio-visual encoder and the text encoder in the first model based on the first audio-visual text pair information, the first long and short text pair information, and the third loss function to obtain the second model includes: The audio-visual information and text information in the first audio-visual text pair information are respectively input into the audio-visual encoder and the text encoder in the first model to obtain the second audio-visual vector and the third text vector; The first long and short text pair information is input into the text encoder in the first model to obtain the fourth text vector corresponding to the long text information and the fifth text vector corresponding to the short text information. The second model is obtained by training the audio-visual encoder and the text encoder in the first model based on the second audio-visual vector, the third text vector, the fourth text vector, the fifth text vector, and the third loss function.

10. The information query method according to claim 7, characterized in that, The step of training the prediction network in the second model based on the second text information to obtain the target model includes: Construct a fourth loss function to match predicted information with real information; Based on the text encoder in the second model, the prediction network in the second model is trained according to the second text information and the fourth loss function to obtain the target model.

11. The information query method according to claim 7, characterized in that, The text encoder in the second model is a monolingual text encoder; before training the prediction network in the second model based on the second text information to obtain the target model, the information query method further includes: Obtain the first multilingual text pair information; Construct a fifth loss function to match text vectors from different languages; Based on the first multilingual text pair information and the fifth loss function, the translation model is trained using the text encoder in the second model to obtain a multilingual text encoder; The multilingual text encoder is used instead of the monolingual text encoder as the text encoder in the second model.

12. The information query method according to claim 7, characterized in that, After training the prediction network in the second model based on the second text information to obtain the target model, the information query method further includes: Acquire the second audio-visual text pair information, the second long and short text pair information, and the third text information containing the third intent information; A sixth loss function is constructed based on the third loss function used to train the first model and the fourth loss function used to train the second model; Based on the second audio-visual text pair information, the second long and short text pair information, the third text information, and the sixth loss function, the audio-visual encoder, the text encoder, and the prediction network in the target model are trained to obtain the trained target model.

13. An information query device, characterized in that, The information query device includes: An input module is configured to respond to user-inputted query information by inputting the query information into a target model to obtain first intent information, content information, and a first text vector. A first determining module is configured to determine a corresponding target database based on the first intent information, wherein the target database is an audio-visual database and / or a text database. The second determining module is configured to determine, based on the content information, a target vector in the target database that matches the first text vector; An output module is configured to output target information corresponding to the target vector.

14. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the information query method as described in any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the terminal, the terminal is able to perform the information query method as described in any one of claims 1 to 12.

Images