Query method, device, server, medium and product based on multimodal information

Abstract

The present application discloses a query method, device, server, medium and product based on multimodal information, which can be applied to the financial field or other fields. After obtaining multimodal information, if the multimodal information includes video, multiple frames of target video images are obtained from the video, and the target video images include the actions performed by the target object on the electronic device; if the multimodal information includes voice information, a target voice segment is obtained from the voice information, and the target voice segment is the voice uttered by the target object on the electronic device; query is performed based on the target multimodal information, because the target multimodal information includes at least two of the multiple frames of target video images, target voice segments and text information, and does not include actions and voices that are not performed on the electronic device, so the target knowledge points obtained based on the target multimodal information are more accurate, that is, the target knowledge points obtained based on the query of the target multimodal information are the hit results expected by the target object.

Description

Technical Field

[0001] The present application relates to the field of information query technology, and more specifically, to a query method, device, server, medium and product based on multimodal information. Background Art

[0002] With the development of natural language processing technology, intelligent query, such as intelligent question and answer, is becoming more and more common, for example, intelligent robots in banks. At present, intelligent query has higher requirements on the multimodality of information, so query based on multimodal information has emerged, and multimodal information includes: video, voice and / or text.

[0003] Currently, the hit results obtained by querying based on multimodal information are inconsistent with the results expected by users. Summary of the invention

[0004] In view of this, the present application provides a query method, device, server, medium and product based on multimodal information to overcome the problem that the hit results obtained by querying based on multimodal information are inconsistent with the results expected by the user.

[0005] To achieve the above objectives, this application provides the following technical solutions:

[0006] According to a first aspect of an embodiment of the present disclosure, a query method based on multimodal information is provided, comprising:

[0007] determining a target object to interact with the electronic device;

[0008] Acquire multimodal information of the target object, the multimodal information comprising: at least two of: a video containing an action performed by the target object, voice information of the target object, and text information input by the target object in the electronic device;

[0009] If the multimodal information includes the video, acquiring multiple frames of target video images from the video, the target video images including actions performed by the target object on the electronic device;

[0010] If the multimodal information includes the voice information, obtaining a target voice segment from the voice information, where the target voice segment is a voice uttered by the target object to the electronic device;

[0011] Query target knowledge points matching target multimodal information from a knowledge base, wherein the target multimodal information includes at least two of the multiple frames of target video images, the target voice clips, and the text information, and the knowledge base includes multiple knowledge points, each of which includes at least two of the knowledge of video modality, the knowledge of voice modality, and the knowledge of text modality.

[0012] According to a second aspect of an embodiment of the present disclosure, a query device based on multimodal information is provided, including:

[0013] A first determination module, used to determine a target object to interact with the electronic device;

[0014] A first acquisition module is used to acquire multimodal information of the target object, wherein the multimodal information includes at least two of: a video containing an action performed by the target object, voice information of the target object, and text information input by the target object in the electronic device;

[0015] A second acquisition module is configured to acquire a plurality of frames of target video images from the video if the multimodal information includes the video, wherein the target video images include actions performed by the target object on the electronic device;

[0016] a third acquisition module, configured to acquire a target voice segment from the voice information if the multimodal information includes the voice information, wherein the target voice segment is a voice uttered by the target object to the electronic device;

[0017] A query module is used to query target knowledge points matching target multimodal information from a knowledge base, wherein the target multimodal information includes at least two of the multiple frames of target video images, the target voice segments, and the text information, and the knowledge base includes multiple knowledge points, each of which includes at least two of the knowledge of video modality, the knowledge of voice modality, and the knowledge of text modality.

[0018] According to a third aspect of an embodiment of the present disclosure, a server is provided, comprising:

[0019] processor;

[0020] a memory for storing instructions executable by the processor;

[0021] The processor is configured to execute the instructions to implement the query method based on multimodal information as described in the first aspect.

[0022] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of a server, the server is enabled to execute the query method based on multimodal information as described in the first aspect.

[0023] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, which can be directly loaded into an internal memory of a computer. The memory is the memory included in the server described in the third aspect above, and contains software code. After the computer program is loaded and executed, it can implement the query method based on multimodal information as described in the first aspect.

[0024] It can be known from the above technical solutions that in the query method based on multimodal information provided by the present application, a target object interacting with an electronic device is determined; multimodal information of the target object is obtained, and the multimodal information includes: a video containing actions performed by the target object, voice information of the target object, and at least two of text information input by the target object in the electronic device; if the multimodal information includes a video, multiple frames of target video images are obtained from the video, and the target video images contain actions performed by the target object on the electronic device; if the multimodal information includes voice information, a target voice segment is obtained from the voice information, and the target voice segment is the voice emitted by the target object on the electronic device; and target knowledge points matching the target multimodal information are queried from the knowledge base, and the target multimodal information includes at least two of multiple frames of target video images, target voice segments, and text information. Since the target multimodal information includes actions and voices performed on the electronic device, but does not include actions and voices that are not performed on the electronic device, the target knowledge points obtained based on the target multimodal information are more accurate, that is, the target knowledge points obtained based on the query of the target multimodal information are the hit results expected by the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0026] Figure 1 A structural diagram of the hardware architecture involved in the embodiments of the present application;

[0027] Figure 2 A flowchart of a query method based on multimodal information provided in an embodiment of the present application;

[0028] Figure 3 A structural diagram of an implementation method of the action anti-interference model provided in an embodiment of the present application;

[0029] Figure 4 is a block diagram of a query device based on multimodal information according to an exemplary embodiment;

[0030] Figure 5 It is a block diagram of a device for executing a query method based on multimodal information according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] The embodiments of the present application provide a query method, device, server, medium and product based on multimodal information. Before introducing the technical solution provided by the embodiments of the present application, the hardware architecture involved in the present application is first described.

[0033] like Figure 1 , which is a structural diagram of the hardware architecture involved in an embodiment of the present application, the hardware architecture includes: an electronic device 11 and a server 12.

[0034] Exemplarily, the electronic device 11 can be any electronic product that can interact with a user through one or more methods such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, for example, a mobile phone, a tablet computer, a PDA, a personal computer, a wearable device, a smart TV, a robot, an ATM machine, a smart device in a bank, etc.

[0035] Exemplarily, the electronic device 11 may be a bank robot; exemplary, the electronic device 11 may be a computer at a bank counter.

[0036] Exemplarily, the server 12 may be a single server, or a server cluster consisting of multiple servers, or a cloud computing service center.

[0037] In an optional implementation, the hardware architecture further includes one or more cameras 13 , which can be used to capture a video of a target object interacting with the electronic device 11 .

[0038] For example, if the electronic device 11 is a bank robot, the camera can be deployed in the electronic device 11, or in a business hall.

[0039] For example, if the electronic device 11 is a bank robot, the target object may be a user who needs business consultation at the bank; if the electronic device 11 is a computer at a bank counter, the target object may be a bank teller.

[0040] During the communication between the target object and the electronic device 11 , the target object will output multimodal information. Exemplarily, the multimodal information includes but is not limited to at least two of video, voice information and text information.

[0041] In an optional implementation, a video including actions performed by the target object may be captured by a camera included in the electronic device 11 and / or a camera 13 deployed at a corresponding location.

[0042] In an optional implementation, the voice information emitted by the target object may be collected by a voice collection device (eg, a microphone) in the electronic device 11 .

[0043] In an optional implementation, the target object can input text information in the electronic device 11 .

[0044] Exemplarily, the electronic device 11 or the camera 13 can send the captured video to the server 12, and the electronic device 11 can send text information to the server 12, so that the server 12 executes the query method based on multimodal information provided in the embodiment of the present application.

[0045] Exemplarily, the electronic device may execute the query method based on multimodal information provided in the embodiments of the present application.

[0046] It is understandable that the text information input by the target object in the electronic device 11 is text information related to this round of inquiry, but the actions performed by the target object contained in the video are not necessarily for this round of inquiry, and the voice clips contained in the voice information are not necessarily for this round of inquiry.

[0047] The following example is used to illustrate. During the interaction between the target object and the electronic device 11, the target object sends a voice message "How do I fill out the loan application form?" and the action performed is "pointing with a finger to the loan application form that you are holding." Suddenly, another user asks the target object where the restroom is. At this time, the target object may point in the direction of the restroom and say "Turn left and you will be there." At this time, the video containing the action of "pointing with a finger in the direction of the restroom" is not executed for the electronic device, and the voice clip "Turn left and you will be there" is not a voice issued by the electronic device. The video in the multimodal information obtained at this time is: a video clip including the action of "pointing with a finger to the loan application form that you are holding" and a video clip including the action of "pointing with a finger in the direction of the restroom"; the voice information in the multimodal information includes the voice clip of "How do I fill out the loan application form?" and the voice clip of "Turn left and you will be there."

[0048] In summary, the following situations may exist: the actions contained in the video in the multimodal information are not directed to the electronic device, and at least some of the voice segments in the voice information in the multimodal information are not sent to the electronic device. If a query is performed based on the multimodal information, the hit result should be inconsistent with the result expected by the target object.

[0049] To this end, an embodiment of the present application provides a query method based on multimodal information, which can obtain multiple frames of target video images from a video, and the target video images include the action performed by the target object on the electronic device (i.e., the action of "pointing with a finger to the loan application form held by oneself"); and can obtain the target voice segment from the voice information, and the target voice segment is the voice uttered by the target object to the electronic device (i.e., the voice segment of "Please tell me how to fill out the loan application form"). Therefore, the hit results obtained based on at least two queries among the multiple frames of target video images, the target voice segment and the text information are what the user expects.

[0050] Those skilled in the art should understand that the above-mentioned electronic devices and servers are only examples, and other existing or future electronic devices or servers that are applicable to the present disclosure should also be included in the protection scope of the present disclosure and are included here by reference.

[0051] The following describes the query method based on multimodal information provided by the embodiment of the present application in combination with the above-mentioned hardware architecture.

[0052] like Figure 2 As shown, it is a flowchart of a query method based on multimodal information provided in an embodiment of the present application. The method can be applied to the above-mentioned server 12 or electronic device 11. The method includes the following steps S21 to S25 during implementation.

[0053] Step S21: determining a target object for interacting with the electronic device.

[0054] Exemplarily, there are multiple ways to implement step S21, and the embodiments of the present application provide but are not limited to the following three.

[0055] The first implementation of step S21 includes: if a wake-up word for the electronic device is detected, determining that the object that issues the wake-up word is the target object.

[0056] The second implementation of step S21 includes: determining an object whose distance from the electronic device is less than or equal to a preset range as the target object.

[0057] Exemplarily, the preset range may be determined based on actual conditions, which will not be elaborated here.

[0058] A third implementation of step S21 includes: determining that the object whose sight is focused on the electronic device is the target object.

[0059] Step S22: Acquire multimodal information of the target object, wherein the multimodal information includes at least two of: a video containing an action performed by the target object, voice information of the target object, and text information input by the target object in the electronic device.

[0060] Step S23: If the multimodal information includes the video, a plurality of frames of target video images are obtained from the video, wherein the target video images include actions performed by the target object on the electronic device.

[0061] In an optional implementation, the multiple frames of target video images are all images in the video that contain actions performed by the target object on the electronic device.

[0062] It can be understood that a video is composed of multiple frames of continuous video images. Assuming that the video includes a total of M frames of video images, N frames of video images among the M frames of video images contain the actions performed by the target object on the electronic device, and MN frames of video images do not include the actions performed by the target object on the electronic device; at this time, the number of multiple frames of target video images can be N, and N is less than or equal to M.

[0063] In an optional implementation, multiple frames of target video images are extracted at preset time intervals from all images in the video that contain actions performed by the target object on the electronic device.

[0064] For example, the number of the multi-frame target video images is less than N.

[0065] It is understandable that since the human brain takes a certain amount of time to react, the actions of the target object contained in multiple consecutive frames of video images may be the same. Based on this, one frame of video image can be extracted from N frames of video images (these N frames of video images are all images containing the actions performed by the target object on the electronic device) at preset time intervals to obtain multiple frames of target video images.

[0066] Step S24: if the multimodal information includes the voice information, a target voice segment is obtained from the voice information, where the target voice segment is the voice uttered by the target object to the electronic device.

[0067] Step S25: Query target knowledge points that match the target multimodal information from the knowledge base, wherein the target multimodal information includes at least two of the multiple frames of target video images, the target voice clips, and the text information, and the knowledge base includes multiple knowledge points, each of which includes at least two of the knowledge of video modality, the knowledge of voice modality, and the knowledge of text modality.

[0068] In an optional implementation, the knowledge of video modality includes questions in video modality and answers in video modality; the knowledge of speech modality includes: questions in speech modality and answers in speech modality; the knowledge of text modality includes: questions in text modality and answers in text modality. For the same knowledge point, the knowledge of video modality, the knowledge of speech modality and the knowledge of text modality are different modal expressions of the same knowledge. Exemplarily, each knowledge point includes: [{questions in video modality, questions in speech modality, questions in text modality}: {answers in video modality, answers in speech modality, answers in text modality}], wherein questions in video modality correspond to answers in video modality, questions in speech modality correspond to answers in speech modality, and questions in text modality correspond to answers in text modality.

[0069] In an optional visual implementation, the knowledge of video modality includes answers in video modality; the knowledge of voice modality includes: answers in voice modality; the knowledge of text modality includes: answers in text modality. Exemplarily, each knowledge point includes: {answers in video modality, answers in voice modality, answers in text modality}.

[0070] In an optional implementation, one or more of the knowledge in video modality, the knowledge in voice modality, and the knowledge in text modality contained in the target knowledge point may be sent to the user.

[0071] In an optional implementation, for each knowledge point, step S25 includes the following steps A11 to A15.

[0072] Step A11: Calculate a first correlation between multiple frames of target video images and the knowledge of the video modality.

[0073] Exemplarily, if the knowledge of the video modality includes: questions of the video modality and answers of the video modality, then step A11 can calculate the first correlation between multiple frames of target video images and the questions of the video modality, or calculate the first correlation between multiple frames of target video images and the questions of the video modality and the answers of the video modality.

[0074] Step A12: Calculate the second correlation between the target speech segment and the knowledge of the speech modality.

[0075] Exemplarily, if the knowledge of speech modality includes: questions of speech modality and answers of speech modality, step A12 can calculate the second correlation between the target speech segment and the questions of speech modality, or calculate the second correlation between the target speech segment and the questions of speech modality and the answers of speech modality.

[0076] Step A13: Calculate the third relevance between the text information and the knowledge of the text modality.

[0077] Exemplarily, if the knowledge of text modality includes: questions in text modality and answers in text modality, step A13 can calculate the third correlation between the text information and the questions in text modality, or calculate the third correlation between the text information and the questions in text modality and the answers in text modality.

[0078] Exemplarily, any one of the first correlation, the second correlation and the third correlation may be the reciprocal of the cosine distance.

[0079] Step A14: Based on the first relevance, the second relevance, and the third relevance, calculate the relevance between the target multimodal information and the knowledge point.

[0080] Exemplarily, weights corresponding to multiple frames of target video images, target voice clips and text information can be pre-set. Assuming that multiple frames of target video images correspond to the first weight, the target voice clips correspond to the second weight, and the text information corresponds to the third weight, then the relevance of the target multimodal information to the knowledge point = first weight * first relevance + second weight * second relevance + third weight * third relevance.

[0081] Step A15: If the relevance is higher than or equal to a preset threshold, the knowledge point is determined to be a target knowledge point.

[0082] In the query method based on multimodal information provided in the embodiment of the present application, a target object interacting with an electronic device is determined; multimodal information of the target object is obtained, and the multimodal information includes: a video containing actions performed by the target object, voice information of the target object, and at least two of text information input by the target object in the electronic device; if the multimodal information includes video, multiple frames of target video images are obtained from the video, and the target video images contain actions performed by the target object on the electronic device; if the multimodal information includes voice information, a target voice segment is obtained from the voice information, and the target voice segment is the voice uttered by the target object on the electronic device; and target knowledge points matching the target multimodal information are queried from a knowledge base, and the target multimodal information includes at least two of multiple frames of target video images, target voice segments, and text information. Since the target multimodal information includes actions and voices performed on the electronic device, but does not include actions and voices that are not performed on the electronic device, the target knowledge points obtained based on the target multimodal information are more accurate.

[0083] Step S23 is described below. It is understandable that there are multiple implementations of step S23, and the present application embodiment provides but is not limited to the following two.

[0084] The first implementation method of step S23 includes the following steps B11 to B14.

[0085] Step B11: extracting from the video in sequence according to a preset time interval to obtain multiple frames of candidate video images.

[0086] Exemplarily, the preset time interval may be determined based on actual conditions and is not limited here. For example, the preset time interval may be 0.5 seconds.

[0087] Step B12: For each frame of the candidate video image, obtain a focus area image and the relative angle between the orientation of the target object's face and the electronic device when the target object performs the action included in the candidate video image, wherein the focus area image includes the object on which the target object's line of sight is focused when the target object performs the action in the candidate video image.

[0088] It is understandable that when the target object interacts with the electronic device, the target object's sight should be focused on the electronic device. However, if another user suddenly asks the target object during the process of the target object interacting with the electronic device, the target object may interact with the other user, and the target object's sight may be focused on the other user. Therefore, the target object's interaction object can be determined based on the focus area image. The focus area image includes the object on which the target object's sight is focused.

[0089] In an optional implementation, if the face of the target object and the display screen of the electronic device are located in the same horizontal plane, the method for obtaining the relative angle includes but is not limited to the following method, which includes step C11.

[0090] Step C11: determining that the angle between the normal vector of the plane where the face of the target object is located and the normal vector of the plane where the display screen of the electronic device is located is the relative angle.

[0091] The direction of the normal vector of the plane where the face is located is the same as the orientation direction of the face; the normal vector of the plane where the display screen is located is the same as the orientation direction of the display screen.

[0092] by Figure 1 For example, Figure 1 The solid arrow in is the normal vector of the plane where the face is located, and the dotted arrow is the normal vector of the plane where the display screen is located. Then the relative angle can be Figure 1α shown. It can be understood that when the face of the target object and the display screen of the electronic device are in the same horizontal plane, when the face of the target object is facing the display screen of the electronic device, the relative angle α is 180°, when the face of the target object is facing away from the display screen of the electronic device, the relative angle α is 0°, and when the face of the target object is looking to the left or right, the relative angle is less than 180°. Based on this, it can be considered that when the relative angle α is [180°-preset value, 180°+preset value], the face of the target object is facing the display screen of the electronic device. Exemplarily, the preset value can be determined based on the actual situation, for example, the preset value is less than 80°.

[0093] It is understandable that when the face of the target object and the display screen of the electronic device are not located at the same horizontal plane, for example, the face of the target object is higher than the display screen of the electronic device, or the face of the target object is lower than the display screen of the electronic device. In this case, the method for obtaining the relative angle includes but is not limited to the following method, which includes steps C21 to C24.

[0094] Step C21: construct a three-dimensional coordinate system, wherein the XOY plane of the three-dimensional coordinate system is parallel to the horizontal plane, the direction of the X axis of the XOY plane is a first direction, the first direction is the direction of the normal vector of the plane where the face of the target object is located and the normal vector of the plane where the display screen of the electronic device is located, and when the face of the target object is facing the display screen of the electronic device, the normal vector of the plane where the face of the target object is located is parallel to the normal vector of the plane where the face of the target object is located. The origin O of the three-dimensional coordinate system is the face of the target object.

[0095] like Figure 1 As shown, if the face of the target object and the display screen of the electronic device are located in the same horizontal plane, the first direction refers to the direction of the normal vector of the plane where the face of the target object is located when the angle between the normal vector of the plane where the face of the target object is located and the normal vector of the plane where the display screen of the electronic device is located is 180°.

[0096] The starting point of the normal vector of the plane where the face of the target object is located is the origin O of the three-dimensional coordinate system.

[0097] In an optional implementation, the target object and the electronic device may not move at all, and the three-dimensional coordinate system will not change.

[0098] In an optional implementation, the target object and the electronic device may be moving. For example, in a bank business hall, a robot and a customer are walking and chatting, and the origin of the three-dimensional coordinate system is constantly changing.

[0099] Step C22: Obtain the vertical angle γ between the normal vector of the plane where the face of the target object is located and the XOY plane, and the horizontal angle β between the normal vector of the plane where the face of the target object is located and the XOZ plane.

[0100] Step C23: Determine the angle η between the ray starting from the origin O and ending at the position of the display screen of the electronic device in the three-dimensional coordinates and the XOY plane.

[0101] If the electronic device does not have a display screen, step C23 is to determine the angle η between the ray starting from the origin O and ending at the position of the electronic device in the three-dimensional coordinates and the XOY plane.

[0102] Step C24: Determine the angle difference between the vertical angle γ and the angle η on the XOZ plane and the horizontal angle β as the relative angle.

[0103] Exemplarily, if the horizontal angle β is in [180°-preset value, 180°+preset value], and the angle difference is in [0°, 90°), it means that the face of the target object is facing the electronic device, otherwise, it means that the face of the target object is not facing the electronic device.

[0104] Step B13: For each frame of candidate video image, the candidate video image, the focus area image corresponding to the candidate video image, and the relative angle corresponding to the candidate video image are input into a pre-constructed action anti-interference model, and the action prediction result of the candidate video image is obtained through the action anti-interference model, wherein the action prediction result is whether the action contained in the candidate video image is an action executed for the electronic device or the action contained in the candidate video image is not an action executed for the electronic device.

[0105] It can be understood that the candidate video image contains actions performed by the target object, for example, whether the target object is making a call, so it is possible to determine whether the target object is doing actions unrelated to the electronic device based on the candidate video image, and thus it is possible to determine whether the actions contained in the candidate video image are directed to the electronic device.

[0106] Exemplarily, the motion anti-interference model is obtained by pre-training a machine learning model through multiple sample image sets. The machine learning model can adopt at least one of a neural network model, a support vector machine (SVM), a decision tree, a logistic regression model, Adaboost, XGboost, and a Transformer-Encoder model.

[0107] In an optional implementation, the sample image set includes a sample candidate video image, a focus area image corresponding to the sample candidate video image, and a relative angle corresponding to the sample candidate video image; when annotating the sample image set, you can refer to the above-mentioned description of the relative angle, the description of the focus area image, and the description of the candidate video image to manually determine the standard result corresponding to the sample candidate video image.

[0108] Train machine learning models based on a collection of sample images and human-annotated standard results.

[0109] The neural network model can be at least one of a convolutional neural network (CNN), an artificial neural network (ANN), a recurrent neural network (RNN), or a GRU (Gate Recurrent Unit) network.

[0110] Exemplarily, the machine learning model can be at least one of an attention-based deep model, a memory network-based deep model, a deep learning-based short text classification model, a recurrent neural network-based model, a convolutional neural network-based model, and a deep hybrid model of a Transformer-encoder-based classification model.

[0111] Among them, the short text classification model based on deep learning is a recurrent neural network (RNN) or a convolutional neural network (CNN) or a variant based on the recurrent neural network or the convolutional neural network.

[0112] For example, some simple domain adaptation modifications can be made on the pre-trained model to obtain a machine learning model.

[0113] Exemplarily, “simple domain adaptation” includes but is not limited to re-pre-training an already pre-trained model using large-scale unsupervised domain corpus, and / or compressing an already pre-trained model by model distillation.

[0114] In an embodiment of the present application, the candidate video image, the focus area image and the relative angle are used as inputs of the motion anti-interference model, which takes into account the action contained in the candidate video image, the object on which the target object's line of sight is focused, and whether the target object's face is facing the electronic device, so that the obtained motion prediction result is more accurate.

[0115] Step B14: Determine the candidate image whose action prediction result is that the action included in the candidate video image is an action executed for the electronic device as the target video image.

[0116] The second implementation method of step S23 includes the following steps B21 to B25.

[0117] Step B21: extracting from the video in sequence according to a preset time interval to obtain multiple frames of candidate video images.

[0118] Step B22: For each frame of the candidate video image, obtain a focus area image and the relative angle between the orientation of the target object's face and the electronic device when the target object performs the action included in the candidate video image, wherein the focus area image includes the object on which the target object's line of sight is focused when the target object performs the action in the candidate video image.

[0119] Step B23: for each frame of candidate video image, the candidate video image, the focus area image corresponding to the candidate video image, and the relative angle corresponding to the candidate video image are input into a pre-constructed motion interference prevention model.

[0120] The steps of executing the action interference prevention model include step B24 to step B26.

[0121] In order for those skilled in the art to better understand steps B24 to B26, the structure of the action anti-interference model is first described below.

[0122] like Figure 3 As shown, it is a structural diagram of an implementation method of the action anti-interference model provided in an embodiment of the present application. The action anti-interference model includes: a feature information extraction module 31, an action association detection module 32 and a result prediction module 33.

[0123] Exemplarily, the feature information extraction module 31 includes: a convolution layer and a pooling layer.

[0124] Exemplarily, the action association relationship detection module 32 may learn the correlation between actions contained in two temporally adjacent candidate video images.

[0125] Exemplarily, the action association relationship detection module 32 may be: a GRU network.

[0126] Step B24: extracting the features of the candidate video image, the focus area image and the relative angle through the feature information extraction module included in the motion interference prevention model to obtain a feature vector.

[0127] Step B25: determining the vector correlation between the feature vector and a previous feature vector through the action association relationship detection module included in the action anti-interference model, wherein the previous feature vector is a feature vector of a previous frame of candidate video image that is temporally adjacent to the candidate video image.

[0128] The method for obtaining the feature vector corresponding to the previous frame of video image that is adjacent in time is shown in step B24.

[0129] For example, multiple frames of candidate video images are sorted in time from early to late as: candidate video image 1, candidate video image 2, candidate video image 3, candidate video image 4; then the previous frame of candidate video image adjacent to candidate video image 3 in time is candidate video image 2.

[0130] Step B26: Determine the motion prediction result of the candidate video image based on the feature vector, the vector correlation and the motion prediction result of the previous frame of the candidate video image through the result prediction module included in the motion anti-interference model.

[0131] It can be understood that the greater the vector correlation, the greater the influence of the action prediction result of the previous frame candidate video image on the action prediction result of the currently predicted candidate video image; the smaller the vector correlation, the smaller the influence of the action prediction result of the previous frame candidate video image on the action prediction result of the currently predicted candidate video image.

[0132] It is understandable that, since human actions are continuous, if the action prediction result of the previous frame of the candidate video image is that the action contained in the candidate video image is not an action performed on the electronic device, then the action contained in the current frame of the candidate video image is highly likely not an action performed on the electronic device. Based on this, the obtained action prediction result is more accurate.

[0133] In an optional implementation, there are multiple implementations of step S24, and the embodiments of the present application provide but are not limited to the following two.

[0134] The first implementation method of step S24 includes the following steps D11 to D14.

[0135] Step D11: Obtain the text corresponding to the voice information.

[0136] Step D12: Acquire the voice information contained in the previous round of inquiry made by the target object to the electronic device, and the reply text contained in the previous round of inquiry replied by the electronic device.

[0137] In the embodiment of the present application, when the target object asks a question and the electronic device responds to the question, it is called a round of inquiry.

[0138] Step D13: Input the voice information mentioned in step D11, the text, the voice information contained in the previous round of inquiries, and the reply text into a pre-built semantic relevance detection model, and obtain the voice prediction result of the voice information mentioned in step D11 through the semantic relevance detection model, and the voice prediction result is that the voice information mentioned in step D11 is irrelevant to the previous round of inquiries, or the voice information mentioned in step D11 is related to the previous round of inquiries.

[0139] Exemplarily, the semantic relevance detection model is obtained by pre-training a machine learning model through multiple sample information sets. The machine learning model can adopt at least one of a neural network model, a support vector machine (SVM), a decision tree, a logistic regression model, Adaboost, XGboost, and a Transformer-Encoder model.

[0140] In an optional implementation, the sample information set includes sample voice information, text corresponding to the sample voice information, voice information included in the previous round of inquiries made by the target object to the electronic device, and reply text of the electronic device replying to the previous round of inquiries. When marking the sample information set, the relevance of the sample voice information in the sample information set to the voice information and reply text included in the previous round of inquiries is manually determined, and a standard result is manually obtained.

[0141] Train machine learning models based on sample information sets and manually annotated standard results.

[0142] The neural network model can be at least one of a convolutional neural network (CNN), an artificial neural network (ANN), a recurrent neural network (RNN), or a GRU (Gate Recurrent Unit) network.

[0143] Exemplarily, the machine learning model can be at least one of an attention-based deep model, a memory network-based deep model, a deep learning-based short text classification model, a recurrent neural network-based model, a convolutional neural network-based model, and a deep hybrid model of a Transformer-encoder-based classification model.

[0144] Among them, the short text classification model based on deep learning is a recurrent neural network (RNN) or a convolutional neural network (CNN) or a variant based on the recurrent neural network or the convolutional neural network.

[0145] For example, some simple domain adaptation modifications can be made on the pre-trained model to obtain a machine learning model.

[0146] Exemplarily, “simple domain adaptation” includes but is not limited to re-pre-training an already pre-trained model using large-scale unsupervised domain corpus, and / or compressing an already pre-trained model by model distillation.

[0147] Step D14: If the prediction result of the voice information mentioned in step D11 is related to the previous round of inquiry, determine that the voice information mentioned in step D11 is the target voice segment.

[0148] It is understandable that, in the voice information mentioned in step D11, only some voice segments may not be directed to the electronic device, while some voice segments are directed to the electronic device. In view of this, an embodiment of the present application provides a second implementation method of step S24.

[0149] The second implementation method of step S24 includes the following steps D21 to D25.

[0150] Step D21: Acquire multiple voice segments from the voice information.

[0151] Exemplarily, the time length of each voice segment may be the same, and the multiple voice segments are obtained from the voice information in chronological order.

[0152] For example, it is understandable that the target object usually pauses when switching from one topic (for example, telling others the direction of the restroom) to another topic (continuing to interact with the electronic device), so the voice information can be detected to contain pauses greater than or equal to a preset duration, and the voice information can be divided into multiple voice segments with pauses as intervals.

[0153] Step D22: Obtain texts corresponding to the plurality of voice segments.

[0154] Exemplarily, the text corresponding to the voice segment can be obtained based on speech-to-text technology.

[0155] Step D23: Acquire the voice information contained in the previous round of inquiry made by the target object to the electronic device, and the reply text of the electronic device in the previous round of inquiry.

[0156] Step D24: For each of the voice segments, the voice segment, the text corresponding to the voice segment, the voice information contained in the previous round of inquiries, and the reply text are input into a pre-constructed semantic relevance detection model, and a voice prediction result of the voice segment is obtained through the semantic relevance detection model, wherein the voice prediction result is that the voice segment is irrelevant to the previous round of inquiries, or the voice segment is related to the previous round of inquiries.

[0157] Exemplarily, the timbre, intonation and other contents of the target object may be obtained through voice clips.

[0158] Step D25: Determine the speech segment that is related to the previous round of inquiry as the target speech segment.

[0159] The embodiment of the present application extracts the target voice information from the voice information, rather than judging the voice information as a whole. If the voice information is judged as a whole, if the voice information includes voice segments that are not targeted at the electronic device, the voice prediction result of the voice information is irrelevant to the previous round of inquiries. At this time, the multimodal information will lack the voice modality, and the accuracy of the query will be reduced when performing a query.

[0160] The method is described in detail in the embodiments disclosed in the above-mentioned application. The method of the application can be implemented by various forms of devices. Therefore, the application also discloses a device, and a specific embodiment is given below for detailed description.

[0161] Figure 4 is a block diagram of a query device based on multimodal information according to an exemplary embodiment. Figure 4 The device comprises: a first determination module 41, a first acquisition module 42, a second acquisition module 43, a third acquisition module 44 and a query module 45, wherein:

[0162] A first determining module 41, used to determine a target object to interact with the electronic device;

[0163] A first acquisition module 42 is used to acquire multimodal information of the target object, wherein the multimodal information includes at least two of: a video containing an action performed by the target object, voice information of the target object, and text information input by the target object in the electronic device;

[0164] A second acquisition module 43 is configured to acquire a plurality of frames of target video images from the video if the multimodal information includes the video, wherein the target video images include actions performed by the target object on the electronic device;

[0165] A third acquisition module 44 is configured to acquire a target voice segment from the voice information if the multimodal information includes the voice information, wherein the target voice segment is a voice emitted by the target object to the electronic device;

[0166] The query module 45 is used to query the target knowledge points matching the target multimodal information from the knowledge base, wherein the target multimodal information includes at least two of the multiple frames of target video images, the target voice segments and the text information, and the knowledge base includes multiple knowledge points, each of which includes at least two of the knowledge of video modality, the knowledge of voice modality and the knowledge of text modality.

[0167] In an optional implementation, the second acquisition module includes:

[0168] A first acquisition unit, used to extract from the video in sequence according to a preset time interval to obtain multiple frames of candidate video images;

[0169] A second acquisition unit is used to acquire, for each frame of the candidate video image, a focus area image and a relative angle between the orientation of the face of the target object and the electronic device when the target object performs an action included in the candidate video image, wherein the focus area image includes an object on which the target object's line of sight is focused when the target object performs the action in the candidate video image;

[0170] A third acquisition unit is used to input, for each frame of candidate video image, the candidate video image, the focus area image corresponding to the candidate video image, and the relative angle corresponding to the candidate video image into a pre-built action interference prevention model, and obtain an action prediction result of the candidate video image through the action interference prevention model, wherein the action prediction result is that the action included in the candidate video image is an action executed for the electronic device or the action included in the candidate video image is not an action executed for the electronic device;

[0171] The first determining unit is configured to determine, as the target video image, a candidate image whose action prediction result is that the action included in the candidate video image is an action executed for the electronic device.

[0172] In an optional implementation, the third obtaining unit includes:

[0173] An extraction subunit, configured to extract features of the candidate video image, the focus area image, and the relative angle through a feature information extraction module included in the motion interference prevention model to obtain a feature vector;

[0174] A first determination subunit is used to determine the vector correlation between the feature vector and a previous feature vector through an action association relationship detection module included in the action anti-interference model, wherein the previous feature vector is a feature vector of a previous frame of a candidate video image that is temporally adjacent to the candidate video image;

[0175] The second determination subunit is used to determine the action prediction result of the candidate video image based on the feature vector, the vector correlation and the action prediction result of the previous frame of the candidate video image through the result prediction module included in the action anti-interference model.

[0176] In an optional implementation, the third acquisition module includes:

[0177] A fourth acquisition unit, configured to acquire a plurality of voice segments from the voice information;

[0178] A fifth acquisition unit, used to acquire texts corresponding to the plurality of voice segments respectively;

[0179] A sixth acquisition unit, configured to acquire voice information contained in a previous round of inquiry made by the target object to the electronic device, and a reply text of the electronic device replied to the previous round of inquiry;

[0180] a seventh acquisition unit, for each of the voice segments, inputting the voice segment, the text corresponding to the voice segment, the voice information contained in the previous round of inquiry, and the reply text into a pre-constructed semantic relevance detection model, and acquiring a voice prediction result of the voice segment through the semantic relevance detection model, wherein the voice prediction result is that the voice segment is irrelevant to the previous round of inquiry, or that the voice segment is related to the previous round of inquiry;

[0181] The second determining unit is configured to determine the speech segment related to the previous round of inquiry as the target speech segment.

[0182] In an optional implementation, the query module includes:

[0183] A calculation unit, for calculating, for any knowledge point, a first correlation between the multiple frames of target video images and the knowledge of the video modality corresponding to the knowledge point, a second correlation between the target voice segment and the knowledge of the voice modality corresponding to the knowledge point, and a third correlation between the text information and the text morphology knowledge corresponding to the knowledge point;

[0184] an eighth acquisition unit, configured to obtain, for any knowledge point, a correlation between the target multimodal information and the knowledge point through the first correlation, the second correlation, and the third correlation, so as to obtain correlations corresponding to a plurality of knowledge points respectively;

[0185] The third determining unit is used to determine the knowledge point whose relevance is higher than or equal to a preset threshold as the target knowledge point.

[0186] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0187] Figure 5 It is a block diagram of a device for executing a query method based on multimodal information according to an exemplary embodiment. The device may be an electronic device 11 or a server 12.

[0188] The device includes but is not limited to: a processor 51 , a memory 52 , a network interface 53 , an I / O controller 54 and a communication bus 55 .

[0189] It should be noted that those skilled in the art can understand that Figure 5 The structure of the device shown in the figure does not constitute a limitation on the device, and the device may include Figure 5 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0190] Combine the following Figure 5 The following is a detailed introduction to each component of the device:

[0191] The processor 51 is the control center of the device, and uses various interfaces and lines to connect various parts of the entire device. By running or executing software programs and / or modules stored in the memory 52, and calling data stored in the memory 52, the processor 51 performs various functions of the device and processes data, thereby monitoring the device as a whole. The processor 51 may include one or more processing units; illustratively, the processor 51 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 51.

[0192] The processor 51 may be a central processing unit (CPU), or an application specific integrated circuit ASIC (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0193] The memory 52 may include a memory, such as a high-speed random access memory (RAM) 521 and a read-only memory (ROM) 522, and may also include a large-capacity storage device 523, such as at least one disk storage, etc. Of course, the device may also include hardware required for other services.

[0194] The memory 52 is used to store instructions executable by the processor 51. The processor 51 has the following functions: determining a target object to interact with the electronic device;

[0195] Acquire multimodal information of the target object, the multimodal information comprising: at least two of: a video containing an action performed by the target object, voice information of the target object, and text information input by the target object in the electronic device;

[0196] If the multimodal information includes the video, acquiring multiple frames of target video images from the video, the target video images including actions performed by the target object on the electronic device;

[0197] If the multimodal information includes the voice information, obtaining a target voice segment from the voice information, where the target voice segment is a voice uttered by the target object to the electronic device;

[0198] Query target knowledge points matching target multimodal information from a knowledge base, wherein the target multimodal information includes at least two of the multiple frames of target video images, the target voice clips, and the text information, and the knowledge base includes multiple knowledge points, each of which includes at least two of the knowledge of video modality, the knowledge of voice modality, and the knowledge of text modality.

[0199] A wired or wireless network interface 53 is configured to connect the device to a network.

[0200] The processor 51, the memory 52, the network interface 53 and the I / O controller 54 can be interconnected via a communication bus 55, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0201] In an exemplary embodiment, the apparatus may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned query method based on multimodal information.

[0202] In an exemplary embodiment, the present disclosure provides a storage medium including instructions, such as a memory 52 including instructions, and the instructions can be executed by a processor 51 of the device to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0203] In an exemplary embodiment, a computer-readable storage medium is also provided, which can be directly loaded into the internal memory of a computer, such as the above-mentioned memory 52, and contains software code. After the computer program is loaded and executed, it can implement the above-mentioned query method based on multimodal information.

[0204] In an exemplary embodiment, a computer program product is also provided, which can be directly loaded into the internal memory of a computer, such as the memory included in the server, and contains software code. After being loaded and executed by the computer, the computer program can implement the above-mentioned query method based on multimodal information.

[0205] The query method, device, server, medium and product based on multimodal information provided by the present invention can be used in the financial field or other fields, for example, can be used in query application scenarios in the financial field. Other fields are any fields other than the financial field, for example, the power field. The above is only an example and does not limit the application field of the query method, device, server, medium and product based on multimodal information provided by the present invention.

[0206] It should be noted that the features described in the various embodiments in this specification can be replaced or combined with each other. For the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0207] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0208] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0209] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A query method based on multimodal information, characterized in that: include: Determining a target object to interact with the electronic device; Acquire multimodal information of the target object, the multimodal information comprising: at least two of: a video containing an action performed by the target object, voice information of the target object, and text information input by the target object in the electronic device; If the multimodal information includes the video, extracting from the video in sequence according to a preset time interval to obtain multiple frames of candidate video images; For each frame of the candidate video image, obtaining a focus area image and a relative angle between the face orientation of the target object and the electronic device when the target object performs an action included in the candidate video image, wherein the focus area image includes an object on which the target object's line of sight is focused when the target object performs the action in the candidate video image; For each frame of candidate video image, the candidate video image, the focus area image corresponding to the candidate video image, and the relative angle corresponding to the candidate video image are input into a pre-built action interference prevention model, and an action prediction result of the candidate video image is obtained through the action interference prevention model, where the action prediction result is that the action included in the candidate video image is an action executed for the electronic device or the action included in the candidate video image is not an action executed for the electronic device; Determine the candidate image whose action prediction result is that the action included in the candidate video image is an action performed on the electronic device as a target video image, wherein the target video image includes the action performed by the target object on the electronic device; If the multimodal information includes the voice information, obtaining a target voice segment from the voice information, where the target voice segment is a voice uttered by the target object to the electronic device; Query target knowledge points matching target multimodal information from a knowledge base, wherein the target multimodal information includes multiple frames of target video images, the target voice clips, and at least two of the text information, the knowledge base includes multiple knowledge points, each of the knowledge points includes at least two of the knowledge of video modality, the knowledge of voice modality, and the knowledge of text modality, and the knowledge of video modality, the knowledge of voice modality, and the knowledge of text modality are different modal expressions for the same knowledge point.

2. The query method based on multimodal information according to claim 1, characterized in that: The step of obtaining the motion prediction result of the candidate video image by using the motion anti-interference model comprises: Extracting features of the candidate video image, the focus area image, and the relative angle through a feature information extraction module included in the motion interference prevention model to obtain a feature vector; Determining the vector correlation between the feature vector and a previous feature vector through an action association relationship detection module included in the action anti-interference model, wherein the previous feature vector is a feature vector of a previous frame of a candidate video image that is temporally adjacent to the candidate video image; The result prediction module included in the action anti-interference model determines the action prediction result of the candidate video image based on the feature vector, the vector correlation and the action prediction result of the previous frame of the candidate video image.

3. The query method based on multimodal information according to any one of claims 1 to 2, characterized in that: The step of obtaining a target voice segment from the voice information comprises: Acquire multiple voice segments from the voice information; Obtaining texts corresponding to the plurality of voice segments respectively; Acquire the voice information contained in the previous round of inquiry made by the target object to the electronic device, and the reply text contained in the previous round of inquiry replied by the electronic device; For each of the voice segments, the voice segment, the text corresponding to the voice segment, the voice information contained in the previous round of inquiry, and the reply text are input into a pre-constructed semantic relevance detection model, and a voice prediction result of the voice segment is obtained through the semantic relevance detection model, wherein the voice prediction result is that the voice segment is irrelevant to the previous round of inquiry, or the voice segment is related to the previous round of inquiry; The speech segment related to the previous round of inquiry according to the speech prediction result is determined as the target speech segment.

4. The query method based on multimodal information according to any one of claims 1 to 2, characterized in that: The step of searching the knowledge base for target knowledge points that match the target multimodal information includes: For any knowledge point, calculate the first correlation between the multiple frames of target video images and the knowledge of the video form corresponding to the knowledge point, calculate the second correlation between the target voice segment and the knowledge of the voice modality corresponding to the knowledge point, and calculate the third correlation between the text information and the text form knowledge corresponding to the knowledge point; For any knowledge point, obtaining the correlation between the target multimodal information and the knowledge point through the first correlation, the second correlation and the third correlation, so as to obtain the correlations corresponding to the plurality of knowledge points respectively; The knowledge points whose relevance is higher than or equal to a preset threshold are determined as the target knowledge points.

5. A query device based on multimodal information, characterized in that: include: A first determination module, used to determine a target object to interact with the electronic device; A first acquisition module is used to acquire multimodal information of the target object, wherein the multimodal information includes at least two of: a video containing an action performed by the target object, voice information of the target object, and text information input by the target object in the electronic device; A second acquisition module is used for extracting from the video in sequence according to a preset time interval to obtain multiple frames of candidate video images if the multimodal information includes the video; For each frame of the candidate video image, obtaining a focus area image and a relative angle between the face orientation of the target object and the electronic device when the target object performs an action included in the candidate video image, wherein the focus area image includes an object on which the target object's line of sight is focused when the target object performs the action in the candidate video image; For each frame of candidate video image, the candidate video image, the focus area image corresponding to the candidate video image, and the relative angle corresponding to the candidate video image are input into a pre-built action interference prevention model, and an action prediction result of the candidate video image is obtained through the action interference prevention model, where the action prediction result is that the action included in the candidate video image is an action executed for the electronic device or the action included in the candidate video image is not an action executed for the electronic device; Determine the candidate image whose action prediction result is that the action included in the candidate video image is an action performed on the electronic device as a target video image, wherein the target video image includes the action performed by the target object on the electronic device; a third acquisition module, configured to acquire a target voice segment from the voice information if the multimodal information includes the voice information, wherein the target voice segment is a voice uttered by the target object to the electronic device; A query module is used to query target knowledge points matching target multimodal information from a knowledge base, wherein the target multimodal information includes multiple frames of target video images, the target voice clips, and at least two of the text information, and the knowledge base includes multiple knowledge points, each of which includes at least two of the knowledge of video modality, the knowledge of voice modality, and the knowledge of text modality.

6. A server, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the query method based on multimodal information as described in any one of claims 1 to 4.

7. A computer-readable storage medium, when instructions in the storage medium are executed by a processor of a server, enables the server to execute the query method based on multimodal information as described in any one of claims 1 to 4.

8. A computer program product that can be directly loaded into the internal memory of a computer, wherein the memory is the memory contained in the server described in claim 6 and contains software code. After being loaded and executed by a computer, the computer program can implement the query method based on multimodal information as described in any one of claims 1 to 4.

Images