Man-machine interaction method and device based on multi-modal data and storage medium

Abstract

The invention relates to a man-machine interaction method and device based on multi-modal data and a storage medium. The multi-modal data man-machine interaction method comprises the following steps: acquiring multi-modal data and an interaction text; respectively extracting semantic features of each modal data in the multi-modal data to obtain first semantic features of each modal data; privacy features in the first semantic features of the modal data are filtered, and second semantic features of the modal data are obtained; and based on the interaction information and the second semantic feature of each modal data, obtaining response information of the multi-modal data responding to the interaction information. According to the man-machine interaction method based on the multi-modal data, filtering operation can be carried out on the multi-modal data and the interaction text, the response information is output, the user privacy exposure probability is reduced, and the user privacy security is protected.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a human-computer interaction method, device and storage medium based on multimodal data. Background Art

[0002] At present, artificial intelligence technology is developing rapidly and has been popularized in various fields of people's lives, including medical diagnosis, video analysis, smart home, smart city and smart security, etc. Artificial intelligence technology can better provide convenience for people's lives.

[0003] In the related technologies, human-computer interaction technology still has problems with image factual errors, cannot effectively handle complex scenes, can only process a single data input source, and cannot solve the problem of user privacy. For example, the camera will collect some redundant information or misunderstand the collected information. Therefore, the above problems cannot meet the user's requirements for the smooth experience and accuracy of human-computer interaction technology. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a human-computer interaction method, device and storage medium based on multimodal data.

[0005] According to a first aspect of an embodiment of the present disclosure, a human-computer interaction method based on multimodal data is provided, comprising: acquiring multimodal data and interactive text; respectively extracting semantic features of each modal data in the multimodal data to obtain first semantic features of each modal data; respectively filtering privacy features in the first semantic features of each modal data to obtain second semantic features of each modal data; and obtaining response information of the multimodal data in response to the interactive information based on the interactive information and the second semantic features of each modal data.

[0006] In one embodiment, the filtering of privacy features in the first semantic features of each modal data respectively includes: determining the similarity between the first semantic feature and the privacy feature in a privacy database, wherein the privacy database includes privacy features that map privacy data based on pre-training data; filtering out the first semantic feature with a portion having a similarity greater than a similarity threshold with the privacy feature, retaining the absolute value of the similarity of the processed first semantic feature to obtain a first data feature; filtering out features in the processed first semantic feature with a similarity greater than a preset threshold, and merging them with the first data feature into a second data feature; using the second data feature and the first semantic feature to filter out the portion having a similarity greater than a similarity threshold with the privacy feature, filtering out the portion with an absolute difference greater than a preset threshold, and outputting the second semantic feature.

[0007] In one embodiment, the method of obtaining response information of the multimodal data in response to the interaction information based on the interaction information and the second semantic features of each modal data includes: mapping the second semantic features of each modal data into features of the same dimension, and concatenating the features under the same dimension to obtain a third semantic feature; inputting the interaction information and the third semantic feature into a language model to obtain response information of the multimodal data in response to the interaction information.

[0008] In one embodiment, the step of inputting the interaction information and the third semantic feature into a language model to obtain response information of the multimodal data in response to the interaction information includes: inputting the third semantic feature into a language model to obtain first response information of the multimodal data in response to the interaction information; performing interactive text filtering on the interaction information and the first semantic feature based on the first response information, the first semantic feature and the interaction information to obtain filtered interaction information; and inputting the filtered interaction information and the third semantic feature into the language model to obtain second response information of the multimodal data in response to the interaction information.

[0009] In one embodiment, the interactive text filtering of the interactive information and the first semantic feature based on the first response information, the first semantic feature and the interactive information includes: performing word segmentation on the first response information and the interactive information respectively to obtain word segmentation features with the same length; performing a similarity table lookup on the word segmentation features to filter out word segmentation features whose similarity between the first response information and the interactive information is lower than a similarity threshold to obtain a keyword table, wherein the keyword table includes word segmentation features whose similarity between the first response information and the interactive information is greater than a preset threshold; performing text feature mapping on the keyword table to obtain a first mapping feature; filtering the first mapping feature and the first semantic feature to obtain filtered interactive information and the third semantic feature.

[0010] In one embodiment, the filtering of the first mapping feature and the first semantic feature to obtain filtered interaction information and the third semantic feature includes: performing convolution mapping on the first mapping feature to obtain a second mapping feature, and performing convolution mapping on the first semantic feature to obtain a third mapping feature; concatenating the second mapping feature and the third mapping feature to obtain a fourth mapping feature; sorting the fourth mapping feature in time sequence according to the time sequence correlation from strong to weak; screening out a set number of fourth mapping features whose sorting positions are from weak to strong; and determining the screened out fourth mapping features as the filtered interaction information and the third semantic feature.

[0011] In one implementation, the method further includes: performing privacy filtering on the response information, and outputting the privacy filtered response information.

[0012] According to a second aspect of an embodiment of the present disclosure, a human-computer interaction device based on multimodal data is provided, including: a receiving unit, used to obtain multimodal data and interactive text; an execution unit, used to extract semantic features of each modal data in the multimodal data respectively, to obtain a first semantic feature of each modal data; filtering privacy features in the first semantic features of each modal data respectively, to obtain a second semantic feature of each modal data; based on the interactive information and the second semantic features of each modal data, obtaining response information of the multimodal data in response to the interactive information.

[0013] In one embodiment, the execution unit filters the privacy features in the first semantic features of each modal data respectively in the following manner: determining the similarity between the first semantic feature and the privacy feature in a privacy database, wherein the privacy database includes privacy features that map privacy data based on pre-training data; filtering out the first semantic feature with a portion having a similarity greater than a similarity threshold with the privacy feature, retaining the absolute value of the similarity of the processed first semantic feature to obtain a first data feature; filtering out the features in the processed first semantic feature with a similarity greater than a preset threshold, and merging them with the first data feature into a second data feature; using the second data feature and the first semantic feature to filter out the portion having a similarity greater than a similarity threshold with the privacy feature, filtering out the portion with an absolute difference greater than a preset threshold, and outputting the second semantic feature.

[0014] In one embodiment, the execution unit obtains response information of the multimodal data in response to the interaction information based on the interaction information and the second semantic features of each modal data in the following manner: mapping the second semantic features of each modal data into features of the same dimension, and concatenating the features under the same dimension to obtain a third semantic feature; inputting the interaction information and the third semantic feature into a language model to obtain response information of the multimodal data in response to the interaction information.

[0015] In one embodiment, the execution unit inputs the interaction information and the third semantic feature into a language model in the following manner to obtain response information of the multimodal data in response to the interaction information: inputs the third semantic feature into a language model to obtain first response information of the multimodal data in response to the interaction information; based on the first response information, the first semantic feature and the interaction information, performs interactive text filtering on the interaction information and the first semantic feature to obtain filtered interaction information; inputs the filtered interaction information and the third semantic feature into the language model to obtain second response information of the multimodal data in response to the interaction information.

[0016] In one embodiment, the execution unit performs interactive text filtering on the interactive information and the first semantic feature based on the first response information, the first semantic feature and the interactive information in the following manner: respectively perform word segmentation on the first response information and the interactive information to obtain word segmentation features with the same length; perform a similarity table lookup on the word segmentation features to filter out word segmentation features whose similarity between the first response information and the interactive information is lower than a similarity threshold to obtain a keyword table, wherein the keyword table includes word segmentation features whose similarity between the first response information and the interactive information is greater than a preset threshold; perform text feature mapping on the keyword table to obtain a first mapping feature; filter the first mapping feature and the first semantic feature to obtain filtered interactive information and the third semantic feature.

[0017] In one embodiment, the execution unit filters the first mapping feature and the first semantic feature in the following manner to obtain filtered interaction information and the third semantic feature: convolution mapping is performed on the first mapping feature to obtain a second mapping feature, and convolution mapping is performed on the first semantic feature to obtain a third mapping feature; the second mapping feature and the third mapping feature are concatenated to obtain a fourth mapping feature; the fourth mapping feature is time-series sorted in a manner from strong to weak in terms of time correlation; a set number of fourth mapping features whose sorting positions are from weak to strong are screened out; and the screened out fourth mapping features are determined as the filtered interaction information and the third semantic feature.

[0018] In one implementation, the execution unit is further configured to: perform privacy filtering on the response information, and output the privacy filtered response information.

[0019] According to a third aspect of an embodiment of the present disclosure, a human-computer interaction device based on multimodal data is provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to: execute the human-computer interaction method based on multimodal data described in the first aspect or any one of the embodiments of the first aspect.

[0020] According to the fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the human-computer interaction method based on multimodal data described in the first aspect or any one embodiment of the first aspect.

[0021] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: after receiving multimodal data and interactive text, filtering operations are performed on the multimodal data and interactive text, and response information is output, thereby reducing the probability of user privacy exposure and protecting user privacy security.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0024] Figure 1 The figure is a flow chart of a human-computer interaction method of multimodal data according to an exemplary embodiment.

[0025] Figure 2 The present invention is a flowchart of a method for filtering a privacy feature in a first semantic feature according to an exemplary embodiment.

[0026] Figure 3 It is a schematic diagram of a privacy filtering method according to an exemplary embodiment.

[0027] Figure 4 It is a flowchart of a method for acquiring response information according to interaction information and a second semantic feature according to an exemplary embodiment.

[0028] Figure 5 It is a flowchart of a method for acquiring response information according to interaction information and a third semantic feature according to an exemplary embodiment.

[0029] Figure 6 The present invention is a flowchart of a method for interactive text filtering according to an exemplary embodiment.

[0030] Figure 7The invention is a flowchart showing a method of sorting and filtering first mapping features according to an exemplary embodiment.

[0031] Figure 8 The diagram is a schematic diagram of an interactive text filtering method according to an exemplary embodiment.

[0032] Fig. 9 The present invention is a flowchart of a method for outputting first processed data by a language model according to an exemplary embodiment.

[0033] Fig.10 It is a schematic diagram of a human-computer interaction method of multimodal data according to an exemplary embodiment.

[0034] Fig.11 The present invention is a block diagram of a human-computer interaction device for multimodal data according to an exemplary embodiment.

[0035] Fig.12 It is a block diagram of a device for human-computer interaction of multimodal data according to an exemplary embodiment. DETAILED DESCRIPTION

[0036] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0037] The human-computer interaction method for multimodal data provided by the embodiments of the present disclosure is applied to a terminal, which may be a mobile phone, a camera, a tablet computer, or the like.

[0038] In daily life, mobile phones, cameras and other terminals have multiple functions, and people use the functions of the terminals to meet their needs. Among them, some functions of the terminal involve human-computer interaction technology, and the application scenarios of functions involving human-computer interaction technology can be understood as: the terminal obtains data information such as video, images and / or audio from the outside world and obtains interactive information from the user, analyzes and processes the above data information, and finally outputs information to the user. In the related technology, the functions involving human-computer interaction technology in the terminal still have shortcomings such as: there are errors in the factuality of the image, it is impossible to effectively handle complex scenes, it can only process a single data input source, and it cannot solve user privacy issues.

[0039] For example, the process of human-computer interaction between a home surveillance camera and a user is used as an example. The home surveillance camera is placed in the user's home for monitoring, thereby protecting the user's life and property. The home surveillance camera monitors the user's home by collecting video content and audio content, and can receive the user's interactive information and make corresponding processing. In a possible scenario, the home surveillance camera monitors the user's home, but due to the hijacking of the interactive information by the offender, the home surveillance camera is misled to turn to the location where the user places the property. Since the home surveillance camera itself has no judgment ability for the monitoring screen, the offender can easily find the location where the user places the property and commit crimes.

[0040] For another example, the process of human-computer interaction between a smart car with an automatic driving function and a user is used as an example. The user drives the smart car and uses the automatic driving function to assist in driving, thereby reducing the user's driving fatigue and improving the user's driving pleasure. The automatic driving function identifies and analyzes roads, vehicles, pedestrians, etc. by collecting video content and audio content, and can receive the user's interactive information and make corresponding processing. In one possible scenario, the user turns on the automatic driving function to assist in driving, the user concentrates on driving forward, and the automatic driving function analyzes objects in the nearby scene. However, since the automatic driving function still has certain limitations in handling complex scenes, if a certain object is not detected, such as a turning vehicle, it is very likely to cause a traffic accident.

[0041] The above-mentioned human-computer interaction method for multimodal data reveals that the relevant technology still has errors in image factuality, cannot effectively handle complex scenes, can only process a single data input source, and cannot solve the problem of user privacy.

[0042] It is understandable that the embodiments of the present disclosure use home surveillance cameras and smart car assisted driving as examples for illustrative purposes only, but the human-computer interaction scenarios of the embodiments of the present disclosure are not limited thereto and may also be other application scenarios, such as assisting in the analysis and processing of medical images or assisting in the analysis and processing of urban traffic environments.

[0043] In view of this, the present disclosure provides a human-computer interaction method for multimodal data, in which an electronic device can filter redundant information from received data information and output text information to a user.

[0044] Figure 1 is a flowchart of a human-computer interaction method for multimodal data according to an exemplary embodiment. Figure 1 As shown, the human-computer interaction method of multimodal data includes the following steps S11 to S14, and each step will be introduced below.

[0045] In step S11, multimodal data and interactive text are obtained.

[0046] In the disclosed embodiments, the electronic device receives multimodal data and user interaction information from the outside world, such as the electronic device receives video content captured by a camera, the electronic device receives audio content transmitted by a sensor, and the electronic device receives interaction information from a user, etc.

[0047] For example, a home surveillance camera can obtain video and audio information of the user's home through cameras and sensors. In addition, the user can remotely interact with the home surveillance camera using an application that comes with the home surveillance camera. For example, if you say "Please look to the left for me", the home surveillance camera will turn its view to the left.

[0048] In step S12, semantic features of each modal data in the multimodal data are extracted respectively to obtain a first semantic feature of each modal data.

[0049] In the disclosed embodiment, multimodal data needs to be encoded by an encoder and mapped into a high-dimensional feature matrix before subsequent corresponding processing can be performed.

[0050] In step S13, the privacy features in the first semantic features of each modal data are filtered respectively to obtain the second semantic features of each modal data.

[0051] In the disclosed embodiment, the electronic device performs filtering operation on the redundant information according to the received first semantic feature, and then outputs the second semantic feature. The redundant information may be data information related to illegal crimes such as pornography, violence, and political sensitivity.

[0052] For example, when home surveillance cameras obtain video and audio information in a user's home through cameras and sensors, and this data and information contains data and information related to pornography, violence, political sensitivity, and other illegal crimes, these data and information can be eliminated.

[0053] In step S14, based on the interaction information and the second semantic features of each modal data, response information of the multimodal data in response to the interaction information is obtained.

[0054] In the disclosed embodiment, the electronic device provides response information to the user according to the received user interaction information and the second semantic feature.

[0055] In the disclosed embodiments, the human-computer interaction method using multimodal data can effectively filter the privacy of multimodal data and interactive text, reduce the probability of user privacy exposure, and protect user privacy security.

[0056] It can be understood that the operation of filtering privacy information in multimodal data and interactive data disclosed in the present invention has multiple links, including a privacy filtering link and an interactive text filtering link.

[0057] A method for filtering according to the privacy feature in the first semantic feature is described below.

[0058] Figure 2 is a flow chart showing a method for filtering a privacy feature in a first semantic feature according to an exemplary embodiment. Figure 2 As shown, the following steps S21-S24 are included.

[0059] In step S21, the similarity between the first semantic feature and the privacy features in the privacy database is determined, where the privacy database includes the privacy features that are mapped to the privacy data based on the pre-training data.

[0060] In the disclosed embodiment, the electronic device pre-builds a privacy text database, which contains a large amount of pre-stored text data such as privacy, sensitive, and dangerous content. These text data can be stored in the privacy text database in the form of basic text storage such as txt, pdf, tar, and zip.

[0061] In the embodiment of the present disclosure, the present disclosure does not limit the similarity measurement standard between the first semantic feature and the privacy feature in the privacy database, and cosine similarity can be used as the similarity measurement standard.

[0062] In the disclosed embodiment, the purpose of setting the privacy filtering link for the first semantic feature is to reduce the spread of pornographic, violent, politically sensitive and other information.

[0063] In step S22, the first semantic feature is filtered out of the part whose similarity with the privacy feature is greater than the similarity threshold, and the absolute value of the similarity of the processed first semantic feature is retained to obtain the first data feature.

[0064] In the disclosed embodiment, the first semantic feature is screened out for which the similarity between the first semantic feature and the privacy feature is greater than a similarity threshold. This is to effectively implement privacy filtering and remove privacy-sensitive information contained in the first semantic feature.

[0065] In step S23, the features with similarity higher than a preset threshold in the first semantic features after screening are selected and merged with the first data features into the second data features.

[0066] In the embodiment of the present disclosure, the purpose of merging the first data feature and the features whose similarity among the screened-out first semantic features is higher than a preset threshold into the second data feature is to ensure the effectiveness and integrity of privacy filtering.

[0067] In step S24, the second data feature and the first semantic feature are used to filter out the parts whose similarity with the privacy feature is greater than the similarity threshold, and the parts whose absolute difference is greater than the preset threshold are filtered out to output the second semantic feature.

[0068] In the disclosed embodiment, the second data feature and the first semantic feature are filtered by removing the parts whose similarity with the privacy feature is greater than a similarity threshold, in order to normalize the two sets of feature data and perform effective filtering.

[0069] Privacy filtering is mentioned in the embodiments of the present disclosure, and a privacy filtering method is described below.

[0070] Figure 3 FIG. 1 is a schematic diagram of a privacy filtering method according to an exemplary embodiment. Figure 3 As shown, the electronic device pre-builds a privacy text database, which contains a large amount of pre-stored text data such as privacy, sensitive, and dangerous content. The text data in the privacy text database is mapped into multi-dimensional text features through a pre-trained variable autoencoder (VAE) model.

[0071] It can be understood that the VAE model is a deep learning model that can achieve data dimensionality reduction and compression by learning and reconstructing input data. It can be widely used in data dimensionality reduction, generation models, probabilistic graphical models and other fields.

[0072] It is understandable that the reason for using a pre-trained VAE model to map text data is that private text data is just some content, not feature matrix information, and cannot be compared with the feature matrix (Embedding) mapped with image / video feature information and audio feature information. It is understandable that private text data can have multiple dimensions, that is, multiple attributes. For example, sensitive text data can have attributes such as sensitivity and text length.

[0073] The VAE model can be used to map private text data into multi-dimensional text features required by electronic devices, and the features can be mapped into a matrix of dimensions such as (B, M, 8192) through a linear layer, where M represents the length of a single group of text and can be set to 64. The mapped feature is the privacy seed library Embedding.

[0074] According to the privacy seed library Embedding, the privacy similarity is divided with the image / audio / text Embedding. The similarity of different text contents is calculated cyclically, and the similarity is compared for division. The part with a similarity exceeding 50%, for example, can be selected for deletion. The deleted feature part 3 is shown in box 3 in the figure. At the same time, the absolute value of the similarity of the data part with low similarity is retained as the new data feature 1, as shown in box 1 in the figure. For features with high similarity, they need to be screened out from the original input and merged with the absolute value of similarity into a new data feature 2, as shown in box 2 in the figure. In order to ensure the effectiveness and integrity of privacy filtering, the new feature 2 and the screened privacy are filtered. This part mainly performs absolute value comparison filtering, that is, the two sets of feature data are normalized, and the feature 4 whose absolute difference is greater than, for example, 0.5 is compared. As shown in box 4 in the figure, it is screened out again to obtain high confidence score filtering and used as output data.

[0075] In the disclosed embodiment, a filtering method based on a privacy database and a privacy seed library is adopted to filter the visual features (images and audio). In this step, data features such as "pornography, danger, and personal privacy" contained in the visual and auditory information can be effectively filtered out.

[0076] In the disclosed embodiment, effective filtering can be performed on preset information to ensure the security of output information.

[0077] Next, a method for obtaining response information according to the interaction information and the second semantic feature is described.

[0078] Figure 4 is a flow chart of a method for acquiring response information according to interaction information and second semantic features according to an exemplary embodiment. Figure 4 As shown, the following steps S41 to S42 are included, and each step will be introduced separately below.

[0079] In step S41, the second semantic features of each modal data are mapped to features of the same dimension, and the features under the same dimension are concatenated to obtain a third semantic feature.

[0080] In the disclosed embodiment, the second semantic features of each modal data are mapped and concatenated to obtain a third semantic feature.

[0081] In an implementation of a related technology, the electronic device maps the received multimodal data into Embeddings of the same latitude for splicing, and then performs privacy filtering.

[0082] It is understandable that, compared with the disclosed embodiment, the above implementation method performs privacy filtering on the fused multimodal data feature information, and the effect is not as good as that of the disclosed embodiment. It is understandable that after the fusion of multimodal data feature information, it contains information of multiple modes such as audio, picture / video. Assuming that there is privacy in the original picture but no privacy in the audio, fusing the multimodal data feature information and then performing privacy filtering increases the complexity of privacy filtering.

[0083] In step S42, the interaction information and the third semantic feature are input into a language model to obtain response information of the multimodal data in response to the interaction information.

[0084] In the disclosed embodiment, the interaction information and the third semantic feature are input into the language model, and the user can obtain response information.

[0085] In an implementation of a related technology, an electronic device maps multimodal data feature information to Embeddings of the same latitude for splicing, directly establishes a strong correlation mapping, and outputs text information. Compared with the embodiment of the present disclosure, the above implementation needs to consider more details, such as the inability to determine the features with practical significance between different modalities, and the uncertainty of how the modalities are associated. In the embodiment of the present disclosure, the electronic device maps multimodal data feature information to Embeddings of the same latitude for splicing and inputs them into a language model, and uses the language model to understand the relationship between different modal feature information, rather than establishing a strong correlation between different modal feature information, which can effectively explain the relationship between different modal feature information.

[0086] In the disclosed embodiment, the multimodal data feature information is subjected to corresponding privacy filtering respectively, the filtering effect is better and the process is not complicated.

[0087] A method for obtaining response information according to the interaction information and the third semantic feature is explained below.

[0088] Figure 5 is a flow chart of a method for acquiring response information according to interaction information and a third semantic feature according to an exemplary embodiment. Figure 5 As shown, the process includes the following steps S51 to S53, and each step will be introduced below.

[0089] In step S51, the third semantic feature is input into the language model to obtain the first response information of the multimodal data response interaction information.

[0090] In step S52, based on the first response information, the first semantic feature and the interaction information, interactive text filtering is performed on the interaction information and the first semantic feature to obtain filtered interaction information.

[0091] In the disclosed embodiment, the first response information output by the language model can be combined with the first semantic feature and the user's interaction information and filtered.

[0092] In step S53, the filtered interaction information and the third semantic feature are input into a language model to obtain second response information of the multimodal data response interaction information.

[0093] In the disclosed embodiment, the filtered interaction information and the third semantic feature can be re-input into the language model to obtain the second response information.

[0094] In the disclosed embodiment, interactive text filtering is performed on the interactive information and the first semantic feature based on the first response information, the first semantic feature and the interactive information, which can reduce the possibility of human-induced electronic devices to commit crimes, reduce the probability of user privacy exposure, and ensure the privacy security of users.

[0095] In the disclosed embodiment, the response information output by the language model can be combined with the remaining data as a basis to be re-input into the language model, thereby ensuring the long text continuous dialogue capability of the human-computer interaction method based on multimodal data.

[0096] Interactive text filtering is mentioned in the embodiments of the present disclosure, and a method for interactive text filtering is described below.

[0097] Figure 6 is a flow chart of a method for interactive text filtering according to an exemplary embodiment. Figure 6 As shown, the following steps are included: S61-S63.

[0098] In step S61, the first response information and the interactive information are segmented respectively to obtain segmentation features with the same length.

[0099] In the disclosed embodiment, the purpose of performing word segmentation operations on the response information output by the language model and the user's interactive information is to extract text data features.

[0100] In the disclosed embodiment, the answer information output by the language model includes the question content of all previous conversations, not just the question content of the previous round of conversations, in order to ensure the strong relevance of the question content. For example, the user asks "What was the weather like last week" in the first round of conversation, the user asks "What to eat today" in the second round of conversation, and the user asks "What is the weather like today" in the third round of conversation. The language model needs to analyze and understand the user's first three rounds of conversations, rather than just the third round of conversations, to ensure the strong relevance of the question content.

[0101] In step S62, a similarity table lookup is performed on the word segmentation features, and the word segmentation features whose similarity between the first response information and the interactive information is lower than the similarity threshold are screened out to obtain a keyword table, which includes the word segmentation features whose similarity between the first response information and the interactive information is higher than the preset threshold.

[0102] In the embodiment of the present disclosure, the present disclosure does not limit the similarity measurement standard between the first response information and the interaction information, and cosine similarity can be used as the similarity measurement standard.

[0103] In the disclosed embodiment, the keyword table is a table composed of words with high similarity in the response information and the interaction information. For example, if the response information contains words about last week's weather and the interaction information contains words about today's weather, the keyword table can include weather as a keyword.

[0104] In step S63, text feature mapping is performed on the keyword table to obtain a first mapping feature.

[0105] In the disclosed embodiment, after the keyword table and the first semantic feature are mapped and concatenated, they are sorted to generate a feature representation containing different temporal features.

[0106] In step S64, the first mapping feature and the first semantic feature are filtered to obtain filtered interaction information and a third semantic feature.

[0107] In the disclosed embodiment, the temporal correlation sorting can establish a strong correlation between the modal information and the interactive text. That is, a strong correlation is established between the content of the interactive text and the content actually captured by the electronic device. For example, if the user inputs "what clothes does a walking person wear" in the interactive text, and the content actually captured by the electronic device includes a person walking, a strong correlation can be established between the modal information and the interactive text.

[0108] In the disclosed embodiment, the purpose of filtering interactive text is to prevent users from artificially inducing electronic devices to do immoral or even illegal things. For example, using an application that is compatible with a home surveillance camera, you can remotely interact with the home surveillance camera, such as "Please help me check the bedroom". If the home surveillance camera detects that someone is changing clothes in the room, it will process the pornographic scenes in the video accordingly to prevent the privacy of the person captured by the surveillance screen from being leaked.

[0109] In the disclosed embodiment, the purpose of performing similarity table lookup includes: establishing the correlation between the previous round of dialogue and the new round of dialogue, outputting a keyword table, and sending key feature information to the language model; ensuring that each dialogue is relevant.

[0110] In the disclosed embodiments, interactive text filtering can reduce the possibility of human-induced electronic devices to commit crimes, reduce the probability of user privacy exposure, and ensure the privacy security of users.

[0111] Figure 7 is a flowchart showing a method of sorting and filtering first mapping features according to an exemplary embodiment. Figure 7 As shown, it includes steps S71 to S73.

[0112] In step S71, convolution mapping is performed on the first mapping feature to obtain a second mapping feature, and convolution mapping is performed on the first semantic feature to obtain a third mapping feature.

[0113] In the disclosed embodiment, the purpose of convolving the first mapping feature and the first semantic feature is to perform high-quality feature splicing.

[0114] In step S72, the second mapping feature and the third mapping feature are concatenated to obtain a fourth mapping feature.

[0115] In step S73, the fourth mapping features are time-sequentially sorted in a manner from strong to weak time-sequential correlation.

[0116] In the embodiment of the present disclosure, sorting the fourth mapping features according to the time series correlation is beneficial for the electronic device to select features with strong time series correlation.

[0117] In step S74, a set number of fourth mapping features whose sorting positions are from weak to strong are screened out.

[0118] In the disclosed embodiment, deleting features with weak temporal correlation is beneficial to ensuring the correlation of multiple rounds of dialogue.

[0119] In step S75, the eliminated fourth mapping feature is determined as the filtered interaction information and the third semantic feature.

[0120] In the disclosed embodiment, the benefit of sorting and filtering the first mapping features is to feed semantic features with strong temporal correlation into the language model, thereby ensuring that the response information output by the language model maintains a strong correlation with the user's previous multiple rounds of conversations.

[0121] Next, the specific steps of interactive text filtering are explained.

[0122] Figure 8 FIG. 1 is a schematic diagram showing a method for interactive text filtering according to an exemplary embodiment. Figure 8As shown in the figure, the input of this method mainly includes three parts: the output of the large language model (LLM), which aims to provide the algorithm content output of the current interaction; the interactive text, which aims to provide the question information of the current interlocutor; and the audio and video coding features, which aims to provide effective visual and auditory data for subsequent content interaction. The electronic device performs word segmentation on the output of the LLM and the interactive text, dividing them into tokens of different numbers but the same length, and performs similar table lookup based on the word segmentation results. The principle is to use the tokens of the interactive text as this table, find key information from the tokens of the LLM output information, and map this information and the interactive text as a keyword table into text Embedding. This part is mainly implemented through the pre-trained Transformer algorithm, and the Embedding is mapped to a feature list of a specific dimension through a convolution function (Conv1D) initialized with all "1". At the same time, the audio and video coding features are also mapped to a feature list of the same dimension through Conv1D initialized with all "1", and spliced ​​into feature dimension 1 under the same feature dimension. Next, the concatenated feature dimension 1 is sent to the "Time Series Correlation Sorting" module for sorting. The main purpose is to generate feature representations containing different time series features. As shown in the figure, 2, 3, 4, 5, and 6 are feature representations of different time series features. 2 has the strongest time series correlation, 3, 4, and 5 have decreasing time series correlations, and 6 has the weakest time series correlation. Features with strong time series correlation are used as reference features and sent back to the text buffer after privacy filtering.

[0123] In the disclosed embodiment, time series correlation sorting mainly sorts a set of feature sequence data to determine the strength and direction of the correlation between them, which can help us understand the relationship between time series data. The implementation principle is to use the autocorrelation function (ACF) and cross-correlation function (CCF) of two sets of features to calculate the correlation. ACF is a correlation function between a time series and its own lagged version, while CCF is a correlation function between two time series. The sorting results are divided into segments with a length of 1024 and an offset of 256, for example, and sent to the Text-Buffer as new features generated by the interactive text filtering mechanism.

[0124] In the disclosed embodiment, the purpose of performing similarity table lookup includes: establishing the correlation between the previous round of dialogue and the new round of dialogue, outputting a keyword table, and sending key feature information to the language model; ensuring that each dialogue is relevant.

[0125] In the disclosed embodiments, interactive text filtering can reduce the possibility of human-induced electronic devices to commit crimes, reduce the probability of user privacy exposure, and ensure the privacy security of users.

[0126] The following is an explanation of a method for a language model to output response information.

[0127] Fig. 9 is a flowchart of a method for outputting response information using a language model according to an exemplary embodiment. Fig. 9 As shown, the following steps S91-S92 are included.

[0128] In step S91, privacy filtering is performed on the response information.

[0129] In step S92, the response information after privacy filtering is output.

[0130] In the disclosed embodiment, when the electronic device does not receive the user interaction information within the first time, it is deemed that the human-computer interaction is ended, and the current round of response information output by the language model is output as the response information after privacy filtering.

[0131] In the disclosed embodiment, privacy filtering of response information can reduce the exposure rate of user privacy and ensure the privacy security of the user.

[0132] The complete process of human-computer interaction of multimodal data is described below.

[0133] Fig.10 is a schematic diagram of a human-computer interaction method for multimodal data according to an exemplary embodiment. Fig.10As shown, inside the electronic device, for the audio and video data sent to the technical solution, the video or picture data enters the picture encoder for content encoding, and the video (Video) information is mapped to the semantic features of the high-dimensional space through the picture encoder (Video Q-Former). At the same time, the audio data enters the audio encoder for content encoding, and the audio (Audio) information is mapped to the semantic features of the high-dimensional space through the audio encoder (Audio Q-Former). Privacy filtering is performed on the semantic information of the two parts of the high-dimensional space respectively. The filtered information is mapped to Embeddings of the same dimension and spliced, and a Text-Buffer is established as valid content information. The data of the Text-Buffer is sent to the LLM, and valid data is provided for the content generation of the LLM as a prompt text. Before the output response information of the LLM, the current content is privacy filtered again. At the same time, the response information and the interactive text information are sent to the interactive text filtering module as a reference. Its purpose is to obtain the non-existent information N in the understanding content of the current interactive process based on the interactive information and the generated information, and to establish a connection between the information N and the semantic features of the most original audio / video in the high-dimensional space, and send it to the Text-Buffer for fusion again, as the database information for the next interaction, to re-judge whether the non-existent information N is really in the visual and auditory content. Among them, the secondary judgment of the flow box in the figure is equivalent to the interactive text filtering mechanism disclosed in this disclosure. Such an interactive mechanism and interactive text filtering mechanism also establish the ability to understand and interact with long texts.

[0134] In the disclosed embodiment, the Text-Buffer not only contains the valid information currently heard and seen, but also contains the response information from the interactive information and the current output, which is equivalent to a temporary database, integrating what is heard and seen as well as the interactive information and the response information. Its purpose is, firstly, to establish the contextual understanding ability of long texts and improve the long-term interactive ability. Secondly, through dialogue interaction, the factuality / model hallucination problem of image description can be significantly improved, thereby improving the accuracy of content understanding.

[0135] In the disclosed embodiment, before the LLM outputs the response information, the current content is again subjected to "privacy filtering" to remove bad dialogue guidance and erroneous data in the long text comprehension process. Finally, the visually understood content is output.

[0136] Through the present disclosure, electronic devices can perform privacy filtering on received multimodal data to avoid user privacy exposure, and can also perform interactive text filtering on user interactive text to avoid artificially guiding electronic devices to help commit crimes.

[0137] Based on the same concept, the embodiment of the present disclosure also provides a human-computer interaction device 100 for multimodal data.

[0138] It is understandable that in order to realize the above functions, the human-computer interaction device 100 for multimodal data provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0139] Fig.11 is a block diagram of a multi-modal data human-computer interaction device according to an exemplary embodiment. Fig.11 The device 100 includes a receiving unit 101 and an execution unit 102.

[0140] The receiving unit 101 is used to obtain multimodal data and interactive text.

[0141] The execution unit 102 extracts the semantic features of each modal data in the multimodal data respectively to obtain the first semantic features of each modal data; filters the privacy features in the first semantic features of each modal data respectively to obtain the second semantic features of each modal data; and obtains response information of the multimodal data in response to the interaction information based on the interaction information and the second semantic features of each modal data.

[0142] In the disclosed embodiment, the execution unit 102 is configured to determine the similarity between the first semantic feature and the privacy feature in the privacy database, where the privacy database includes privacy features that map the privacy data based on pre-trained data; the first semantic feature is filtered out with a portion whose similarity with the privacy feature is greater than a similarity threshold, and the absolute value of the similarity of the processed first semantic feature is retained to obtain the first data feature; the features in the processed first semantic feature whose similarity is higher than a preset threshold are filtered out, and are merged with the first data feature into a second data feature; the second data feature and the first semantic feature are used to filter out the portion whose similarity with the privacy feature is greater than the similarity threshold, and the portion whose absolute difference is greater than the preset threshold is filtered out, and the second semantic feature is output.

[0143] In the disclosed embodiment, the execution unit 102 is configured to map the second semantic features of each modal data into features of the same dimension, and to concatenate the features under the same dimension to obtain a third semantic feature; the interaction information and the third semantic feature are input into the language model to obtain response information of the multimodal data in response to the interaction information.

[0144] In the disclosed embodiment, the execution unit 102 is configured to input the third semantic feature into the language model to obtain first response information of the multimodal data response interaction information; based on the first response information, the first semantic feature and the interaction information, perform interactive text filtering on the interaction information and the first semantic feature to obtain filtered interaction information; input the filtered interaction information and the third semantic feature into the language model to obtain second response information of the multimodal data response interaction information.

[0145] In the disclosed embodiment, the execution unit 102 is configured to perform word segmentation on the first response information and the interaction information respectively to obtain word segmentation features having the same length; perform a similarity table lookup on the word segmentation features to filter out word segmentation features whose similarity between the first response information and the interaction information is lower than a similarity threshold to obtain a keyword table, wherein the keyword table includes word segmentation features whose similarity between the first response information and the interaction information is greater than a preset threshold; perform text feature mapping on the keyword table to obtain a first mapping feature; and filter the first mapping feature and the first semantic feature to obtain filtered interaction information and the third semantic feature.

[0146] In the disclosed embodiment, the execution unit 102 is configured to perform convolution mapping on the first mapping feature to obtain a second mapping feature, and to perform convolution mapping on the first semantic feature to obtain a third mapping feature; concatenate the second mapping feature and the third mapping feature to obtain a fourth mapping feature; perform temporal sorting on the fourth mapping feature in a manner from strong to weak in terms of temporal correlation; filter out a set number of fourth mapping features whose sorting positions are from weak to strong; and determine the filtered fourth mapping features as the filtered interaction information and the third semantic feature.

[0147] In the embodiment of the present disclosure, the execution unit 102 is further configured to perform privacy filtering on the response information and output the privacy filtered response information.

[0148] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0149] Fig.12 1 is a block diagram of a device 200 for human-computer interaction of multimodal data according to an exemplary embodiment. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0150] Reference Fig.12, the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0151] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0152] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0153] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0154] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0155] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0156] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0157] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0158] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0159] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0160] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0161] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0162] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0163] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0164] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0165] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0166] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A human-computer interaction method based on multimodal data, characterized in that: include: Acquire multimodal data and interactive text; Extracting semantic features of each modal data in the multimodal data respectively to obtain a first semantic feature of each modal data; Filtering the privacy features in the first semantic features of each modal data respectively to obtain the second semantic features of each modal data; Based on the interaction information and the second semantic feature of each modal data, response information of the multimodal data in response to the interaction information is obtained.

2. The method according to claim 1, characterized in that The filtering of the privacy features in the first semantic features of each modal data respectively includes: Determine a similarity between the first semantic feature and a privacy feature in a privacy database, wherein the privacy database includes privacy features that map the privacy data based on pre-training data; The first semantic feature is filtered out of a part whose similarity with the privacy feature is greater than a similarity threshold, and the absolute value of the similarity of the first semantic feature after processing is retained to obtain a first data feature; Select the features with similarity higher than a preset threshold from among the first semantic features after screening, and merge them with the first data feature into a second data feature; The second data feature and the first semantic feature are used to filter out the part whose similarity with the privacy feature is greater than a similarity threshold, and the part whose absolute difference is greater than a preset threshold is filtered out to output the second semantic feature.

3. The method according to claim 1 or 2, characterized in that: The obtaining, based on the interaction information and the second semantic features of each modal data, response information of the multimodal data in response to the interaction information includes: Mapping the second semantic features of each modal data into features of the same dimension, and concatenating the features under the same dimension to obtain a third semantic feature; The interaction information and the third semantic feature are input into a language model to obtain response information of the multimodal data in response to the interaction information.

4. The method according to claim 3, characterized in that The step of inputting the interaction information and the third semantic feature into a language model to obtain response information of the multimodal data in response to the interaction information includes: Inputting the third semantic feature into a language model to obtain first response information of the multimodal data in response to the interaction information; Based on the first response information, the first semantic feature, and the interaction information, performing interactive text filtering on the interaction information and the first semantic feature to obtain filtered interaction information; The filtered interaction information and the third semantic feature are input into the language model to obtain second response information of the multimodal data in response to the interaction information.

5. The method according to claim 4, characterized in that The performing interactive text filtering on the interactive information and the first semantic feature based on the first response information, the first semantic feature, and the interactive information includes: Performing word segmentation on the first response information and the interaction information respectively to obtain word segmentation features having the same length; Performing a similarity table lookup on the word segmentation features, filtering out word segmentation features whose similarity between the first response information and the interactive information is lower than a similarity threshold, and obtaining a keyword table, wherein the keyword table includes word segmentation features whose similarity between the first response information and the interactive information is higher than a preset threshold; Performing text feature mapping on the keyword table to obtain a first mapping feature; The first mapping feature and the first semantic feature are filtered to obtain filtered interaction information and the third semantic feature.

6. The method according to claim 5, characterized in that The filtering the first mapping feature and the first semantic feature to obtain the filtered interaction information and the third semantic feature includes: Performing convolution mapping on the first mapping feature to obtain a second mapping feature, and performing convolution mapping on the first semantic feature to obtain a third mapping feature; Concatenate the second mapping feature and the third mapping feature to obtain a fourth mapping feature; sorting the fourth mapping features in time sequence according to the time sequence correlation from strong to weak; Screening out a set number of fourth mapping features whose ranking positions are from weak to strong; The screened-out fourth mapping feature is determined as the filtered interaction information and the third semantic feature.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The response information is privacy filtered, and the response information after the privacy filtering is output.

8. A human-computer interaction device based on multimodal data, characterized in that: The device comprises: A receiving unit, used to obtain multimodal data and interactive text; The execution unit extracts the semantic features of each modal data in the multimodal data respectively to obtain the first semantic features of each modal data; filters the privacy features in the first semantic features of each modal data respectively to obtain the second semantic features of each modal data; and obtains response information of the multimodal data in response to the interactive information based on the interactive information and the second semantic features of each modal data.

9. The device according to claim 8, characterized in that The execution unit filters the privacy features in the first semantic features of each modal data respectively in the following manner: Determine a similarity between the first semantic feature and a privacy feature in a privacy database, wherein the privacy database includes privacy features that map the privacy data based on pre-training data; The first semantic feature is filtered out of a part whose similarity with the privacy feature is greater than a similarity threshold, and the absolute value of the similarity of the first semantic feature after processing is retained to obtain a first data feature; Select the features with similarity higher than a preset threshold from among the first semantic features after screening, and merge them with the first data feature into a second data feature; The second data feature and the first semantic feature are used to filter out the part whose similarity with the privacy feature is greater than a similarity threshold, and the part whose absolute difference is greater than a preset threshold is filtered out to output the second semantic feature.

10. The device according to claim 8 or 9, characterized in that The execution unit obtains the response information of the multimodal data in response to the interaction information based on the interaction information and the second semantic feature of each modal data in the following manner: Mapping the second semantic features of each modal data into features of the same dimension, and concatenating the features under the same dimension to obtain a third semantic feature; The interaction information and the third semantic feature are input into a language model to obtain response information of the multimodal data in response to the interaction information.

11. The device according to claim 10, characterized in that The execution unit inputs the interaction information and the third semantic feature into the language model to obtain the response information of the multimodal data in response to the interaction information in the following manner: Inputting the third semantic feature into a language model to obtain first response information of the multimodal data in response to the interaction information; Based on the first response information, the first semantic feature, and the interaction information, performing interactive text filtering on the interaction information and the first semantic feature to obtain filtered interaction information; The filtered interaction information and the third semantic feature are input into the language model to obtain second response information of the multimodal data in response to the interaction information.

12. The device according to claim 11, characterized in that The execution unit performs interactive text filtering on the interactive information and the first semantic feature based on the first response information, the first semantic feature and the interactive information in the following manner: Performing word segmentation on the first response information and the interaction information respectively to obtain word segmentation features having the same length; Performing a similarity table lookup on the word segmentation features, filtering out word segmentation features whose similarity between the first response information and the interactive information is lower than a similarity threshold, and obtaining a keyword table, wherein the keyword table includes word segmentation features whose similarity between the first response information and the interactive information is higher than a preset threshold; Performing text feature mapping on the keyword table to obtain a first mapping feature; The first mapping feature and the first semantic feature are filtered to obtain filtered interaction information and the third semantic feature.

13. The device according to claim 12, characterized in that The execution unit filters the first mapping feature and the first semantic feature in the following manner to obtain the filtered interaction information and the third semantic feature: Performing convolution mapping on the first mapping feature to obtain a second mapping feature, and performing convolution mapping on the first semantic feature to obtain a third mapping feature; Concatenate the second mapping feature and the third mapping feature to obtain a fourth mapping feature; sorting the fourth mapping features in time sequence according to the time sequence correlation from strong to weak; Screening out a set number of fourth mapping features whose ranking positions are from weak to strong; The screened-out fourth mapping feature is determined as the filtered interaction information and the third semantic feature.

14. The device according to any one of claims 8 to 13, characterized in that The execution unit is also used for: The response information is privacy filtered, and the response information after the privacy filtering is output.

15. A multimodal data human-computer interaction device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the human-computer interaction method of multimodal data as described in any one of claims 1-7.

16. A storage medium, characterized in that: The storage medium stores instructions. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to execute the human-computer interaction method for multimodal data described in any one of claims 1 to 7.

Images