Device control method and apparatus, and appliance device

Abstract

This application relates to a device control method and apparatus, and an appliance device, and relates to the field of intelligent control technologies. The method includes: obtaining a cross-modal device control signal, and obtaining control data of a plurality of modalities based on the device control signal, where the modalities include at least an image, audio, and text; encoding the control data of each modality, to generate a semantic vector corresponding to each modality; performing cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based on a multi-head attention mechanism, to obtain a fused semantic vector; and determining a control instruction based on the fused semantic vector, and outputting the control instruction. According to this method, a device control intention carried in the device control signal can be more accurately captured by using multi-modal semantic information, thereby improving device control efficiency and ensuring user experience of a device.

Description

[0001] DEVICE CONTROL METHOD AND APPARATUS, AND APPLIANCE DEVICE TECHNICAL FIELD This application relates to the field of intelligent control technologies, and in particular, to a device control method and apparatus, and an appliance device. BACKGROUND The human-computer interaction technology is a wide field. Currently, popular technologies are a speech recognition technology and a gesture recognition technology. The speech recognition technology can help a user to control a device, enter a text, perform an operation, or the like by using a speech. The gesture recognition technology can enable the user to control a computer or another intelligent device by using a gesture. More companies apply the two technologies to products and services, to provide more diversified and natural human- computer interaction experience for the user. However, although the speech recognition technology and the gesture recognition technology provide great convenience for the user, there are still great limitations in some cases. For example, in a noisy environment, or when the user has a strong accent or unclear language expression, accuracy of speech recognition may be reduced, resulting in a control instruction of the user being misunderstood or unrecognizable. For another example, in an environment with insufficient light, or a gesture made by the user is relatively complex, accuracy of gesture recognition may also be affected. SUMMARY Based on this, it is necessary to provide a device control method and apparatus, and an appliance device. According to a first aspect, this application provides a device control method. The method includes: obtaining a cross-modal device control signal, and obtaining control data of a plurality of modalities based on the device control signal, where the modalities include at least an image, audio, and text; encoding the control data of each modality, to generate a semantic vector corresponding to each modality; performing cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based on a multi-head attention mechanism, to obtain a fused semantic vector; and determining a control instruction based on the fused semantic vector, and outputting the control instruction. In an embodiment, the cross-modal device control signal is an audio and video signal associated with a control behavior; and the obtaining control data of a plurality of modalities based on the device control signal includes: extracting audio data and video data in the audio and video signal; performing speech recognition on the audio data, to generate text data; and selecting a video frame included in the video data as image data. In an embodiment, the semantic vectors include at least an image semantic vector, an audio semantic vector, and a text semantic vector; and the performing cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based on a multi-head attention mechanism, to obtain a fused semantic vector includes: performing vector integration on image semantic vectors corresponding to the plurality of pieces of image data when a plurality of pieces of image data are obtained from a plurality of video frames, to obtain an integrated image semantic vector; and performing cross-modal attention calculation on the audio semantic vector, the text semantic vector, and the integrated image semantic vector based on the multi-head attention mechanism, to obtain the fused semantic vector. In an embodiment, the performing cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based on a multi-head attention mechanism, to obtain a fused semantic vector includes: separately performing multilayer single-head attention calculation on the semantic vector corresponding to each modality, to obtain a single-modal attention vector corresponding to each modality; separately performing cross-modal attention calculation on any two of the semantic vectors corresponding to the plurality of modalities, to obtain a plurality of cross-modal attention vectors; and generating the fused semantic vector based on the single-modal attention vector corresponding to each modality and the plurality of cross-modal attention vectors. In an embodiment, the separately performing multilayer single-head attention calculation on the semantic vector corresponding to each modality, to obtain a single-modal attention vector corresponding to each modality includes: using a semantic vector corresponding to each modality as a first layer of vector; sequentially merging sub-vectors in a plurality of adjacent rows in a previous layer of vector, to obtain a new layer of vector, until a quantity of rows of the new layer of vector reaches a preset value; performing single-head attention calculation on each layer of vector, to obtain an attention vector corresponding to each layer; and obtaining the single-modal attention vector corresponding to each modality based on the attention vector corresponding to each layer. In an embodiment, the cross-modal attention calculation includes: selecting a semantic vector corresponding to a first modality as a query vector, and selecting a semantic vector corresponding to a second modality as a key and value vector; performing dot product calculation on the query vector and the key vector, to obtain a similarity score; performing similarity normalization on the similarity score, to obtain an attention weight matrix; and applying the attention weight matrix to the value vector, to obtain the cross-modal attention vector of the first modality relative to the second modality. In an embodiment, the generating the fused semantic vector based on the single-modal attention vector corresponding to each modality and the plurality of cross-modal attention vectors includes: concatenating the single-modal attention vector corresponding to each modality and the plurality of cross-modal attention vectors, to obtain a composite attention vector; and multiplying the composite attention vector by a preset fused weight matrix, to generate the fused semantic vector. In an embodiment, the determining a control instruction based on the fused semantic vector includes: inputting the fused semantic vector to a functional classifier on which pretraining is completed, to obtain target category information; and determining a control instruction corresponding to the target category information. According to a second aspect, this application further provides a device control apparatus. The apparatus includes: a data obtaining module, configured to obtain a cross-modal device control signal, and obtain control data of a plurality of modalities based on the device control signal, where the modalities include at least an image, audio, and text; a data encoding module, configured to encode the control data of each modality, to generate a semantic vector corresponding to each modality; an attention calculation module, configured to perform cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based on a multi-head attention mechanism, to obtain a fused semantic vector; and an instruction output module, configured to determine a control instruction based on the fused semantic vector, and output the control instruction. According to a third aspect, this application further provides an appliance device. The appliance device includes a memory and a processor. The memory stores a computer program. The processor, when executing the computer program, implements the device control method according to the first aspect. According to a fourth aspect, this application further provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program. The processor, when executing the computer program, implements the device control method according to the first aspect. According to a fifth aspect, this application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. The computer program, when executed by a processor, implements the device control method according to the first aspect. According to a sixth aspect, this application further provides a computer program product. The computer program product includes a computer program. The computer program, when executed by a processor, implements the device control method according to the first aspect. According to the device control method disclosed in this application, an intelligent processor obtains a cross-modal device control signal, obtains control data of a plurality of modalities based on the device control signal, and encodes the control data of each modality, to generate a semantic vector corresponding to each modality; performs cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based on a multi- head attention mechanism, to obtain a fused semantic vector; and determines a control instruction based on the fused semantic vector, and outputs the control instruction. In this way, the device control signals of the plurality of modalities are obtained, and semantic analysis is performed on the device control signals in a multi-modal semantic space, so that a device control intention carried in the device control signal can be more accurately captured by using multi-modal semantic information, thereby improving device control efficiency and ensuring user experience of a device. In addition, data of a plurality of modalities is transferred into the same embedding space for encoding and integration, so that processing complexity of multi- modal data can be reduced, thereby improving processing efficiency. The device control method disclosed in the embodiments is applied to the field of appliances, so that non-contact human-computer interaction can be implemented, and physical contact between a user and a device can be avoided. This not only improves user experience, but also helps a user who cannot perform physical control to use an appliance more conveniently. BRIEF DESCRIPTION OF THE DRAWINGS FIG.1 is a schematic flowchart of a device control method according to an embodiment; FIG. 2 is a schematic flowchart of obtaining control data of a plurality of modalities according to an embodiment; FIG. 3 is a schematic flowchart of generating a fused semantic vector according to an embodiment; FIG. 4 is a schematic flowchart of single-head attention calculation according to an embodiment; FIG. 5 is a schematic flowchart of cross-modal attention calculation according to an embodiment; FIG.6 is a schematic structural diagram of a joint encoder according to an embodiment; FIG. 7 is a schematic structural diagram of a device control apparatus according to an embodiment; and FIG.8 is a diagram of an inner structure of a computer device according to an embodiment. DETAILED DESCRIPTION To make the objectives, technical solutions, and advantages of this application clearer, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that, the specific embodiments described herein are only used for explaining this application, and are not used for limiting this application. A speech recognition technology and a gesture recognition technology are two human- computer interaction technologies that are relatively widely applied currently. A speech and a gesture of a user are mainly detected, collected, and recognized through an intelligent device, so that a control intention of the user is determined, thereby converting the control intention into a specific control instruction. However, in a specific case, the speech recognition technology and the gesture recognition technology still have relatively large limitations. For the speech recognition technology, in a noisy environment such as a public place or an industrial place, speech recognition may be interfered with by background noise, and it is difficult for the intelligent device to accurately detect and collect a voice of the user in time. When the user has a strong accent or unclear language expression, it may be difficult for the intelligent device to accurately recognize a real control intention of the user from collected voice. For the gesture recognition technology, in an environment with insufficient light, for example, in a dark room or at night, the intelligent device possibly cannot accurately capture a gesture of the user. Consequently, detection, collection, and recognition processes are probably affected. When the user makes some complex or uncommon gestures, it is also difficult for the intelligent device to correctly recognize the control intention of the user. To resolve the problems in the related art, an embodiment of this application provides a device control method. The method is applicable to an intelligent processor. The intelligent processor may be any unit having a data read and write function and a data processing function, and may be configured to: convert an input device control signal of different modalities into a control instruction that can be executed by a device and output the control instruction. The intelligent processor may be specifically a control chip, a CPU, a GPU, a digital signal processor, or the like. The intelligent processor may be embedded in the intelligent device, and is configured to implement a manual interaction function of the intelligent device. The intelligent device may be specifically an intelligent digital device, an intelligent household device, an intelligent in-vehicle device, an intelligent security device, or the like. This embodiment is described by using an example in which an intelligent device is an intelligent appliance, and an intelligent processor is embedded in the intelligent appliance. Other cases are similar to this, and supplementary descriptions are provided as necessary. In an embodiment, as shown in FIG.1, a device control method is provided, including the following steps. Step 102: Obtain a cross-modal device control signal, and obtain control data of a plurality of modalities based on the device control signal. A modality refers to a specific information source or a data type, and includes at least an image, audio, and text. The cross-modal device control signal may be a device control signal including data of the plurality of modalities. During implementation, the intelligent device may be built with a multi-modal signal collection assembly, to collect device control signals of different modalities. For example, the intelligent device may be built with sensors such as a camera and a microphone, which are respectively configured to collect an image, a video signal, and an audio signal. After obtaining the cross-modal device control signal, the intelligent processor may analyze the device control signal, to extract the control data of the plurality of modalities from the device control signal. The cross-modal device control signal herein may be sent to the intelligent processor after being collected by the signal collection assembly; or may be input to the intelligent processor by using an intermediate storage medium after being collected by the signal collection component. Further, the cross-modal device control signal may be an audio and video signal associated with a control behavior. Correspondingly, a process of obtaining the control data in step 102 may be detailed into a process shown in FIG.2. Step 1021: Extract audio data and video data in the audio and video signal. Step 1022: Perform speech recognition on the audio data, to generate text data. Step 1023: Select a video frame included in the video data as image data. During implementation, the signal collection assembly may continuously collect the control behavior performed by the user on the intelligent device; or collect the audio and video signal after a control process is started, and provide all collected audio and video signals to the intelligent processor. Herein, the user may set a starting condition of the control process on the intelligent device according to an actual requirement, so that the intelligent device can trigger the signal collection assembly to collect the audio and video signal when the starting condition is satisfied. After obtaining the audio and video signal, the intelligent processor may first determine whether the audio and video signal is associated with the control behavior, and if the audio and video signal is associated with the control behavior, the intelligent processor may further extract the audio data and the video data in the audio and video signal. Then, the intelligent processor may perform speech recognition on the audio data, to extract text data included in the audio data, and may use the video frame included in the video data as the image data, to obtain the control data of the plurality of modalities (an image, audio, and text). It should be noted that whether the audio and video signal is associated with the control behavior may be determined by determining whether the audio and video signal satisfies a preset condition. For a specific manner, refer to a related technology. This is not limited in this embodiment. Step 104: Encode the control data of each modality, to generate a semantic vector corresponding to each modality. During implementation, an encoder for each modality may be preconfigured in the intelligent processor. The encoder may be a machine learning model based on artificial intelligence. The encoder learns a mode and a feature in material data through a training process, and can map new data to a specific vector space, to encode and represent data. After obtaining the control data of the plurality of modalities, the intelligent processor may encode the control data of each modality through the encoder, to generate the semantic vector corresponding to each modality. The semantic vector herein is a low-dimension vector representation, is used for capturing and representing semantic information in input data, and may be used for performing various semantics-related tasks, such as classification, clustering, and semantic matching. Specifically, when a modality includes at least an image, audio, and text, at least a corresponding image encoder, audio encoder, and text encoder may exist, to separately encode control data of each modality independently and extract features. The image encoder usually extracts a feature of an image by using a model such as a convolutional neural network (CNN), a generative adversarial network (GAN), or a vision transformer (VIT), and converts the image into a low-dimension vector representation. The audio encoder may include a mel-frequency cepstral coefficient (MFCC) encoder, a long short-term memory (LSTM) encoder, a temporal convolutional network (TCN) encoder, and the like, and is configured to extract a feature of audio and convert the feature into a vector representation. The text encoder converts text into a vector representation, and usually uses models such as a Global Vector for Word Representation (GloVe) model, a bag of words (BoW) model, or a more complex bidirectional encoding representation from transformers (BERT) and generative pre-trained transformer (GPT) model. Further, the semantic vector may be a vector including a plurality of tokens, and the token may be a smallest unit carrying semantic information. For example, a text token included in a text semantic vector refers to a basic unit when the text data is processed. To be specific, the text data is first decomposed into text tokens, and each text token is mapped to a vector space to obtain a vector of the text token, so that the vectors of the text tokens may be combined to obtain the text semantic vector. An image token included in an image semantic vector is a basic unit carrying an image feature, and may usually be a pixel block formed by a group of pixels. In this way, a plurality of pixel blocks (that is, image tokens) can be obtained from the image data. Each pixel block corresponds to one or more local features in the image data. Then, the pixel block is mapped to a vector space, to obtain a vector of an image token. Therefore, the vectors of the plurality of image tokens may be combined to obtain the image semantic vector. An audio token included in an audio semantic vector may be a basic unit carrying audio information, and may be an audio frame or an audio segment. Therefore, continuous audio data may be decomposed into a plurality of audio tokens at specific time intervals. Each audio token includes a part of audio data. Feature extraction is then performed on the audio token to generate a vector of the audio token. Finally, the vectors of the audio tokens are combined to obtain the audio semantic vector. For example, an example in which the intelligent device is an intelligent cooker hood embedded with an intelligent processor is used. A camera and a microphone are built in the intelligent cooker hood, can capture a gesture and a speech of the user, and recognize the speech by using a real-time speech recognition technology to generate text data. An example in which the user sends a speech instruction of "turn on the power" is used. The intelligent cooker hood obtains related audio data through the built-in microphone, and transfers the audio data as an input to the audio encoder. The audio encoder may convert the audio data into a low-dimension vector representation, and the vector representation can reserve semantic information of the audio data. The intelligent processor may further perform speech recognition on the audio data, convert the audio data into text data of "turn on the power", and input the text data to the text encoder. The text encoder may convert "turn on the power" into a vector sequence, where each vector represents an expression or a character. For example, "turn on the power" may be divided into two phrases: "turn on" and "the power", and each phrase is converted into a vector, or "turn on the power" may be represented as a character sequence, where each character is converted into a vector. In addition, if the user makes a gesture corresponding to "turn on the power", the intelligent cooker hood may obtain related image data through the camera, and transmit the image data as an input to the image encoder. The image encoder may convert the image data into a low-dimension vector representation, and the vector representation can reserve semantic information of the image data. Step 106: Perform cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based on a multi-head attention mechanism, to obtain a fused semantic vector. A multi-head attention mechanism is a common self-attention mechanism, configured to exchange each element in an input sequence with another element in the sequence, and calculate importance of the element for the another element. The multi-head attention mechanism extends the self-attention mechanism to a plurality of attention heads, which can improve an expression capability and a generalization capability of a model. In a multi-head attention mechanism, an input sequence is first respectively mapped to a plurality of different vector spaces (that is, multi-head). Each vector space has an independent attention mechanism, so that the importance of each element in the input sequence in the vector space can be calculated. Calculation results of all vector spaces are concatenated, and a final output vector is obtained through linear transformation. The multi-head attention mechanism can capture semantic relationships at different positions in an input sequence, and consider importance of a plurality of semantic dimensions at the same time. An ordinary multi-head attention mechanism is performed in a same modality. A cross-modal multi-head attention mechanism is used, attention calculation may be performed on vectors of different modalities, and multi- modal semantic information can be captured better with reference to a multi-head attention mechanism with a hierarchical structure. During implementation, after encoding to generate the semantic vector corresponding to each modality, the intelligent processor may establish a semantic association between semantic vectors of different modalities by using the multi-head attention mechanism, to capture a complex relationship between the control data of different modalities. In detail, the intelligent processor may perform cross-modal attention calculation on the semantic vectors corresponding to the plurality of modalities, to obtain outputs of the plurality of vector spaces, and then concatenate the outputs of the plurality of vector spaces. After linear transformation, a unified fused semantic vector may be formed. The fused semantic vector may include information from the control data of the different modalities, and effectively integrate the information in a high-dimensional semantic space. Optionally, when the cross-modal device control signal is an audio and video signal associated with the control behavior, the semantic vector generated in step 104 includes at least an image semantic vector, an audio semantic vector, and a text semantic vector. Further, processing of step 106 may be: performing cross-modal attention calculation on the audio semantic vector, the text semantic vector, and the image semantic vector based on the multi-head attention mechanism, to obtain the fused semantic vector. Further, considering that when the control operation of the intelligent device is a continuously changing gesture or action, a plurality of video frames having different picture content need to be selected from the video data as image data, thereby obtaining a plurality of pieces of image data. Herein, a video frame selection manner may be determined based on a picture ratio of a subject object and picture content similarity between video frames. A control- related subject object, such as a human hand or a human face, may be first determined. A video frame of which a picture ratio of the subject object is greater than a preset threshold is selected. The first selected video frame is used as the first piece of image data, a subsequent video frame is compared with the first piece of image data, a next video frame of which a picture content similarity is less than the preset threshold is used as the second piece of image data, and a subsequent video frame is compared with the second piece of image data. The plurality of pieces of image data can be selected through sequentially repeating the steps. Certainly, the video frames may also be selected in other manners. For example, the video frames are selected at fixed time intervals, or video frames at specified positions are directly selected. For example, a start frame, a middle frame, and an end frame of the video data are directly selected. In this case, processing of step 106 may be specifically: performing vector integration on image semantic vectors corresponding to the plurality of pieces of image data when a plurality of pieces of image data are obtained from a plurality of video frames, to obtain an integrated image semantic vector; and performing cross-modal attention calculation on the audio semantic vector, the text semantic vector, and the integrated image semantic vector based on the multi- head attention mechanism, to obtain the fused semantic vector. During implementation, if extracting a plurality of pieces of image data from the device control signal, the intelligent processor may encode each piece of image data, to generate an image semantic vector corresponding to each piece of image data. Then, the intelligent processor may perform vector integration on the image semantic vectors corresponding to the plurality of pieces of image data, to obtain an integrated image semantic vector. In this embodiment, vector integration may be performed on image semantic vectors corresponding to a plurality of frames of image data in a plurality of manners. For example, image semantic vectors corresponding to frames of image data may be directly concatenated. Alternatively, to reduce a data volume participating in calculation, image data may be selected at fixed intervals of a quantity of frames, and image semantic vectors corresponding to the selected image data are concatenated. This is not limited in this embodiment. Based on the foregoing processing, when performing cross-modal attention calculation, the intelligent processor may perform calculation by using the integrated image semantic vector and semantic vectors of remaining modalities, to obtain the fused semantic vector. Optionally, the fused semantic vector may be generated by using the single-modal attention vector and the cross-modal attention vector. Correspondingly, processing of step 106 may be detailed into the procedure shown in FIG.3, and includes the following steps. Step 1061: Separately perform multilayer single-head attention calculation on the semantic vector corresponding to each modality, to obtain a single-modal attention vector corresponding to each modality. Step 1062: Separately perform cross-modal attention calculation on any two of the semantic vectors corresponding to the plurality of modalities, to obtain a plurality of cross- modal attention vectors. Step 1063: Generate the fused semantic vector based on the single-modal attention vector corresponding to each modality and the plurality of cross-modal attention vectors. During implementation, during cross-modal attention calculation performed on the semantic vectors corresponding to the plurality of modalities, the single-modal attention vector and the cross-modal attention vector calculation may be separately performed, and then the fused semantic vector is calculated based on the single-modal attention vector and the cross- modal attention vector. The single-modal attention vector may be important information captured from features of each modality (such as text, an image, and audio) after multilayer single-head attention calculation is performed on the modality. For example, single-modal attention calculation performed on text may capture correlations between words in the text; single-modal attention calculation performed an image may be used for determining a key region of the image and an association relationship between different regions of the captured image; and single-modal attention calculation performed on audio may recognize an audio feature and a corresponding time with reference to a spectrum feature in audio data. The cross- modal attention vector may be correlation and complementary information between multi- modal data that is captured after features of different modalities (such as text and an image, text and audio, and an image and audio) are exchanged and fused through the multi-head attention mechanism. For example, cross-modal attention calculation between text and an image may capture an association relationship between an object in the image and a keyword in the text; cross-modal attention calculation between text and audio may capture complementary information between text content and audio features (such as feelings and intonations); and cross-modal attention calculation between an image and audio may be used for establishing an association relationship between an audio feature and image content. The fused semantic vector may be a comprehensive representation vector formed by fusing rich semantic information within each modality and between modalities. In detail, processing of the multilayer single-head attention calculation in step 1061 may be detailed into a procedure shown in FIG.4, and includes the following steps. Step 10611: Use a semantic vector corresponding to each modality as a first layer of vector. A semantic vector corresponding to each modality may be used as an input of the single- head attention calculation, and the semantic vector may be used as a first layer of vector required by the single-head attention calculation. It is worth mentioning that before the single-head attention calculation is performed, output vectors corresponding to a plurality of modalities may be converted into vectors of a same length through linear transformation. It is assumed that a semantic vector X includes a text semantic vector , an image semantic vector , and an audio semantic vector . Linear transformation matrices , and are introduced. A text semantic vector T, an image semantic vector V, and an audio semantic vector A after the linear transformation can be obtained: m represents a quantity of text tokens included in the text semantic vector, and dt represents a vector length of each text token; n represents a quantity of image tokens included in the image semantic vector, and dv represents a vector length of each image token; s represents a quantity of audio tokens included in the audio semantic vector, and darepresents a vector length of each audio token; and k is a preset vector length, to be specific, vector lengths of the text semantic vector T, the image semantic vector V, and the audio semantic vector A obtained through linear transformation Wt0, Wv0, and Wa0 respectively performed on a text semantic vector T', an image semantic vector V', and an audio semantic vector A' are uniformly k. Herein, the linear transformation matrices Wt0, Wv0, and Wa0 may be obtained through continuous optimization through model training. For details, refer to the following descriptions for the encoder. Step 10612: Sequentially merge sub-vectors in a plurality of adjacent rows in a previous layer of vector, to obtain a new layer of vector, until a quantity of rows of the new layer of vector reaches a preset value. During implementation, a semantic vector is used as a first layer of vector, the semantic vector includes a plurality of sub-vectors, and sub-vectors in a plurality of adjacent rows may be combined into a row according to an arrangement order of the sub-vectors, to obtain a new layer of vector. Then, combination of adjacent sub-vectors may be continued based on a previous layer of vector, to continuously generate a new layer of vector until a quantity of rows of the new layer of vector reaches a preset value. Assuming that the semantic vector , adjacent l sub-vectors are combined, to obtain a new layer of vector . Then, sub- vector combination is repeated to obtain a vector , until a new layer of vector with a quantity of rows reaches is the sub-vector combination using every l sub-vectors as one group, if a number of remaining uncombined sub- vectors is less than l, an empty vector may be added. Step 10613: Perform single-head attention calculation on each layer of vector, to obtain an attention vector corresponding to each layer. For the single-head attention calculation, an attention function may be used. For each layer of vector , a corresponding attention vector Attention(Xi, Xi, Xi) may be obtained. Step the single-modal attention vector corresponding to each modality based on the attention vector corresponding to each layer. During implementation, after attention calculation results Attention (Xi, Xi, Xi) of different layers are obtained, the attention calculation results may be concatenated and multiplied by a weight matrix W1. Therefore, a single-modal attention vector corresponding to each modality may be obtained. H(X) is for HierarchicalAttention (X), representing performing multilayer single- head attention calculation on X. The weight matrix W1is used for combining the attention calculation results of the layers to form a final single-modal attention vector. There may be corresponding weight matrices , , and respectively for the text semantic vector T, the image semantic vector vector A. The weight matrices , , and may be obtained by continuously optimization through model training. For details, refer to the following descriptions for the encoder. Similarly, processing of the cross-modal attention calculation in step 1062 may be detailed into a procedure shown in FIG.5, and includes the following steps. Step 10621: Select a semantic vector corresponding to a first modality as a query vector, and select a semantic vector corresponding to a second modality as a key and value vector. Step 10622: Perform dot product calculation on the query vector and the key vector, to obtain a similarity score. Step 10623: Perform similarity normalization on the similarity score, to obtain an attention weight matrix. Step 10624: Apply the attention weight matrix to the value vector, to obtain the cross- modal attention vector of the first modality relative to the second modality. The first modality may be any one of the plurality of modalities, and the second modality may be any one of the plurality of modalities that is different from the first modality. During implementation, the intelligent processor may randomly select one modality from the plurality of modalities as the first modality, randomly select another modality as the second modality, use a semantic vector corresponding to the first modality as the query vector, and use semantic vectors corresponding to the second modality as the key vector and the value vector. During the cross-modal attention calculation, similarity calculation may be performed through the query vector and the key vector, to determine a correlation or a matching degree between the query vector and the key vector, thereby determining the attention weight. A final output is a weighted sum calculated based on the value vector and the corresponding attention weight. Therefore, a dot product calculation may be performed on the query vector and the key vector, to obtain a similarity score, and then Softmax function normalization is performed on the similarity score, to obtain an attention weight matrix whose weight sum is 1. The attention weight matrix may be a probability distribution, and indicates importance of data of each dimension in the key vector to particular data in the query vector. Next, weighted summation may be performed on the attention weight matrix and each value vector, to obtain the cross- modal attention vector of the first modality relative to the second modality. Specifically, refer to the following formula: C(X, Y, Y) is short for indicates that cross-modal attention calculation is performed between X and Y, where X is used as the query vector, and Y is used as the key and value vector. Based on the foregoing process, a cross-modal attention calculation result of one modality relative to another modality can be obtained, and then all possible modality pairs in the plurality of modalities, such as "[text, image], [image, text], [audio, image], [text, audio], [audio, text], and [image, audio]" are traversed. The foregoing cross-modal attention calculation is separately performed on each modality pair, to obtain a plurality of cross-modal attention vectors. It may be understood that, the modality pairs [A, B] and [B, A] are two different modality pairs, and two corresponding cross-modal attention vectors can be obtained through cross-modal attention calculation. It is worth mentioning that before the cross-modal attention calculation is performed, output vectors corresponding to a plurality of modalities may also be converted into vectors of a same length through linear transformation. It is assumed that a semantic vector X includes a text semantic vector , an image semantic vector , and an audio semantic vector . Linear transformation matrices , and are introduced. A text semantic vector T, an image semantic vector V, audio semantic vector A after the linear transformation can be obtained: m represents a quantity vector, and dt represents a vector length of each text token; n represents a quantity of image tokens included in the image semantic vector, and dv represents a vector length of each image token; s represents a quantity of audio tokens included in the audio semantic vector, and darepresents a vector length of each audio token; and k is a preset vector length, to be specific, vector lengths of the text semantic vector T, the image semantic vector V, and the audio semantic vector A obtained through linear transformation Wt0, Wv0, and Wa0 respectively performed on a text semantic vector T', an image semantic vector V', and an audio semantic vector A' are uniformly k. Herein, the linear transformation matrices , , and may be obtained through continuous optimization through model training. For details, refer to the following descriptions for the encoder. Further, after the single-modal attention vector and the multi-modality attention vector are obtained, a process of generating the fused semantic vector mentioned in step 1063 may be specifically as follows: concatenating the single-modal attention vector corresponding to each modality and the plurality of cross-modal attention vectors, to obtain a composite attention vector; and multiplying the composite attention vector by a preset fused weight matrix, to generate the fused semantic vector. Semantic vectors corresponding to q modalities are defined. is an attention vector of a modality i. is a cross-modal attention vector of the modality i relative to a modality attention vector is obtained through concatenation: limited in this embodiment. A preset fused weight matrix Wmis introduced, and a fused semantic vector may be generated. The fused weight matrix Wmmay be obtained through continuous optimization through model training. For details, refer to the following descriptions for the encoder. Assuming that the semantic vector X includes the text semantic vector T, the image semantic vector V, and the audio semantic vector A, the composite attention vector may be: The fused semantic vector may be: . Step 108: Determine a control instruction based on the fused semantic vector, and output the control instruction. During implementation, after obtaining the fused semantic vector, the intelligent processor may analyze the fused semantic vector, to recognize user intention information carried in the fused semantic vector. The intelligent processor may then determine a device action corresponding to the user intention information carried in the fused semantic vector with reference to a prestored correspondence between a user intention and a device action, to determine one or more control instructions included in the device action, and finally output the one or more control instructions. In an embodiment, the user intention information carried in the fused semantic vector may be recognized by using a functional classifier. Therefore, processing of determining the control instruction may be as follows: inputting the fused semantic vector to a functional classifier on which pretraining is completed, to obtain target category information; and determining a control instruction corresponding to the target category information. The functional classifier is a machine learning model, and is configured to classify input data into specific categories. The functional classifier may learn features of different categories in a training process, and classify new data according to the features. The functional classifier may be a simple linear classifier, or may be a more complex neural network classifier, such as a multilayer perceptron or a convolutional neural network. During training of the functional classifier, a semantic vector labeled with a category may be selected as a training material, and a dimension of the semantic vector may be consistent with a dimension of the fused semantic vector. For a specific training step, refer to an existing related procedure, and details are not described in this embodiment again. During implementation, the intelligent processor may be configured with a functional classifier on which pretraining is completed. The functional classifier may map different semantic vectors to particular categories (that is, user intentions). Therefore, the intelligent processor may input the fused semantic vector into the functional classifier, to obtain target category information (that is, the user intention information), and then may determine control instruction corresponding to the target category information. It should be supplemented that most of the foregoing processing may be implemented by a joint encoder shown in FIG. 6. The joint encoder may be an artificial intelligence model configured in the intelligent processor, and includes an encoder (such as the image encoder, the audio encoder, and the text encoder mentioned above) corresponding to each modality and a multi-head attention calculation model. The encoder may be a semantic analysis model for each modality. The encoder outputs a semantic vector in each modality. The multi-head attention calculation model performs independent attention calculation on the semantic vector by introducing a plurality of attention heads. Each attention head is an attention mechanism unit running in parallel, to process the semantic vector in an independent vector space. A vector space of each attention head may be defined by using a corresponding linear transformation matrix, and different vector spaces correspond to different linear transformation matrices, as , , and described above. The attention heads perform attention calculation on a semantic vector in a corresponding vector space, to obtain respective attention distributions, that is, the foregoing single-modal attention vector or cross-modal attention vector. Finally, output vectors of the heads are concatenated together, and are integrated through linear transformation to form a fused semantic vector. The fused semantic vector corresponds to a high-dimensional vector space, where each dimension represents different semantic information. Therefore, feature vectors of different modalities can be mapped to a multi-modal semantic space by using a joint encoder, to model semantic relationships between the different modalities, to implement intelligent analysis and processing on multi-modal data. It may be understood that the joint encoder can effectively implement analysis and processing of multi- modal data by integrating data features of different modalities into one unified multi-modal semantic space. By introducing a cross-modal multi-head attention mechanism, performance and a generalization capability of a model can be improved, and intelligent processing and application of multi-modality data can be implemented. It is worth mentioning that the joint encoder is essentially a comprehensive deep learning model. Before cross-modal attention calculation is performed by using the joint encoder, model training needs to be performed on the joint encoder for a plurality of times. A material for model training may be a model input and a model output that have a known matching degree. A model training process may be performing overall training on the joint encoder, or may be separately training a single sub-model included in the joint encoder, or may be performing joint training on some of a plurality of sub-models included in the joint encoder. A specific training process is not limited in this embodiment. In a model training process, a feature rule in a material may be learned by using an optimization algorithm, thereby obtaining parameters required by the model, such as , , , , , , and described above. According to the device control method disclosed in this application, an intelligent processor obtains a cross-modal device control signal, obtains control data of a plurality of modalities based on the device control signal, and encodes the control data of each modality, to generate a semantic vector corresponding to each modality; performs cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based on a multi- head attention mechanism, to obtain a fused semantic vector; and determines a control instruction based on the fused semantic vector, and outputs the control instruction. In this way, the device control signals of the plurality of modalities are obtained, and semantic analysis is performed on the device control signals in a multi-modal semantic space, so that a device control intention carried in the device control signal can be more accurately captured by using multi-modal semantic information, thereby improving device control efficiency and ensuring user experience of a device. In addition, data of a plurality of modalities is transferred into the same embedding space for encoding and integration, so that processing complexity of multi- modal data can be reduced, thereby improving processing efficiency. The device control method disclosed in the embodiments is applied to the field of appliances, so that non-contact human-computer interaction can be implemented, and physical contact between a user and a device can be avoided. This not only improves user experience, but also helps a user who cannot perform physical control to use an appliance device more conveniently. It should be understood that, although the steps in the flowcharts involved in the embodiments described above are displayed in sequence as indicated by arrows, these steps are not necessarily performed in sequence as indicated by the arrows. Unless otherwise explicitly specified in this application, execution of the steps is not strictly limited, and the steps may be performed in other sequences. Moreover, at least part of the steps in each embodiment may include a plurality of steps or a plurality of stages. The steps or stages are not necessarily performed at the same moment but may be performed at different moments. Execution of the steps or stages is not necessarily performed in sequence, but may be performed alternately with other steps or at least part of steps or stages of other steps. Based on the same invention concept, as shown in FIG. 7, an embodiment of this application further provides a device control apparatus 700. The apparatus 700 includes: a data obtaining module 702, configured to obtain a cross-modal device control signal, and obtain control data of a plurality of modalities based on the device control signal, where the modalities include at least an image, audio, and text; a data encoding module 704, configured to encode the control data of each modality, to generate a semantic vector corresponding to each modality; an attention calculation module 706, configured to perform cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based on a multi- head attention mechanism, to obtain a fused semantic vector; and an instruction output module 708, configured to determine a control instruction based on the fused semantic vector, and output the control instruction. In an embodiment, the cross-modal device control signal is an audio and video signal associated with a control behavior; and the data obtaining module 702 is specifically configured to: extract audio data and video data in the audio and video signal; perform speech recognition on the audio data, to generate text data; and select a video frame included in the video data as image data. In an embodiment, the semantic vectors include at least an image semantic vector, an audio semantic vector, and a text semantic vector; and the attention calculation module 706 is specifically configured to: perform vector integration on image semantic vectors corresponding to the plurality of pieces of image data when a plurality of pieces of image data are obtained from a plurality of video frames, to obtain an integrated image semantic vector; and perform cross-modal attention calculation on the audio semantic vector, the text semantic vector, and the integrated image semantic vector based on the multi-head attention mechanism, to obtain the fused semantic vector. In an embodiment, the attention calculation module 706 is specifically configured to: separately perform multilayer single-head attention calculation on the semantic vector corresponding to each modality, to obtain a single-modal attention vector corresponding to each modality; separately perform cross-modal attention calculation on any two of the semantic vectors corresponding to the plurality of modalities, to obtain a plurality of cross-modal attention vectors; and generate the fused semantic vector based on the single-modal attention vector corresponding to each modality and the plurality of cross-modal attention vectors. In an embodiment, the attention calculation module 706 is specifically configured to: use a semantic vector corresponding to each modality as a first layer of vector; sequentially merge sub-vectors in a plurality of adjacent rows in a previous layer of vector, to obtain a new layer of vector, until a quantity of rows of the new layer of vector reaches a preset value; perform single-head attention calculation on each layer of vector, to obtain an attention vector corresponding to each layer; and obtain the single-modal attention vector corresponding to each modality based on the attention vector corresponding to each layer. In an embodiment, the attention calculation module 706 is specifically configured to: select a semantic vector corresponding to a first modality as a query vector, and select a semantic vector corresponding to a second modality as a key and value vector; perform dot product calculation on the query vector and the key vector, to obtain a similarity score; perform similarity normalization on the similarity score, to obtain an attention weight matrix; and apply the attention weight matrix to the value vector, to obtain the cross-modal attention vector of the first modality relative to the second modality. In an embodiment, the attention calculation module 706 is specifically configured to: concatenate the single-modal attention vector corresponding to each modality and the plurality of cross-modal attention vectors, to obtain a composite attention vector; and multiply the composite attention vector by a preset fused weight matrix, to generate the fused semantic vector. In an embodiment, the instruction output module 708 is specifically configured to: input the fused semantic vector to a functional classifier on which pretraining is completed, to obtain target category information; and determine a control instruction corresponding to the target category information. An intelligent household scenario is used as an example. The device control apparatus 700 may be an independent control device, controlling appliance devices by cooperating with the appliance devices. Specifically, the device control apparatus 700 may obtain a device control signal for the appliance device, and perform multi-modal reasoning on control data in the device control signal to recognize a control intention for the appliance device, thereby sending a control instruction corresponding to the control intention to the appliance device. The device control signal may be collected by an independent camera and microphone, or may be collected by a camera and a microphone built in a control device or an appliance device. The device control apparatus 700 may alternatively be embedded into an appliance device, for example, embedded into a cooker hood, capture video data and audio data that express a device control intention of a user by a camera and a microphone that are built in the appliance device, and convert an audio signal into text data by using a real-time speech recognition technology. Then, the device control apparatus 700 performs multi-modal reasoning according to the video data, the audio data, and the text data, to recognize and output a control instruction for the appliance device. In an embodiment, an appliance device is provided, including a memory and a processor. The memory stores a computer program. The processor, when executing the computer program, can implement the steps of the method described in step 102 to step 108. In an embodiment, a computer device is provided, and an internal structural diagram of the computer device may be shown in FIG. 8. The computer device includes a processor, a memory, an input / output (I / O) interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data. The input / output interface of the computer device is configured to exchange data between the processor and an external device. The communication interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program, when executed by the processor, implements a device control method. A person skilled in the art may understand that the structure shown in FIG. 8 is merely a block diagram of a partial structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. Specifically, the computer device may include more or fewer components than those shown in the figure, or some components may be combined, or a different component deployment may be used. In an embodiment, a computer-readable storage medium is provided, storing a computer program. The computer program, when executed by a processor, implements the steps in the foregoing method embodiments. In an embodiment, a computer program product is provided, including a computer program. The computer program, when executed by a processor, implements the steps in the foregoing method embodiments. It should be noted that the user information (including, but not limited to, user device information and user personal information) and data (including, but not limited to, data for analysis, stored data, and displayed data) involved in this application are all authorized by the user or information and data fully authorized by all parties. In addition, collection, use, and processing of relevant data need to comply with relevant regulations. A person of ordinary skill in the art may understand that all or some of procedures of the method in the foregoing embodiments may be implemented by a computer program instructing relevant hardware. The computer program may be stored in a non-volatile computer-readable storage medium. When the computer program is executed, the procedures of the foregoing method embodiments may be implemented. References to the memory, the database, or other media used in the embodiments provided in this application may all include at least one of a non-volatile or a volatile memory. The non-volatile memory may include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical memory, a high-density embedded non-volatile memory, a resistive random access memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, or the like. The volatile memory may include a random access memory (RAM), an external cache, or the like. As an illustration and not a limitation, the RAM may be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM). The processor involved in the embodiments provided in this application may be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a quantum computing-based data processing logic device, and are not limited thereto. The technical features of the above embodiments may be randomly combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, the combinations of these technical features shall be considered as falling within the scope recorded in this specification provided that no conflict exists.

[0002] The foregoing embodiments show only several implementations of this application. Descriptions of the embodiments are described in detail and specifically, but not to be construed as a limitation to the patent scope of this application. It should be noted that for a person of ordinary skill in the art, several transformations and improvements can be made without departing from the idea of this application. These transformations and improvements belong to the protection scope of this application. Therefore, the protection scope of this application is subject to the appended claims.

Claims

CLAIMS What is claimed is:

1. A device control method, characterized by comprising: obtaining a cross-modal device control signal, and obtaining control data of a plurality of modalities based on the device control signal, wherein the modalities comprise at least an image, audio, and text; encoding the control data of each modality, to generate a semantic vector corresponding to each modality; performing cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based on a multi-head attention mechanism, to obtain a fused semantic vector; and determining a control instruction based on the fused semantic vector, and outputting the control instruction.

2. The method according to claim 1, characterized in that the cross-modal device control signal is an audio and video signal associated with a control behavior; and the obtaining control data of a plurality of modalities based on the device control signal comprises: extracting audio data and video data in the audio and video signal; performing speech recognition on the audio data, to generate text data; and selecting a video frame comprised in the video data as image data.

3. The method according to claim 2, characterized in that the semantic vectors comprise at least an image semantic vector, an audio semantic vector, and a text semantic vector; and the performing cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based on a multi-head attention mechanism, to obtain a fused semantic vector comprises: performing vector integration on image semantic vectors corresponding to a plurality of pieces of image data when the plurality of pieces of image data are obtained from a plurality of video frames, to obtain an integrated image semantic vector; and performing cross-modal attention calculation on the audio semantic vector, the text semantic vector, and the integrated image semantic vector based on the multi-head attention mechanism, to obtain the fused semantic vector.

4. The method according to claim 1, characterized in that the performing cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based ona multi-head attention mechanism, to obtain a fused semantic vector comprises: separately performing multilayer single-head attention calculation on the semantic vector corresponding to each modality, to obtain a single-modal attention vector corresponding to each modality; separately performing cross-modal attention calculation on any two of the semantic vectors corresponding to the plurality of modalities, to obtain a plurality of cross-modal attention vectors; and generating the fused semantic vector based on the single-modal attention vector corresponding to each modality and the plurality of cross-modal attention vectors.

5. The method according to claim 4, characterized in that the separately performing multilayer single-head attention calculation on the semantic vector corresponding to each modality, to obtain a single-modal attention vector corresponding to each modality comprises: using the semantic vector corresponding to each modality as a first layer of vector; sequentially merging sub-vectors in a plurality of adjacent rows in a previous layer of vector, to obtain a new layer of vector, until a quantity of rows of the new layer of vector reaches a preset value; performing single-head attention calculation on each layer of vector, to obtain an attention vector corresponding to each layer; and obtaining the single-modal attention vector corresponding to each modality based on the attention vector corresponding to each layer.

6. The method according to claim 4, characterized in that the cross-modal attention calculation comprises: selecting a semantic vector corresponding to a first modality as a query vector, and selecting a semantic vector corresponding to a second modality as a key and value vector; performing dot product calculation on the query vector and the key vector, to obtain a similarity score; performing similarity normalization on the similarity score, to obtain an attention weight matrix; and applying the attention weight matrix to the value vector, to obtain the cross-modal attention vector of the first modality relative to the second modality.

7. The method according to claim 4, characterized in that the generating the fused semantic vector based on the single-modal attention vector corresponding to each modality and the plurality of cross-modal attention vectors comprises: concatenating the single-modal attention vector corresponding to each modality and theplurality of cross-modal attention vectors, to obtain a composite attention vector; and multiplying the composite attention vector by a preset fused weight matrix, to generate the fused semantic vector.

8. The method according to claim 1, characterized in that the determining a control instruction based on the fused semantic vector comprises: inputting the fused semantic vector to a functional classifier on which pretraining is completed, to obtain target category information; and determining a control instruction corresponding to the target category information.

9. A device control apparatus, characterized by comprising: a data obtaining module, configured to: obtain a cross-modal device control signal, and obtain control data of a plurality of modalities based on the device control signal, wherein the modalities comprise at least an image, audio, and text; a data encoding module, configured to encode the control data of each modality, to generate a semantic vector corresponding to each modality; an attention calculation module, configured to perform cross-modal attention calculation on semantic vectors corresponding to the plurality of modalities based on a multi-head attention mechanism, to obtain a fused semantic vector; and an instruction output module, configured to determine a control instruction based on the fused semantic vector, and output the control instruction.

10. An appliance device, characterized by comprising a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 8.

11. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, storing a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images