Intelligent household electrical appliance AI engine system and multi-modal data fusion method thereof

Abstract

The invention provides an intelligent household electrical appliance AI engine system and a multi-modal data fusion method thereof, and the system comprises a plurality of different types of collection units which are used for carrying out the data collection of the space information of a household space where the intelligent household electrical appliance AI engine system is arranged, so as to collect the information data of various modals; the edge node is in communication connection with the plurality of acquisition units, and is used for acquiring the multi-modal fusion information and the trained decision model, reasoning decision service data according to the decision model and the multi-modal fusion information, and sending the decision service data to a third-party service platform in communication connection with the edge node; wherein the multi-modal fusion information is obtained by performing data fusion on information data of multiple modals with the same language meaning. According to the intelligent household electrical appliance AI engine system and the multi-modal data fusion method thereof, the privacy of the original data of the user can be improved, and the cost of data transmission in the intelligent household electrical appliance AI engine system is reduced.

Description

Technical Field

[0001] The present invention relates to the field of smart home appliances, and particularly to a smart home appliance AI engine system and a multi-modal data fusion method thereof. Background Art

[0002] With the development of technology and the popularity of intelligent home, more and more families begin to use various sensor devices and perform intelligent control on home appliances. For example, various sensors are used to detect environmental variables such as indoor temperature, humidity, and carbon dioxide concentration, so as to achieve intelligent adjustment; various sensors are used to detect data such as room light intensity, human position, and sound, so as to achieve intelligent lighting control, security monitoring, etc.; through data from various sensors such as voice recognition, human detection, and temperature sensors, intelligent lighting, intelligent air conditioner control, etc. can be achieved.

[0003] Currently, for the home appliances in the prior art, the controllers provided thereon pre-store control programs for generating corresponding control instructions according to environmental information or data. Thus, the home appliances can automatically serve users. However, the control programs are stored in the controllers of the home appliances after the home appliances leave the factory and cannot be adaptively modified, and can only perform simple automatic control and cannot perform more complex functions, failing to meet the needs of users. Furthermore, the home appliance systems in the prior art adopt a solution of adding a cloud server. Since the cloud server has powerful computing capabilities, more functions can be realized, and even function upgrades can be completed according to the home environment or user needs, meeting more needs of users. Such a home appliance system will directly send the environmental information or data collected by its sensors to the cloud server, and the cloud server will generate control instructions according to the environmental information or data, and the cloud server will send the control instructions to the corresponding home appliances, and the home appliances will then complete the actions corresponding to the control instructions. However, some of the environmental information or data collected by the sensors involve a large amount of user privacy. In this way, after these original data are uploaded to the cloud server, the privacy of the user's original data is difficult to guarantee; and because the amount of environmental information or data collected by the sensors is large, or there are duplicates in the data content between multiple sensors, the cost of data transmission is very high. Summary of the Invention

[0004] An object of the present invention is to provide a smart home appliance AI engine system based on an AI (Artificial Intelligence) engine and a multi-modal data fusion method thereof that can solve at least one of the above-mentioned defects in the prior art.

[0005] A further object of the present invention is to improve the privacy of the original data of the users of the smart home appliance AI engine system.

[0006] Another further object of the present invention is to reduce the cost of data transmission in the AI engine system of smart home appliances.

[0007] In particular, the present invention provides an AI engine system for smart home appliances, comprising:

[0008] A plurality of different types of acquisition units for collecting spatial information of the home space arranged by the AI engine system of smart home appliances to collect multi-modal information data;

[0009] An edge node communicatively connected to a plurality of acquisition units for obtaining multi-modal fusion information and a trained decision model, inferring decision service data according to the decision model and the multi-modal fusion information, and sending the decision service data to a third-party service platform communicatively connected to the edge node, and the third-party service platform is used to convert the decision service data into control instructions for household appliances in the home space; wherein, the multi-modal fusion information is obtained by fusing information data of multiple modalities having the same language meaning.

[0010] Furthermore, the plurality of acquisition units are further used to respectively extract the shallow feature data of the information data collected by each of them, and after projecting the extracted plurality of shallow feature data into a low-dimensional semantic space, send the plurality of shallow feature data to the edge node;

[0011] The edge node is further used to receive the plurality of shallow feature data, and is used to extract the shared features of the plurality of shallow feature data, and respectively extract the unique features of each of the shallow feature data, and after fusing the plurality of unique features, fuse the fused unique features with the shared features to obtain multi-modal fusion information.

[0012] Furthermore, the edge node uses the encoder in the Lite Transformer model to extract the shared features of the plurality of shallow feature data, and uses the encoder in the reversible neural network model to respectively extract the unique features of each of the shallow feature data.

[0013] Furthermore, the acquisition unit for which the collected information data is image data uses the encoder in the DEiT (Data-efficient Image Transformers) model to extract the shallow feature data;

[0014] The acquisition unit for which the collected information data is voice signal data uses the encoder in the wav2vec model to extract the shallow feature data;

[0015] The acquisition unit for the information data collected as light sensor data uses a convolutional neural network model to extract shallow feature data.

[0016] Further, the edge node is also used to perform data reconstruction on the shared features and the respective unique features of each shallow feature data after extracting the shared features of multiple shallow feature data and the unique features of each shallow feature data respectively.

[0017] Further, the intelligent home appliance AI engine system includes:

[0018] A cloud server, communicatively connected to the edge node, for sending a trained decision model to the edge node.

[0019] Further, the edge node is also used to upload the multi-modal fusion information to the cloud server after fusing the fused unique features and the shared features, so as to facilitate the cloud server to update and train the decision model.

[0020] In particular, the present invention also provides a multi-modal data fusion method for an intelligent home appliance AI engine system, which is applied to the above intelligent home appliance AI engine system; and,

[0021] The multi-modal data fusion method includes:

[0022] Multiple different types of acquisition units collect spatial information of the home space arranged by the intelligent home appliance AI engine system to collect multi-modal information data;

[0023] The edge node obtains multi-modal fusion information, where the multi-modal fusion information is obtained by fusing information data of multiple modalities with the same linguistic meaning;

[0024] The edge node obtains a trained decision model;

[0025] The edge node infers decision service data according to the decision model and the multi-modal fusion information;

[0026] The edge node sends the decision service data to a third-party service platform communicatively connected to the edge node, and the third-party service platform is used to convert the decision service data into control instructions for household appliances in the home space.

[0027] Further, before the steps of the edge node obtaining the multi-modal fusion information and the trained decision model, the multi-modal data fusion method further includes:

[0028] Multiple acquisition units respectively extract shallow feature data of the information data collected by each of them to extract multiple shallow feature data;

[0029] Multiple acquisition units project the multiple shallow feature data they extract into a low-dimensional semantic space;

[0030] Multiple acquisition units send the multiple shallow feature data to the edge node; and,

[0031] The steps for the edge node to obtain the multimodal fusion information include:

[0032] The edge node receives the multiple shallow feature data;

[0033] The edge node extracts the shared features of the multiple shallow feature data, and respectively extracts the unique features of each shallow feature data;

[0034] The edge node fuses the multiple unique features;

[0035] The edge node fuses the fused unique features and the shared features to obtain the multimodal fusion information.

[0036] Further, after the steps in which the edge node extracts the shared features of the multiple shallow feature data and respectively extracts the unique features of each shallow feature data, the multimodal data fusion method further includes:

[0037] The edge node performs data reconstruction on the shared features and each unique feature respectively.

[0038] In the intelligent home appliance AI engine system of the present invention, since an edge node is added that can infer decision service data based on a decision model and multimodal fusion information, and the edge node can also send the decision service data to a third-party service platform communicatively connected to the edge node, and the third-party service platform is used to convert the decision service data into control instructions for household appliances in the home space. Furthermore, the intelligent home appliance AI engine system of the present invention can control the corresponding home appliance devices according to the space information of the home space and a more intelligent decision model without uploading the information data collected by the acquisition unit to the network side (for example, a cloud server, etc.), so that the home appliance devices complete corresponding actions to provide services with more complex and comprehensive functions to the user. Therefore, the intelligent home appliance AI engine system of the present invention can improve the privacy of the user's original data.

[0039] Further, in the intelligent home appliance AI engine system of the present invention, since the edge node can obtain multimodal fusion information obtained by fusing information data based on multiple modalities with the same language meaning, when the edge node transmits data to other modules (for example, a cloud server, etc.), it does not need to transmit the original information data of multiple modalities collected by the acquisition unit, and can directly transmit the multimodal fusion information. Therefore, the intelligent home appliance AI engine system of the present invention can reduce the data transmission cost in the intelligent home appliance AI engine system.

[0040] The multimodal data fusion method of the intelligent home appliance AI engine system of the present invention, since it can be applied in the above-mentioned intelligent home appliance AI engine system, thus has the same beneficial technical effects as those of the above-mentioned intelligent home appliance AI engine system.

[0041] From the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more clear about the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Some specific embodiments of the present invention will be described in detail hereinafter with reference to the accompanying drawings in an exemplary but not restrictive manner. The same reference numerals in the drawings denote the same or similar components or parts. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0043] Figure 1 is a schematic diagram of data interaction of an intelligent home appliance AI engine system according to an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of a data fusion process of an intelligent home appliance AI engine system according to an embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of a process of a multimodal data fusion method of an intelligent home appliance AI engine system according to an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of a process of a multimodal data fusion method of an intelligent home appliance AI engine system according to another embodiment of the present invention;

[0047] Figure 5 is a schematic diagram of a process of a multimodal data fusion method of an intelligent home appliance AI engine system according to still another embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] In the description of this embodiment, it should be understood that, unless otherwise clearly specified and limited, terms such as "deploy", "connect", etc. should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or integrated; it can be a mechanical connection, an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal communication of two components or the interaction relationship between two components, unless otherwise clearly limited. Those of ordinary skill in the art should be able to understand the specific meanings of the above terms in the present invention according to specific circumstances.

[0049] Unless otherwise defined, all terms (including technical and scientific terms) used in the description of this embodiment have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs.

[0050] In the description of this embodiment, the descriptions referring to terms such as "this embodiment" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0051] The following will Figures 1 to 2 be used to detail the intelligent home appliance AI engine system of the present invention. Figure 1 is a schematic diagram of data interaction of the intelligent home appliance AI engine system according to an embodiment of the present invention. Figure 2 is a schematic diagram of the data fusion process of the intelligent home appliance AI engine system according to an embodiment of the present invention.

[0052] Referring to Figure 1 , in this embodiment, the intelligent home appliance AI engine system includes a plurality of different types of acquisition units 100 and edge nodes 200. The plurality of different types of acquisition units 100 are used to collect data on the spatial information of the home space where the intelligent home appliance AI engine system is arranged, so as to collect various modal information data; the edge nodes 200 are communicatively connected to the plurality of acquisition units 100, and the edge nodes 200 are used to obtain multi-modal fusion information and a trained decision model, infer decision service data according to the decision model and the multi-modal fusion information, and send the decision service data to the third-party service platform 400 communicatively connected to the edge nodes 200, and the third-party service platform 400 is used to convert the decision service data into control instructions for household appliances in the home space; wherein, the multi-modal fusion information is obtained by fusing information data of multiple modalities with the same semantic meaning.

[0053] Since the intelligent home appliance AI engine system of the present invention adds an edge node 200 that can infer decision service data based on a decision model and multi-modal fusion information, and the edge node 200 can also send the decision service data to a third-party service platform 400 communicatively connected to the edge node 200, and the third-party service platform 400 is used to convert the decision service data into control instructions for household appliances in the home space. Furthermore, the intelligent home appliance AI engine system of this embodiment can control the corresponding household appliances according to the spatial information of the home space and a more intelligent decision model without uploading the information data collected by the acquisition unit 100 to the network side (for example, the cloud server 300, etc.), enabling the household appliances to complete corresponding actions to provide users with more complex and comprehensive services. Therefore, the intelligent home appliance AI engine system of this embodiment can enhance the privacy of the user's original data.

[0054] It can be understood that the edge node 200 can be an edge gateway or an edge server with a certain computing power. And the spatial information can be indoor information, outdoor information, weather information, and can also be environmental information. For example, temperature and humidity data information of the indoor and outdoor environments, etc., can also be the position information and physiological state information of indoor users, as well as the layout information of various electrical appliances in the room, etc. And information data of multiple modalities with the same semantic meaning, such as RGB images, voice signals, and light sensor data that all express the semantic meaning of "the weather is sunny today". And the third-party service platform can include platforms such as mobile phone APPs, mini-programs, web pages, and intelligent robots, which are used to convert decision service data into control instructions for household appliances in the home space, so as to provide users with personalized intelligent services, such as chat services, intelligent air conditioner control, scene-device intelligent linkage, etc.

[0055] Moreover, since the edge node 200 of the intelligent home appliance AI engine system of this embodiment can obtain multi-modal fusion information obtained by fusing information data of multiple modalities with the same semantic meaning, when the edge node 200 transmits data to other modules (for example, the cloud server 300, etc.), it does not need to transmit a large amount of redundant original information data of multiple modalities collected by the acquisition unit 100, and can directly transmit the multi-modal fusion information. Therefore, the intelligent home appliance AI engine system of this embodiment can reduce the data transmission cost in the intelligent home appliance AI engine system, while ensuring the integrity and accuracy of the data in the intelligent home appliance AI engine system, and improving the data utilization efficiency and transmission efficiency.

[0056] In addition, in the prior art, multi-modal data fusion technology refers to the effective fusion of data from different perception modes, so as to achieve more comprehensive, accurate and detailed data collection and real-time analysis, and improve the accuracy and efficiency of data processing. The core of multi-modal data fusion technology is the design of data fusion algorithms. First, different modal data are processed and extracted. Then, the associations between different modal data are analyzed to fuse the information between modalities. The implementation of multi-modal fusion perception technology can adopt different methods and techniques, including sensor (data) fusion, feature fusion, decision fusion, etc. Among them, sensor fusion is to fuse the raw data from multiple sensors together to provide more comprehensive information. Feature fusion is to convert various types of sensor data into the same type of feature vectors for comparison and fusion. Decision fusion is to fuse the decision results of data from multiple modalities to determine the most likely scenario or event. The disadvantages of data fusion are that there are large dimensional differences between modal data, making it difficult to directly perform data fusion; feature fusion is simple but only a low-level information fusion; decision fusion requires pre-training multiple decision models in the training stage, thus increasing the computational cost of the fusion model.

[0057] The data fusion method based on traditional machine learning uses manually crafted features for fusion classification. This method is often designed based on specific domain knowledge, which limits its portability across different modal data. In contrast, deep learning-based techniques can extract discriminative features from shallow to deep abstractions from raw data, and have better portability than traditional models. Currently, deep learning-based techniques have become an important technical means for multi-modal data fusion.

[0058] There are mainly two types of multi-modal data fusion based on deep learning, namely the multi-branch feature stacking fusion strategy and the context-aware feature cross-fusion strategy. The former uses multiple branch networks to independently learn heterogeneous features of each modality. The context-aware feature cross structure simultaneously learns multi-modal features through context-aware cross for each modality, which is more comprehensive and sufficient in multi-modal feature expression. There is also an intermediate fusion method that uses neural networks to transform different modal data into high-dimensional feature expressions, and then learns the commonalities between modalities in the high-dimensional space. However, different modal features have both common and unique parts in semantic representation. Most of the aforementioned fusion networks are only fusions of low-level global shared information between modalities or high-level local unique information of modalities, either ignoring the common information between modalities or ignoring the unique information of each modality, thus unable to obtain a more comprehensive and detailed modal description.

[0059] To address at least one of the above technical deficiencies in the prior art, refer to Figure 1 and Figure 2, in this embodiment, the multiple acquisition units 100 are further configured to separately extract the shallow feature data of the information data collected by each of them, and after projecting the multiple extracted shallow feature data into a low-dimensional semantic space, send the multiple shallow feature data to the edge node 200; the edge node 200 is further configured to receive the multiple shallow feature data, extract the shared features of the multiple shallow feature data, and separately extract the unique features of each shallow feature data, and after fusing the multiple unique features, fuse the fused unique features with the shared features to obtain multi-modal fusion information.

[0060] It can be known that the process of obtaining multi-modal fusion information by fusing information data of multiple modalities with the same language meaning in the intelligent home appliance AI engine system of this embodiment can be that the acquisition unit 100 separately extracts the shallow feature data of the information data collected by each of them, and the acquisition unit 100 projects the shallow feature data into a low-dimensional semantic space by using a projection layer; and the edge node 200 extracts the shared features of the multiple shallow feature data and the unique features of each shallow feature data, and after fusing the multiple unique features by the edge node 200, fuses the fused unique features with the shared features.

[0061] Among them, the acquisition unit 100 separately extracts the shallow feature data of the information data collected by each of them, and projects the shallow feature data into a low-dimensional semantic space through the acquisition unit 100, thereby avoiding the situation that it is difficult to directly fuse the original data of each modality, and reducing the calculation amount and transmission cost in the subsequent data processing process. The projection layer can be a simple matrix multiplication (a dense linear layer without non-linear activation), and the specific matrix parameters can be learned through the training process. And the shallow feature data can describe more detailed information and local features of the information data.

[0062] Moreover, the edge node 200 extracts the shared features of multiple shallow feature data and the unique features of each shallow feature data. After fusing the multiple unique features through the edge node 200 to obtain the multi-modal unique features, the fused multi-modal unique features are fused with the shared features. It can be seen that by adopting the idea of feature decoupling, the shared features of multiple modal data and the unique features of each modal data are fused to eliminate redundant data, ensure the data display ability, and ensure the comprehensiveness and detail of the data of the intelligent home appliance AI engine system. Moreover, the multi-modal fusion information obtained after fusion can be high-level information, so that the multi-modal fusion information can better reflect the essence and laws of the data, so as to make the multi-modal fusion information accurate and efficient in the subsequent transmission and calculation (for example, inferring decision service data according to the decision model and multi-modal fusion information); at the same time, the portability of the data is ensured, as well as the accuracy and reliability of the subsequent decision model inference.

[0063] In addition, the edge node 200 can fuse multiple unique features by using the decoder in the reversible neural network model to obtain multi-modal unique features. And the edge node 200 can perform dimensional splicing of the fused multi-modal unique features and the shared features in the channel by using the decoder in the Transformer model to obtain multi-modal fusion information.

[0064] Referring to Figure 2 , in this embodiment, the edge node 200 uses the encoder in the Lite Transformer model to extract the shared features of multiple shallow feature data, that is, the encoder in the Lite Transformer model is used as a global encoder to extract the shared features of multiple shallow feature data; and the edge node 200 uses the encoder in the reversible neural network model to extract the unique features of each shallow feature data respectively, that is, a reversible neural network is introduced into the local encoder to extract the unique features of each shallow feature data respectively.

[0065] It should be understood that since the Lite Transformer model based on spatial attention can capture long-distance dependent features, the common parts of each input data can be obtained. Moreover, the Lite Transformer model is a lightweight Transformer model with high computational efficiency and low computational performance requirements for the required modules. Furthermore, the shared features of multiple shallow feature data can be extracted through the encoder in the Lite Transformer model, and the high efficiency of data processing of the intelligent home appliance AI engine system in this embodiment can be effectively ensured, and the deployment cost of the intelligent home appliance AI engine system can be reduced. And in the process of using the encoder in the reversible neural network model to extract the unique features of each shallow feature data respectively, the corresponding shallow feature data can be restored according to the unique features to ensure the accuracy of the extracted unique features. And the extracted unique features can well retain the detailed information of the shallow feature data. And the reversible neural network model can be lightweight with low memory consumption, and thus the deployment cost of the intelligent home appliance AI engine system can be reduced.

[0066] In addition, the reversible neural network model can be combined with Affine Coupling Layers to separately extract the unique features of each shallow feature data, so as to further improve the accuracy of extracting unique features.

[0067] Moreover, in addition to the Lite Transformer model, the edge node 200 can also extract the shared features of multiple shallow feature data through the encoders in models such as VisualBERT (Vision-and-Language BERT), VL-BERT (Visual-Linguistic BERT), or TinyBERT (Distilling BERT for Natural Language Understanding); in addition to the encoders in the reversible neural network model, the edge processing center can also use the encoders in models such as ResNet (Residual Network), VGGNet (Visual Geometry Group Network), or Inception model (Inception Module) to separately extract the unique features of each shallow feature data.

[0068] In this embodiment, the acquisition unit 100 may be an infrared camera, a voice remote control, a millimeter-wave radar sensor, a temperature sensor, or a sensor carried by a household appliance, etc., which has a certain data processing ability.

[0069] Referring to Figure 2 , in this embodiment, the acquisition unit 100 whose acquired information data is image data uses the encoder in the DEiT model to extract shallow feature data; the acquisition unit 100 whose acquired information data is voice signal data uses the encoder in the wav2vec model to extract shallow feature data; the acquisition unit 100 whose acquired information data is light sensor data uses a convolutional neural network model to extract shallow feature data.

[0070] It should be understood that the acquisition unit 100 can determine the neural network model it needs according to the characteristics of the data it acquires to effectively model multi-modal data, that is, to prompt each acquisition unit 100 to extract shallow feature data from the data it acquires, and further extract unique features and shared features. Among them, the DEiT model is lightweight and has low memory consumption to ensure that the intelligent home appliance AI engine system can have a low deployment cost. And for light sensor data, it needs to be transformed into image data through wavelet transform, and then the convolutional neural network model is used to extract the shallow feature data of the acquired information data. For example, it is further processed through LeNet-5, AlexNet (Alex neural network), or VGGNet (Visual Geometry Group Network) to extract the shallow feature data of the acquired information data.

[0071] In addition, the acquisition unit 100 whose acquired information data is image data can also use the encoder in the VisionTransformer (visual transformer) model to extract shallow feature data; the acquisition unit 100 whose acquired information data is voice signal data can also use the encoder in MobileNetV2 (mobile neural network V2) or SpeechTransformer (speech transformer) to extract shallow feature data.

[0072] Referring to Figure 2 , in this embodiment, the edge node 200 is also used to perform data reconstruction on the shared features and each unique feature respectively after extracting the shared features of multiple shallow feature data and extracting the unique features of each shallow feature data respectively.

[0073] It can be understood that the shared features and each specific feature can be respectively reconstructed through a modal data reconstruction decoder, and the purpose of this decoder is to ensure that the original modal data (information data) can be restored from the encoded shared features and specific features, so as to ensure the accuracy of the encoded shared features and specific features.

[0074] In addition, the design of the modal data reconstruction decoder needs to refer to the encoder design in each acquisition unit 100. For example, when the acquisition unit 100 uses the encoder in the DEiT model to extract shallow feature data, the data reconstruction decoder in the DEiT model can be used to reconstruct the shared features and the corresponding specific features; when the acquisition unit 100 uses the encoder in the wav2vec model to extract shallow feature data, the data reconstruction decoder in the wav2vec model can be used to reconstruct the shared features and the corresponding specific features; when the acquisition unit 100 uses a convolutional neural network model to extract shallow feature data, the convolutional neural network model can be used to reconstruct the shared features and the corresponding specific features.

[0075] Refer to Figure 1 , in this embodiment, the intelligent home appliance AI engine system includes a cloud server 300. The cloud server 300 is communicatively connected to the edge node 200, and the cloud server 300 is used to send the trained decision model to the edge node 200.

[0076] It can be understood that the decision model obtained by the edge node 200 can be obtained from the cloud server 300, so as to prompt the intelligent home appliance AI engine system to complete the control of the corresponding home appliance device according to the collected data.

[0077] In addition, the cloud server 300 also has functions such as resource collaboration and data collaboration.

[0078] Refer to Figure 1 , in this embodiment, the edge node 200 is further used to upload the multi-modal fusion information to the cloud server 300 after fusing the fused specific features and shared features, so that the cloud server 300 can update and train the decision model.

[0079] It can be understood that since the training of the decision model requires a large amount of computing power, the training and update of the decision model can be deployed on the cloud server 300 for execution. Moreover, even if the cloud server 300 needs to train the decision model, it only needs to upload the multi-modal fusion information to the cloud server 300, and it is not necessary to upload the information data collected by the acquisition unit 100 to the cloud server 300. Furthermore, the intelligent home appliance AI engine system in this embodiment can further ensure the privacy of the user's original data.

[0080] The following will be combined with Figures 3 to 5 to describe in detail the multi-modal data fusion method of the intelligent home appliance AI engine system of this embodiment.

[0081] Figure 3 is a schematic flowchart of the multi-modal data fusion method of the intelligent home appliance AI engine system according to an embodiment of the present invention. Referring to Figure 3 , in this embodiment, the multi-modal data fusion method of the intelligent home appliance AI engine system can be applied to the intelligent home appliance AI engine system of the above embodiment; and the multi-modal data fusion method includes:

[0082] Step S302, multiple different types of acquisition units collect spatial information of the home space arranged by the intelligent home appliance AI engine system to collect information data of multiple modalities.

[0083] Step S304, the edge node obtains multi-modal fusion information. The multi-modal fusion information is obtained by fusing information data of multiple modalities with the same semantic meaning.

[0084] Step S306, the edge node obtains a trained decision model.

[0085] Step S308, the edge node infers decision service data according to the decision model and the multi-modal fusion information.

[0086] Step S310, the edge node sends the decision service data to a third-party service platform communicatively connected to the edge node, and the third-party service platform is used to convert the decision service data into control instructions for household appliances in the home space.

[0087] It can be understood that since the multi-modal data fusion method of the intelligent home appliance AI engine system of this embodiment can be applied to the intelligent home appliance AI engine system implemented above, the beneficial technical effects that can be achieved by the intelligent home appliance AI engine system of the above embodiment are also possessed by the multi-modal data fusion method of the intelligent home appliance AI engine system of this embodiment. And the beneficial technical effects that can be achieved by the corresponding steps of this embodiment and the following embodiments have been described in detail at the corresponding hardware devices of the intelligent home appliance AI engine system of the above embodiment, and will not be elaborated here.

[0088] Figure 4 is a schematic flowchart of the multi-modal data fusion method of the intelligent home appliance AI engine system according to another embodiment of the present invention. Referring to Figure 4 , in this embodiment, the multi-modal data fusion method of the intelligent home appliance AI engine system includes:

[0089] Step S402: Multiple different types of acquisition units collect spatial information of the home space arranged by the smart home appliance AI engine system to collect various modalities of information data.

[0090] Step S404: Multiple acquisition units respectively extract the shallow feature data of the information data they collected to extract multiple shallow feature data.

[0091] Step S406: Multiple acquisition units project the multiple shallow feature data they extracted into a low-dimensional semantic space.

[0092] Step S408: Multiple acquisition units send the multiple shallow feature data to the edge node.

[0093] Step S410: The edge node receives the multiple shallow feature data.

[0094] Step S412: The edge node extracts the shared features of the multiple shallow feature data and respectively extracts the unique features of each shallow feature data.

[0095] Step S414: The edge node fuses the multiple unique features.

[0096] Step S416: The edge node fuses the fused unique features and the shared features to obtain multi-modal fusion information.

[0097] Figure 5 It is a schematic flowchart of the multi-modal data fusion method of the smart home appliance AI engine system according to another embodiment of the present invention. Refer to Figure 5 , in this embodiment, after the step where the edge node extracts the shared features of the multiple shallow feature data and respectively extracts the unique features of each shallow feature data, the multi-modal data fusion method further includes:

[0098] Step S502: Multiple different types of acquisition units collect spatial information of the home space arranged by the smart home appliance AI engine system to collect various modalities of information data.

[0099] Step S504: Multiple acquisition units respectively extract the shallow feature data of the information data they collected to extract multiple shallow feature data.

[0100] Step S506: Multiple acquisition units project the multiple shallow feature data they extracted into a low-dimensional semantic space.

[0101] Step S508: Multiple acquisition units send the multiple shallow feature data to the edge node.

[0102] Step S510: The edge node receives the multiple shallow feature data.

[0103] In step S512, the edge node extracts the shared features of multiple shallow feature data and respectively extracts the unique features of each shallow feature data.

[0104] In step S514, the edge node performs data reconstruction on the shared features and each unique feature respectively.

[0105] It can be understood that in this multi-modal data fusion method, after performing data reconstruction on the shared features and each unique feature respectively in step S514, it is possible to verify whether the reconstructed information data is consistent with the original information data based on the original information data. If they are consistent, the step S414 in the above embodiment can be continued. If they are inconsistent, the step S502 in this embodiment, or the step S302 or step S402 in the above embodiment can be returned to execute, so as to further ensure the accuracy of the data during the data processing process of the intelligent home appliance AI engine system.

[0106] At this point, those skilled in the art should recognize that although multiple exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications that conform to the principles of the present invention can still be directly determined or derived from the content disclosed in the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and determined to cover all these other variations or modifications.

Claims

1. An intelligent home appliance AI engine system, comprising: Multiple different types of acquisition units, configured to collect data on the spatial information of the home space where the intelligent home appliance AI engine system is arranged, so as to collect information data of multiple modalities; An edge node, communicatively connected to multiple said acquisition units, configured to obtain multimodal fusion information and a trained decision model, infer decision service data according to the decision model and the multimodal fusion information, and send the decision service data to a third-party service platform communicatively connected to the edge node, and the third-party service platform is configured to convert the decision service data into control instructions for household appliances in the home space; wherein, the multimodal fusion information is obtained by fusing information data of multiple modalities with the same linguistic meaning.

2. The intelligent home appliance AI engine system according to claim 1, wherein, The multiple said acquisition units are further configured to respectively extract shallow feature data of the information data collected by each of them, and after projecting the multiple said shallow feature data into a low-dimensional semantic space, send the multiple said shallow feature data to the edge node; The edge node is further configured to receive the multiple said shallow feature data, and extract shared features of the multiple said shallow feature data, and respectively extract unique features of each of the shallow feature data, and after fusing the multiple said unique features, fuse the fused unique features with the shared features to obtain the multimodal fusion information.

3. The intelligent home appliance AI engine system according to claim 2, wherein, The edge node uses the encoder in the Lite Transformer model to extract the shared features of the multiple said shallow feature data, and uses the encoder in the reversible neural network model to respectively extract the unique features of each of the shallow feature data.

4. The intelligent home appliance AI engine system according to claim 2, wherein, The acquisition unit whose collected information data is image data uses the encoder in the DEiT model to extract the shallow feature data; The acquisition unit whose collected information data is voice signal data uses the encoder in the wav2vec model to extract the shallow feature data; The acquisition unit whose collected information data is light sensor data uses a convolutional neural network model to extract the shallow feature data.

5. The intelligent home appliance AI engine system according to claim 2, wherein, The edge node is further configured to perform data reconstruction on the shared features and each of the unique features respectively after extracting the shared features of the multiple said shallow feature data and respectively extracting the unique features of each of the shallow feature data.

6. The intelligent home appliance AI engine system according to claim 2, comprising: A cloud server, communicatively connected to the edge node, configured to send the trained decision model to the edge node.

7. The intelligent home appliance AI engine system according to claim 6, wherein, The edge node is further configured to upload the multimodal fusion information to the cloud server after fusing the fused specific features with the shared features, so as to facilitate the cloud server to update and train the decision model.

8. A multimodal data fusion method for an intelligent home appliance AI engine system, which is applied to the intelligent home appliance AI engine system according to any one of claims 1-7; and, The multimodal data fusion method includes: Multiple different types of the acquisition units collect data of the spatial information of the home space arranged by the intelligent home appliance AI engine system, so as to collect information data of multiple modalities; The edge node obtains multimodal fusion information, where the multimodal fusion information is obtained by fusing information data of multiple modalities with the same linguistic meaning; The edge node obtains a trained decision model; The edge node infers decision service data according to the decision model and the multimodal fusion information; The edge node sends the decision service data to a third-party service platform communicatively connected to the edge node, and the third-party service platform is configured to convert the decision service data into control instructions for household appliances in the home space.

9. The multimodal data fusion method for the intelligent home appliance AI engine system according to claim 8, wherein, Before the step of the edge node obtaining the multimodal fusion information and the trained decision model, the multimodal data fusion method further includes: Multiple of the acquisition units respectively extract shallow feature data of the information data collected by each of them, so as to extract multiple pieces of the shallow feature data; Multiple of the acquisition units project the multiple pieces of the extracted shallow feature data into a low-dimensional semantic space; Multiple of the acquisition units send the multiple pieces of the shallow feature data to the edge node; and, The step of the edge node obtaining the multimodal fusion information includes: The edge node receives the multiple pieces of the shallow feature data; The edge node extracts the shared features of the multiple pieces of the shallow feature data, and respectively extracts the specific features of each of the shallow feature data; The edge node fuses the multiple specific features; The edge node fuses the fused specific features and the shared features to obtain the multimodal fusion information.

10. The multimodal data fusion method for the intelligent home appliance AI engine system according to claim 9, wherein, After the step of the edge node extracting the shared features of the multiple pieces of the shallow feature data and respectively extracting the specific features of each of the shallow feature data, the multimodal data fusion method further includes: The edge node performs data reconstruction on the shared features and each of the specific features respectively.

Images