Expression mapping system and method based on artificial intelligence Internet of Things

Abstract

The invention discloses an expression mapping system and method based on an artificial intelligence Internet of Things, belongs to the technical field of the artificial intelligence Internet of Things, and aims to solve the technical problem of applying the artificial intelligence Internet of Things to emotional expression of meta universe so that an object in a virtual world can dynamically change according to real-time emotional change of a user. Comprising a camera, a controller, a local client, a server, an Internet of Things client, an Internet of Things server, a meta-universe interoperation module and a meta-universe server, the controller is used for performing emotion recognition on an input facial expression image through a trained emotion recognition model and filtering predicted emotion data based on an emotion list; pushing the filtered emotion data to a client; and the meta-universe server performs data interaction with the Internet of Things server through the meta-universe interoperation module, and is used for receiving and storing the filtered emotions and updating the filtered emotions to objects or roles in the virtual world.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence Internet of Things, and in particular to an expression mapping system and method based on artificial intelligence Internet of Things. Background Art

[0002] With the development of technologies such as virtual reality (VR), augmented reality (AR) and metaverse, the interaction between the virtual and real worlds is increasing. As technology develops, the demand for remote interaction and virtual social interaction has increased dramatically.

[0003] However, on the one hand, the current Metaverse platform mainly focuses on the construction of virtual scenes and the user's visual experience, lacking in-depth support for user emotional expression and interaction. In traditional virtual social environments, users express emotions through emoticons or preset actions, but these methods are usually static and indirect, and cannot accurately reflect the user's immediate emotional changes.

[0004] On the other hand, the development of Internet of Things (IoT) technology has facilitated real-time data transmission and processing, and artificial intelligence (AI) technology, especially deep learning, has made significant progress in facial expression recognition. The integration of AIoT (Artificial Intelligence Internet of Things) technology has brought new possibilities for emotion recognition and real-time data transmission.

[0005] Applying AI IoT to the emotional expression of the metaverse, so that objects in the virtual world can change dynamically according to the real-time emotional changes of users, is a technical problem that needs to be solved. Summary of the invention

[0006] The technical task of the present invention is to address the above shortcomings and provide an expression mapping system and method based on artificial intelligence Internet of Things to solve the technical problem of applying artificial intelligence Internet of Things to metaverse emotional expression and enabling objects in the virtual world to change dynamically according to the user's real-time emotional changes.

[0007] In a first aspect, the present invention provides an expression mapping system based on artificial intelligence Internet of Things, including a camera, a controller, a local client, a server, an Internet of Things client, an Internet of Things server, a metaverse interoperability module, and a metaverse server;

[0008] The camera is used to collect facial expression images and transmit the facial expression images to the controller;

[0009] The controller is electrically connected to the camera, and a trained emotion recognition model is deployed in the controller, and an emotion list is configured, for performing emotion recognition on the input facial expression image through the trained emotion recognition model, outputting predicted emotion data, filtering the predicted emotion data based on the emotion list, obtaining filtered emotion data, and pushing the filtered emotion data to the client;

[0010] The client interacts with the controller to receive filtered emotion data and send the filtered emotion to the server;

[0011] The server is connected to the client to receive and store the filtered emotion data;

[0012] The IoT client performs data interaction with the server to collect filtered emotion data from the server and push the filtered emotion data to the IoT server;

[0013] The IoT server is connected to the client to receive and store the filtered emotion data, and send the filtered emotion data to the Metaverse server through the Metaverse interoperability module;

[0014] The Metaverse server interacts with the IoT server through the Metaverse interoperability module to receive and store filtered emotions and update the filtered emotions to objects or characters in the virtual world.

[0015] Preferably, the emotion recognition model comprises an input layer, a convolution layer, a flattening layer, a fully connected layer and an output layer;

[0016] The input layer receives the facial expression image and preprocesses the input facial expression image to obtain a facial expression object of a predetermined size as the preprocessed facial expression image through the image preprocessing;

[0017] There are three convolutional layers, namely the first convolutional layer, the second convolutional layer and the third convolutional layer. The first convolutional layer is used to extract low-level features of the input preprocessed facial expression image and output a feature map. The second convolutional layer is used to extract high-level features of the input image using the feature map output by the first convolutional layer as the input image and output a feature map. The third convolutional layer is used to extract features of the input image using the feature map output by the second convolutional layer as the input image and output a feature map.

[0018] The flattening layer is used to take the feature map output by the third convolutional layer as input and flatten the multi-dimensional feature map into a one-dimensional vector;

[0019] The fully connected layer is used to take the one-dimensional vector output by the flattening layer as input, map the one-dimensional vector to a high-dimensional space, and output a feature map in combination with the global features;

[0020] The output layer uses the feature map output by the fully connected layer as an input image, is used to perform emotion classification through a Softmax activation function, and outputs a probability distribution of each emotion category.

[0021] Preferably, the first convolution layer, the second convolution layer and the third convolution layer all include a two-dimensional convolution layer Conv2D Layer, an activation function layer ReLU Activation, a pooling layer Max Pooling2D and a dropout layer Dropout, the two-dimensional convolution layer is used to perform a convolution operation on the input image through the Conv2D Layer to extract features, the activation function layer is used to introduce nonlinearity through the ReLU activation function, the pooling layer is used to provide a pooling operation, and the dropout layer is used to prevent the model from overfitting by randomly discarding neurons.

[0022] Preferably, the controller is used to push the filtered emotion data to the client via Socket communication, and the Metaverse interoperability module is used to send the filtered emotion data to the Metaverse server via the Raspberry Juice plug-in, and the Metaverse server is a Minecraft server.

[0023] In a second aspect, the present invention provides an expression mapping method based on artificial intelligence internet of things, which is used to implement expression mapping through an expression mapping system based on artificial intelligence internet of things as described in any one of the first aspects, and the method comprises the following steps:

[0024] Collecting facial expression images through a camera and transmitting the facial expression images to a controller;

[0025] The trained emotion recognition model deployed in the controller is used to perform emotion recognition on the input facial expression image, output predicted emotion data, and filter the predicted emotion data based on the emotion list deployed in the controller to obtain filtered emotion data, and push the filtered emotion data to the client;

[0026] receiving filtered sentiment data through the client and sending the filtered sentiment to the server;

[0027] receiving and storing the filtered emotion data via a server;

[0028] Collect filtered emotion data from the server through the IoT client, and push the filtered emotion data to the IoT server;

[0029] Receive and store the filtered emotion data through the IoT server, and send the filtered emotion data to the Metaverse server through the Metaverse interoperability module;

[0030] The filtered emotions are received and stored via the Metaverse server, and the filtered emotions are updated to objects or characters in the virtual world.

[0031] Preferably, the emotion recognition model comprises an input layer, a convolution layer, a flattening layer, a fully connected layer and an output layer;

[0032] The input layer receives the facial expression image and preprocesses the input facial expression image to obtain a facial expression object of a predetermined size as the preprocessed facial expression image through the image preprocessing;

[0033] There are three convolutional layers, namely the first convolutional layer, the second convolutional layer and the third convolutional layer. The first convolutional layer is used to extract low-level features of the input preprocessed facial expression image and output a feature map. The second convolutional layer is used to extract high-level features of the input image using the feature map output by the first convolutional layer as the input image and output a feature map. The third convolutional layer is used to extract features of the input image using the feature map output by the second convolutional layer as the input image and output a feature map.

[0034] The flattening layer is used to take the feature map output by the third convolutional layer as input and flatten the multi-dimensional feature map into a one-dimensional vector;

[0035] The fully connected layer is used to take the one-dimensional vector output by the flattening layer as input, map the one-dimensional vector to a high-dimensional space, and output a feature map in combination with the global features;

[0036] The output layer uses the feature map output by the fully connected layer as an input image, is used to perform emotion classification through a Softmax activation function, and outputs a probability distribution of each emotion category.

[0037] Preferably, the first convolution layer, the second convolution layer and the third convolution layer all include a two-dimensional convolution layer Conv2D Layer, an activation function layer ReLU Activation, a pooling layer Max Pooling2D and a dropout layer Dropout, the two-dimensional convolution layer is used to perform a convolution operation on the input image through the Conv2D Layer to extract features, the activation function layer is used to introduce nonlinearity through the ReLU activation function, the pooling layer is used to provide a pooling operation, and the dropout layer is used to prevent the model from overfitting by randomly discarding neurons.

[0038] Preferably, the controller pushes the filtered emotion data to the client via Socket communication, and the Metaverse interoperability module sends the filtered emotion data to the Metaverse server via the Raspberry Juice plug-in, and the Metaverse server is a Minecraft server.

[0039] The expression mapping system and method based on artificial intelligence Internet of Things of the present invention have the following advantages:

[0040] 1. Integration of AI IoT and Metaverse: AI IoT devices can capture users’ facial expressions and use deep learning models to identify users’ emotional data. The emotional data is transmitted to the Metaverse server through the IoT platform. The emotional data is mapped to virtual objects in the Metaverse, realizing the interaction between emotions and virtual scenes. The combination of IoT devices and Metaverse significantly expands the interactive experience between users and Metaverse.

[0041] 2. Emotion filtering algorithm: In order to avoid inconsistency between the emotion data in the Metaverse and the actual emotion of the user, the emotion data predicted by the deep learning model is filtered through the emotion list deployed on the controller. The emotion in the emotion list can be a specific emotion specified by the user. Only the specified emotion is allowed to be transmitted to the Metaverse, solving the problem of accuracy of emotion expression.

[0042] 3. Real-time emotion mapping: It realizes the real-time capture and mapping of user emotions. The slight changes in the user's facial expression can be quickly reflected in the virtual objects in the Metaverse (such as "anger" is mapped to flames, "happiness" is mapped to flowers). This instant feedback greatly enhances the emotional interactivity and user immersion in the Metaverse.

[0043] 4. Application scalability: The system can be used in the Metaverse social platform, and can also be expanded to multiple fields such as education, games, and medical care. For example, it can adjust teaching strategies by identifying students' emotional states, or improve medical services by identifying patients' emotional changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] The present invention is further described below in conjunction with the accompanying drawings.

[0046] Figure 1 This is a structural block diagram of an expression mapping system based on artificial intelligence Internet of Things in Example 1. DETAILED DESCRIPTION

[0047] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0048] The embodiments of the present invention provide an expression mapping system and method based on artificial intelligence Internet of Things, which are used to solve the technical problem of applying artificial intelligence Internet of Things to metaverse emotional expression so that objects in the virtual world can change dynamically according to the real-time emotional changes of users.

[0049] Embodiment 1:

[0050] The present invention discloses an expression mapping system based on artificial intelligence Internet of Things, comprising a camera, a controller, a local client, a server, an Internet of Things client, an Internet of Things server, a metaverse interoperability module and a metaverse server.

[0051] The camera is used to collect facial expression images and transmit the facial expression images to the controller.

[0052] As a specific implementation of the camera, a high-definition camera is used. The camera can be electrically connected to the controller via a USB interface or other methods to send the collected facial expression images to the controller.

[0053] The controller is electrically connected to the camera. A trained emotion recognition model is deployed in the controller and an emotion list is configured. The controller is used to perform emotion recognition on an input facial expression image through the trained emotion recognition model, output predicted emotion data, filter the predicted emotion data based on the emotion list, obtain filtered emotion data, and push the filtered emotion data to the client.

[0054] As a specific implementation of the emotion recognition model, it includes an input layer, a convolution layer, a flattening layer, a fully connected layer, and an output layer.

[0055] The input layer receives the facial expression image and preprocesses the input facial expression image, and obtains a facial expression object of a predetermined size as the preprocessed facial expression image through the image preprocessing.

[0056] There are three convolutional layers, namely the first convolutional layer, the second convolutional layer and the third convolutional layer. The first convolutional layer is used to perform low-level feature extraction on the input preprocessed facial expression image and output a feature map. The second convolutional layer is used to use the feature map output by the first convolutional layer as the input image, perform high-level feature extraction on the input image, and output a feature map. The third convolutional layer is used to use the feature map output by the second convolutional layer as the input image, perform feature extraction on the input image, and output a feature map.

[0057] As a specific implementation of the convolutional layer, the first convolutional layer includes the following structure:

[0058] Conv2D Layer: Performs convolution operations on the input image to extract low-level features such as edges and textures;

[0059] ReLU Activation: Introducing nonlinearity through the ReLU activation function;

[0060] Max Pooling2D: performs pooling operations to reduce the dimension of feature maps and retain the main features;

[0061] Dropout: Prevent the model from overfitting by randomly dropping some neurons.

[0062] The second convolutional layer includes the following structure:

[0063] Conv2D Layer: further extracts high-level features of the image;

[0064] ReLU Activation: reintroduces nonlinearity;

[0065] Max Pooling2D: Continues the pooling operation to reduce the feature dimension;

[0066] Dropout: Dropout is applied again to enhance the generalization ability of the model.

[0067] The third convolutional layer includes the following structure:

[0068] Conv2D Layer: further deepens feature extraction;

[0069] ReLU Activation: Use ReLU to activate nonlinear features;

[0070] Max Pooling2D: further pooling to reduce the dimension of features;

[0071] Dropout: Perform a dropout operation again before feature extraction is completed.

[0072] The flattening layer is used to take the feature map output by the third convolutional layer as input and flatten the multi-dimensional feature map into a one-dimensional vector.

[0073] The fully connected layer is used to take the one-dimensional vector output by the flattening layer as input, map the one-dimensional vector to a high-dimensional space, and output a feature map in combination with the global features.

[0074] The output layer uses the feature map output by the fully connected layer as the input image, is used to classify emotions through the Softmax activation function, and outputs the probability distribution of each emotion category.

[0075] The model uses a CNN model to predict seven emotion categories. It receives image input from a webcam, the image size is resized to 48x48, a two-dimensional convolutional layer is used to extract features from the input image, and a dropout layer is used to prevent overfitting. The facial expression recognition data collected through the webcam can go through an emotion filtering process to prevent unexpected facial expression recognition data from being transmitted, because even small changes in facial expressions can be detected and transmitted to the server. This filtering process allows users to filter out unwanted emotions and prevent such problems.

[0076] The client exchanges data with the controller to receive filtered emotion data and send the filtered emotion to the server.

[0077] The server is connected to the client to receive and store the filtered emotion data.

[0078] In this embodiment, the controller is used to push the filtered emotion data to the client through Socket communication. The client can be marked as a client Socket, and the corresponding server is marked as a server Socket.

[0079] The IoT client interacts with the server to collect filtered emotion data from the server and push the filtered emotion data to the IoT server.

[0080] The IoT server is connected to the client to receive and store the filtered emotion data.

[0081] The Metaverse server receives and stores the filtered emotions from the IoT server through the Metaverse interoperability module, and updates the filtered emotions to objects or characters in the virtual world.

[0082] In this embodiment, the Metaverse server obtains data from the Internet of Things server by means of message subscription through the Metaverse interoperability module. The Metaverse server subscribes to the emotion data of a specified topic through the Metaverse interoperability module. After the emotion data of the corresponding topic is uploaded to the Internet of Things server, the Metaverse server reads the emotion data of the relevant topic from the Internet of Things server through the Metaverse interoperability module.

[0083] As a specific implementation, the Metaverse interoperability module is used to send the filtered emotion data to the Metaverse server through the Raspberry Juice plug-in. The Metaverse server is a Minecraft server. The Metaverse server is configured with an emotion block, and the filtered emotion data is stored in the emotion block. A Raspberry Pi is configured between the Metaverse interoperability module and the Metaverse server, and the Raspberry Juice plug-in is installed and deployed on the Raspberry Pi. The Raspberry Juice plug-in is a plug-in that allows users to control the Minecraft game programmatically. It implements the Minecraft Pi API. This plug-in is usually used for education and programming learning, enabling players to interact with the Minecraft game world by writing code, such as building structures, moving items and players in the game, etc.

[0084] The working process of the system in this embodiment is as follows:

[0085] (1) Image acquisition and facial expression recognition: The user’s facial image is collected through the AIoT device (controller), and a deep learning model is used in the controller to recognize facial expressions and extract emotional data;

[0086] (2) Emotion filtering: compare the identified emotions with the emotion list specified by the user and filter out the emotions that are not in the list to ensure that only the emotions that the user is interested in are saved and processed;

[0087] (3) Emotional data transmission: The filtered emotional data is transmitted to the client Socket through Socket communication, and then transmitted to the server Socket;

[0088] (4) Data upload to the IoT platform: The server socket uploads the emotion data to the IoT server based on the oneM2M international standard. The IoT server stores the emotion data collected from the IoT client application and passes it to the Metaverse interoperability module;

[0089] (5) Data subscription and notification: The Metaverse interoperability module subscribes to the emotion data of a specific topic. When the emotion data of the corresponding topic is uploaded to the IoT server, the Metaverse server will receive a notification. This subscription function ensures that the Metaverse server can obtain the emotion data in a timely manner.

[0090] (6) Emotional mapping in the Metaverse: Emotional data is transmitted to the Metaverse server (Minecraft server) through the Raspberry Juice plug-in and API interface, and objects or characters in the virtual world are updated to reflect the user's emotional state. Through a series of APIs and plug-ins (such as the Raspberry Juice plug-in), the Metaverse interoperability module converts the emotional data collected by IoT devices into information that can be used in the Metaverse, allowing the Metaverse environment to respond accordingly.

[0091] (7) Emotional feedback in the metaverse: Users can perceive the emotions of others through the facial expressions of characters in the virtual world, achieving emotional mapping between the real world and the virtual world.

[0092] The AIoT facial expression recognition device collects images from a webcam and uses a deep learning model to recognize facial expressions and collect emotions. In emotion filtering, the user-specified emotions are saved in a filter list; then, the specified emotions are filtered to include only the emotions included in the filter list; the filtered emotions are transmitted to the client socket through socket communication and then sent to the server interface.

[0093] To link the emotion recognition data from Metaverse and real-world users, an Internet of Things (IoT) server platform based on the oneM2M international standard of IoT acts as an infrastructure node-generic service entity to store data from IoT client applications and pass it to the Metaverse interoperability module.

[0094] Subscription and notification functions receive filtered emotion data in the Metaverse. This is achieved by using the emotion block in the Metaverse platform, where the collected data is transmitted to the IoT platform and stored in the emotion block. Once the emotion recognition data collected by the AIoT sensor device is uploaded to the IoT platform, the server that subscribes to the data uploaded to the specific container will receive a notification about the emotion recognition data; then, the emotion recognition data is transmitted to the Metaverse server through the API interface, and the Metaverse object is updated according to the received emotion recognition data.

[0095] Embodiment 2:

[0096] The present invention provides an expression mapping method based on artificial intelligence Internet of Things, which realizes expression mapping through the system disclosed in Example 1. The method comprises the following steps:

[0097] Step S100, collecting facial expression images through a camera, and transmitting the facial expression images to a controller;

[0098] Step S200, performing emotion recognition on the input facial expression image through the trained emotion recognition model deployed in the controller, outputting predicted emotion data, filtering the predicted emotion data based on the emotion list deployed in the controller, obtaining filtered emotion data, and pushing the filtered emotion data to the client;

[0099] Step S300, receiving filtered emotion data through the client and sending the filtered emotion to the server;

[0100] Step S400, receiving and storing the filtered emotion data through the server;

[0101] Step S500: collecting filtered emotion data from the server through the IoT client, and pushing the filtered emotion data to the IoT server;

[0102] Step S600: receiving and storing the filtered emotion data through the IoT server;

[0103] Step S700: The Metaverse server receives and stores the filtered emotions from the Internet of Things server through the Metaverse interoperability module, and updates the filtered emotions to the objects or characters in the virtual world.

[0104] In step S100, the camera uses a high-definition camera, and the camera can be electrically connected to the controller via a USB interface or other means to send the collected facial expression image to the controller.

[0105] This embodiment is a specific implementation of the emotion recognition model, which includes an input layer, a convolution layer, a flattening layer, a fully connected layer and an output layer.

[0106] The input layer receives the facial expression image and preprocesses the input facial expression image, and obtains a facial expression object of a predetermined size as the preprocessed facial expression image through the image preprocessing.

[0107] There are three convolutional layers, namely the first convolutional layer, the second convolutional layer and the third convolutional layer. The first convolutional layer is used to perform low-level feature extraction on the input preprocessed facial expression image and output a feature map. The second convolutional layer is used to use the feature map output by the first convolutional layer as the input image, perform high-level feature extraction on the input image, and output a feature map. The third convolutional layer is used to use the feature map output by the second convolutional layer as the input image, perform feature extraction on the input image, and output a feature map.

[0108] As a specific implementation of the convolutional layer, the first convolutional layer includes the following structure:

[0109] Conv2D Layer: Performs convolution operations on the input image to extract low-level features such as edges and textures;

[0110] ReLU Activation: Introducing nonlinearity through the ReLU activation function;

[0111] Max Pooling2D: performs pooling operations to reduce the dimension of feature maps and retain the main features;

[0112] Dropout: Prevent the model from overfitting by randomly dropping some neurons.

[0113] The second convolutional layer includes the following structure:

[0114] Conv2D Layer: further extracts high-level features of the image;

[0115] ReLU Activation: reintroduces nonlinearity;

[0116] Max Pooling2D: Continues the pooling operation to reduce the feature dimension;

[0117] Dropout: Dropout is applied again to enhance the generalization ability of the model.

[0118] The third convolutional layer includes the following structure:

[0119] Conv2D Layer: further deepens feature extraction;

[0120] ReLU Activation: Use ReLU to activate nonlinear features;

[0121] Max Pooling2D: further pooling to reduce the dimension of features;

[0122] Dropout: Perform a dropout operation again before feature extraction is completed.

[0123] The model uses a CNN model to predict seven emotion categories. It receives image input from a webcam, the image size is resized to 48x48, a two-dimensional convolutional layer is used to extract features from the input image, and a dropout layer is used to prevent overfitting. The facial expression recognition data collected through the webcam can go through an emotion filtering process to prevent unexpected facial expression recognition data from being transmitted, because even small changes in facial expressions can be detected and transmitted to the server. This filtering process allows users to filter out unwanted emotions and prevent such problems.

[0124] In this embodiment, the controller pushes the filtered emotion data to the client through Socket communication. The client can be marked as a client Socket, and the corresponding server is marked as a server Socket.

[0125] In this embodiment, the Metaverse server obtains data from the Internet of Things server by means of message subscription through the Metaverse interoperability module. The Metaverse server subscribes to the emotion data of a specified topic through the Metaverse interoperability module. After the emotion data of the corresponding topic is uploaded to the Internet of Things server, the Metaverse server reads the emotion data of the relevant topic from the Internet of Things server through the Metaverse interoperability module.

[0126] As a specific implementation, the Metaverse interoperability module is used to send the filtered emotion data to the Metaverse server through the Raspberry Juice plug-in. The Metaverse server is a Minecraft server. The Metaverse server is configured with an emotion block, and the filtered emotion data is stored in the emotion block. A Raspberry Pi is configured between the Metaverse interoperability module and the Metaverse server, and the Raspberry Juice plug-in is installed and deployed on the Raspberry Pi. The Raspberry Juice plug-in is a plug-in that allows users to control the Minecraft game programmatically. It implements the Minecraft Pi API. This plug-in is usually used for education and programming learning, enabling players to interact with the Minecraft game world by writing code, such as building structures, moving items and players in the game, etc.

[0127] The method of this embodiment utilizes AIoT devices to recognize facial expressions, and maps the recognized emotion data to virtual objects in the metaverse through the Internet of Things, thereby realizing real-time expression and sharing of user emotions and enhancing the user's immersion and interactive experience in the metaverse.

[0128] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. An expression mapping system based on artificial intelligence Internet of Things, characterized in that: Includes cameras, controllers, local clients, servers, IoT clients, IoT servers, Metaverse interoperability modules, and Metaverse servers; The camera is used to collect facial expression images and transmit the facial expression images to the controller; The controller is electrically connected to the camera, and a trained emotion recognition model is deployed in the controller, and an emotion list is configured, for performing emotion recognition on the input facial expression image through the trained emotion recognition model, outputting predicted emotion data, filtering the predicted emotion data based on the emotion list, obtaining filtered emotion data, and pushing the filtered emotion data to the client; The client interacts with the controller to receive filtered emotion data and send the filtered emotion to the server; The server is connected to the client to receive and store the filtered emotion data; The IoT client performs data interaction with the server to collect filtered emotion data from the server and push the filtered emotion data to the IoT server; The IoT server is connected to the client to receive and store the filtered emotion data; The Metaverse server is used to receive and store filtered emotions from the Internet of Things server through the Metaverse interoperability module, and update the filtered emotions to objects or characters in the virtual world.

2. The expression mapping system based on artificial intelligence Internet of Things according to claim 1 is characterized in that: The emotion recognition model includes an input layer, a convolution layer, a flattening layer, a fully connected layer and an output layer; The input layer receives the facial expression image and preprocesses the input facial expression image to obtain a facial expression object of a predetermined size as the preprocessed facial expression image; There are three convolutional layers, namely the first convolutional layer, the second convolutional layer and the third convolutional layer. The first convolutional layer is used to extract low-level features of the input preprocessed facial expression image and output a feature map. The second convolutional layer is used to extract high-level features of the input image using the feature map output by the first convolutional layer as the input image and output a feature map. The third convolutional layer is used to extract features of the input image using the feature map output by the second convolutional layer as the input image and output a feature map. The flattening layer is used to take the feature map output by the third convolutional layer as input and flatten the multi-dimensional feature map into a one-dimensional vector; The fully connected layer is used to take the one-dimensional vector output by the flattening layer as input, map the one-dimensional vector to a high-dimensional space, and output a feature map in combination with the global features; The output layer uses the feature map output by the fully connected layer as an input image, is used to perform emotion classification through a Softmax activation function, and outputs a probability distribution of each emotion category.

3. The expression mapping system based on artificial intelligence Internet of Things according to claim 2 is characterized in that: The first convolution layer, the second convolution layer and the third convolution layer all include a two-dimensional convolution layer Conv2D Layer, an activation function layer ReLUActivation, a pooling layer Max Pooling2D and a dropout layer Dropout. The two-dimensional convolution layer is used to perform a convolution operation on the input image through Conv2DLayer to extract features, the activation function layer is used to introduce nonlinearity through the ReLU activation function, the pooling layer is used to provide a pooling operation, and the dropout layer is used to prevent the model from overfitting by randomly discarding neurons.

4. The expression mapping system based on artificial intelligence Internet of Things according to any one of claims 1 to 3, characterized in that: The controller is used to push the filtered emotion data to the client through Socket communication, and the Metaverse interoperability module is used to send the filtered emotion data to the Metaverse server through the Raspberry Juice plug-in. The Metaverse server is a Minecraft server.

5. The expression mapping system based on artificial intelligence Internet of Things according to any one of claims 1 to 3, characterized in that: The Metaverse server obtains data from the IoT server by means of message subscription through the Metaverse interoperability module. The Metaverse server subscribes to the emotion data of the specified topic through the Metaverse interoperability module. After the emotion data of the corresponding topic is uploaded to the IoT server, the Metaverse server reads the emotion data of the relevant topic from the IoT server through the Metaverse interoperability module.

6. An expression mapping method based on artificial intelligence Internet of Things, characterized in that: The method is used to realize expression mapping through an expression mapping system based on artificial intelligence Internet of Things as described in any one of claims 1 to 5, the method comprising the following steps: Collecting facial expression images through a camera and transmitting the facial expression images to a controller; The trained emotion recognition model deployed in the controller is used to perform emotion recognition on the input facial expression image, output predicted emotion data, and filter the predicted emotion data based on the emotion list deployed in the controller to obtain filtered emotion data, and push the filtered emotion data to the client; receiving filtered sentiment data through the client and sending the filtered sentiment to the server; receiving and storing the filtered emotion data via a server; Collect filtered emotion data from the server through the IoT client, and push the filtered emotion data to the IoT server; Receive and store filtered emotion data through IoT server; The Metaverse server receives and stores the filtered emotions from the IoT server through the Metaverse interoperability module, and updates the filtered emotions to objects or characters in the virtual world.

7. The expression mapping system based on artificial intelligence Internet of Things according to claim 6 is characterized in that: The emotion recognition model includes an input layer, a convolution layer, a flattening layer, a fully connected layer and an output layer; The input layer receives the facial expression image and preprocesses the input facial expression image to obtain a facial expression object of a predetermined size as the preprocessed facial expression image; There are three convolutional layers, namely the first convolutional layer, the second convolutional layer and the third convolutional layer. The first convolutional layer is used to extract low-level features of the input preprocessed facial expression image and output a feature map. The second convolutional layer is used to extract high-level features of the input image using the feature map output by the first convolutional layer as the input image and output a feature map. The third convolutional layer is used to extract features of the input image using the feature map output by the second convolutional layer as the input image and output a feature map. The flattening layer is used to take the feature map output by the third convolutional layer as input and flatten the multi-dimensional feature map into a one-dimensional vector; The fully connected layer is used to take the one-dimensional vector output by the flattening layer as input, map the one-dimensional vector to a high-dimensional space, and output a feature map in combination with the global features; The output layer uses the feature map output by the fully connected layer as an input image, is used to perform emotion classification through a Softmax activation function, and outputs a probability distribution of each emotion category.

8. The expression mapping system based on artificial intelligence Internet of Things according to claim 7 is characterized in that: The first convolution layer, the second convolution layer and the third convolution layer all include a two-dimensional convolution layer Conv2D Layer, an activation function layer ReLUActivation, a pooling layer Max Pooling2D and a dropout layer Dropout. The two-dimensional convolution layer is used to perform a convolution operation on the input image through Conv2DLayer to extract features, the activation function layer is used to introduce nonlinearity through the ReLU activation function, the pooling layer is used to provide a pooling operation, and the dropout layer is used to prevent the model from overfitting by randomly discarding neurons.

9. The expression mapping system based on artificial intelligence Internet of Things according to any one of claims 6 to 9, characterized in that: The controller pushes the filtered emotion data to the client through Socket communication, and the Metaverse interoperability module sends the filtered emotion data to the Metaverse server through the Raspberry Juice plug-in. The Metaverse server is the Minecraft server.

10. The expression mapping system based on artificial intelligence Internet of Things according to any one of claims 6 to 9, characterized in that: The Metaverse server obtains data from the IoT server by means of message subscription through the Metaverse interoperability module. The Metaverse server subscribes to the emotion data of the specified topic through the Metaverse interoperability module. After the emotion data of the corresponding topic is uploaded to the IoT server, the Metaverse server reads the emotion data of the relevant topic from the IoT server through the Metaverse interoperability module.

Images