Multi-modal smart classroom edge computing control system

Abstract

Disclosed in the present invention is a multi-modal smart classroom edge computing control system. The system comprises an audio data collection unit, a panoramic image collection and AI computing unit, a network connection unit and a device control unit, wherein the audio data collection unit comprises a microphone assembly, which is used for acquiring a target sound; the panoramic image collection and AI computing unit comprises a panoramic camera, which is used for acquiring panoramic image data of the interior of a smart classroom; the network connection unit comprises a WiFi 6 wireless network unit, an optical fiber access unit and an RJ45 wired network unit, which are used for accessing common computing devices, IoT devices and other wireless devices; and the device control unit controls the IoT devices by means of an RS232 bus interface. In the multi-modal smart classroom edge computing control system used in the present invention, voice control, gesture control and multi-modal control methods are added, such that all devices are in interconnection and intercommunication with each other, and compared with the use of manual buttons alone, the use of the system can save time and is convenient and fast.

Description

A multimodal edge computing control system for smart classrooms Technical Field

[0001] The present invention relates to the field of educational informatization technology, and in particular to a multimodal smart classroom edge computing control system. Background Art

[0002] In recent years, with the continuous development of big data and information technology and the advent of the Internet era, people's lives and production methods have undergone tremendous changes. Intelligent and information-based education models have also become part of people's daily lives. The optimization and innovation of educational methods need to start with the design of smart classrooms in colleges and universities. At present, most primary and secondary schools in my country have begun to apply smart classroom technology. The functional departments of the smart classroom Internet of Things application system are linked through control logic. So far, the control methods of smart classrooms are usually manifested in two forms: (1) The control method of smart classrooms is mainly to control multiple subsystems separately, and multiple operations must be performed each time; (2) The control method of smart classrooms can also be achieved by each device independently playing a role, operating and controlling independently, that is, controlling through the buttons, switches, and remote controls of the corresponding devices.

[0003] Therefore, the existing smart classroom control system or control method has the following disadvantages:

[0004] (1) The control method is relatively crude, and the functions and control methods of a single system are relatively simple. Multiple systems are controlled independently, and the systems do not communicate with each other, or the communication methods and methods are limited. Each time the equipment is switched on and off, it needs to be manually operated and controlled for each device or subsystem. Each time the equipment is switched on and off, it is time-consuming and labor-intensive, and inefficient.

[0005] (2) The existing control method for independent devices in smart classrooms requires the use of specific remote controls or keypads for specific devices, which makes it inconvenient and inefficient to use. It cannot match the "smart classroom" and is a simple patchwork of single-function devices.

[0006] Therefore, there is a need for an efficient, easy-to-manage and multimodal smart classroom control system. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal smart classroom edge computing control system. By adding voice control, gesture control, and multimodal control methods, various devices can be interconnected, which saves time and is more convenient than using manual buttons alone.

[0008] To achieve the above-mentioned objectives, the present invention provides a multimodal smart classroom edge computing control system, comprising an audio data acquisition unit, a panoramic image acquisition and AI computing unit, a network connection unit and a device control unit, wherein the audio data acquisition unit, the panoramic image acquisition and AI computing unit, the network connection unit and the device control unit are connected to a controller; the audio data acquisition unit includes a microphone assembly, which includes a microphone, an audio processing chip and a printed circuit board for welding and assembly, for acquiring target sounds; the panoramic image acquisition and AI computing unit includes a panoramic camera, which includes multiple high-definition ordinary cameras with no blind spots, for acquiring panoramic image data inside the smart classroom; the network connection unit includes a WIFI6 wireless network unit, a fiber optic access unit and an RJ45 wired network unit, for accessing ordinary computing devices, IoT devices and other wireless devices; the device control unit controls the IoT device through an RS232 bus interface.

[0009] Preferably, the audio data acquisition unit collects sounds from multiple different directions and performs noise reduction on them, providing a method for noise reduction processing of a speech signal, the method comprising the following steps:

[0010] S1, picking up sound data through a multi-microphone array composed of microphones and pre-storing it in the system memory;

[0011] S2. Preprocess the speech data to be processed to obtain audio data of the first feature; the preprocessing uses a deep learning algorithm to perform preliminary processing on the speech data, including removing environmental background noise in the speech data to retain a clearer human voice;

[0012] S3. Calling a preset second-order noise reduction algorithm to perform secondary processing on the audio data of the first feature, filtering out the stationary noise in the voice data, and obtaining second feature data;

[0013] S4. Inputting the first feature data and the second feature data into a preset noise reduction deep learning algorithm, filtering transient noise in the speech data to be processed, and obtaining third feature data;

[0014] S5. Determine the speech data after noise reduction processing according to the first feature data, the second feature data, and the third feature data.

[0015] Preferably, the panoramic image acquisition and AI calculation unit solves the sound and image captured by the panoramic camera according to the width learning algorithm model, which is used to analyze the students' concentration and emotions, and the learning environment atmosphere.

[0016] Preferably, the width learning algorithm model solves the sound and image collected by the panoramic camera, including the following steps:

[0017] S1. Based on the data of teachers and students in the smart classroom, i.e. user data, a width learning algorithm model is trained;

[0018] S2, simultaneously transmitting user data, environmental sound data and image data to the width learning algorithm model and HMM emotion calculation model that have completed model training;

[0019] S3. Predict the teaching and learning behaviors of teachers and students through a breadth learning algorithm model, and understand the language and action intentions of teachers and students;

[0020] S4. The understood language and action intention data of teachers and students are passed into the HMM emotion computing model for data preprocessing. The user data, smart classroom environment data, and preprocessed intention data are converted into data expression information that can be understood by the HMM emotion computing model.

[0021] S5. Extract feature data related to teachers and students to perform emotion recognition of teachers and students;

[0022] S6. Establish an HMM emotion computing model, use the smart classroom environment data and intention data as sensitive factor indicators, and calibrate the HMM emotion computing model;

[0023] S7. Output emotional expression data that meets the needs of teachers and students.

[0024] Preferably, the device control unit controls the IoT device through the RS232 bus interface, including controlling multiple infrared receiving remote control devices in the smart classroom and accelerating through deep learning algorithms, using a multi-microphone array and a panoramic camera to generate control instructions for voice or gesture control.

[0025] The above-mentioned multimodal smart classroom edge computing control terminal includes a device housing, a printed circuit board and a controller are arranged inside the device housing, the printed circuit board is connected to the controller, a microphone and an audio processing chip are arranged on the printed circuit board, an external device interface and a keypad are arranged on one side of the device housing, a wireless network interface is arranged on the side of the device housing adjacent to the keypad, a wired network interface and a power input interface are arranged on the side of the device housing opposite to the wireless network interface, a panoramic camera is arranged on the side of the device housing opposite to the external device interface and the keypad, and the microphone, audio processing chip, external device interface, keypad, wireless network interface, wired network interface, power input interface, and panoramic camera are all connected to the controller.

[0026] The advantages and positive effects of the multimodal smart classroom edge computing control system described in the present invention are:

[0027] 1. While retaining manual button control, voice control, gesture control and multi-modal control methods are added, which saves time and is more convenient and quick than using manual buttons alone for step-by-step control.

[0028] 2. Change the previous multiple control subsystems and interconnect various devices through a multimodal smart classroom edge computing control system.

[0029] 3. Place voice processing on the local terminal and edge computing instead of cloud processing, which reduces dependence on the network, improves control latency, and makes response faster.

[0030] 4. Fixed gesture recognition, deep learning algorithm, local terminal edge computing and real-time processing.

[0031] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 is a schematic structural diagram of an embodiment of a multimodal smart classroom edge computing control system according to the present invention;

[0033] FIG2 is a schematic structural diagram of an embodiment of a multimodal smart classroom edge computing control system according to another angle of the present invention;

[0034] FIG3 is a flowchart of multi-microphone array speech noise reduction in an embodiment of a multi-modal smart classroom edge computing control system of the present invention;

[0035] FIG4 is a schematic diagram illustrating a panoramic image recognition function of an embodiment of a multimodal smart classroom edge computing control system according to the present invention;

[0036] FIG5 is a control workflow diagram of an embodiment of a multimodal smart classroom edge computing control system of the present invention.

[0037] Reference numerals

[0038] 1. Microphone; 2. Keypad; 3. External device interface; 4. Wireless network antenna interface; 5. Panoramic camera; 6. Wired network interface; 7. Power input interface; 8. Device housing; 9. Audio processing chip. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0040] Unless otherwise defined, the technical or scientific terms used in the present invention shall have the usual meanings understood by persons of ordinary skill in the field to which the present invention belongs. The words "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprise" mean that the elements or objects preceding the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly. Example

[0041] A multimodal smart classroom edge computing control system includes an audio data acquisition unit, a panoramic image acquisition and AI computing unit, a network connection unit, and a device control unit, all of which are connected to a controller. The audio data acquisition unit includes a microphone assembly, which comprises 4-8 ZTS6032M silicon chip microphones, an XMOS-XVF3610 voice processing chip, and a glass fiber PCB substrate for soldering and assembly, for capturing target sounds. The panoramic image acquisition and AI computing unit includes a panoramic camera, which comprises multiple high-definition conventional cameras capable of capturing panoramic images of the smart classroom interior. The network connection unit includes a Wi-Fi 6 wireless network unit, a fiber optic access unit, and an RJ45 wired network unit, for accessing conventional computing devices, IoT devices, and other wireless devices. The device control unit controls the IoT devices via an RS232 bus interface.

[0042] The multimodal smart classroom edge computing control terminal is equipped with multiple microphones, forming a sound pickup array that can effectively extract sounds from all directions in the environment. Because smart classrooms are subject to a wide range of uncontrollable noise, the multimodal smart classroom edge computing control terminal must be able to automatically extract the target sound. The audio processing chip processes the sound captured by the sound pickup microphone array, filtering out unnecessary noise and extracting the useful target sound, ensuring sufficient clarity. Therefore, a multi-microphone noise reduction algorithm is required to ensure the effective collection of audio data.

[0043] FIG3 is a flow chart of multi-microphone array speech noise reduction. As shown in the figure, in order to solve the problem of microphone noise reduction, a method for noise reduction processing of speech signals is provided. The method includes the following steps:

[0044] S1, picking up sound data through a multi-microphone array composed of multiple microphones and pre-storing it in the system memory;

[0045] S2. Preprocess the speech data to be processed to obtain audio data of the first feature; the preprocessing uses a deep learning algorithm to perform preliminary processing on the speech data, including removing environmental background noise in the speech data to retain a clearer human voice;

[0046] S3. Calling a preset second-order noise reduction algorithm to perform secondary processing on the audio data of the first feature, filtering out the stationary noise in the voice data, and obtaining second feature data;

[0047] S4. Inputting the first feature data and the second feature data into a preset noise reduction deep learning algorithm, filtering transient noise in the speech data to be processed, and obtaining third feature data;

[0048] S5. Determine the speech data after noise reduction processing according to the first feature data, the second feature data, and the third feature data.

[0049] Figure 5 shows a control workflow diagram for an embodiment of a multimodal smart classroom edge computing control system. The device control unit controls IoT devices via an RS232 bus interface. As shown, multimodal voice and panoramic image recognition enable voice and gesture control of the smart classroom.

[0050] In multimodal voice control, the multi-microphone array can pick up sound data from the large scene of the smart classroom and automatically filter it. The multi-microphone array can identify and recognize sounds from different directions through microphones in different positions, making the picked up sound more three-dimensional. It can realize voice recognition to control various devices in the smart classroom, such as voice control of electronic touch screen switches, voice control of smart classroom light and curtain switches, and voice control of the working status of the smart recording and broadcasting system in the smart classroom.

[0051] In panoramic image recognition control, a panoramic camera is installed in the smart classroom to obtain panoramic image data inside the smart classroom. The image edge computing algorithm is used to identify the body movements of anyone in the panoramic image. By presetting multiple smart classroom device control actions in the multimodal smart classroom edge computing control terminal system, the terminal system can control various devices in the smart classroom through the body movements or local movements made by teachers or students in the smart classroom, such as gesture control of electronic touch screen switches.

[0052] As shown in Figure 4, the implementation of the panoramic image recognition function includes a panoramic camera, an image processor of a multimodal smart classroom edge computing control terminal, a voice or image prompter, i.e., a voice prompt speaker for confirming the operation command, and a target device for control.

[0053] The multimodal smart classroom edge computing control system can obtain images inside the smart classroom through the panoramic camera inside the smart classroom and calculate the students' concentration, emotions and learning atmosphere.

[0054] The multimodal smart classroom edge computing control system uses the panoramic cameras in the smart classroom to obtain the real-time expressions, body movements, and eye movements of students in class to train the width learning algorithm model. The AI ​​computing unit equipped with the multimodal smart classroom edge computing control terminal is used to solve the images obtained by the panoramic camera according to the width learning algorithm model, and the solution results are fed into the smart classroom learning situation analysis system to analyze students' concentration and emotions in class, thereby helping teachers analyze students' learning outcomes.

[0055] The multimodal smart classroom edge computing control terminal obtains image data from the smart classroom through the panoramic camera in the smart classroom, and uses a multi-microphone array to obtain voice information in the smart classroom. The AI ​​computing unit equipped with the multimodal smart classroom edge computing control terminal analyzes the voice data and image data of the smart classroom based on the width learning algorithm model. The analysis results include speech semantics, speech emotions, students' and teachers' body movements and facial expressions, etc., and derive the learning environment atmosphere, thereby helping teachers to identify the internal factors that affect students' learning outcomes by analyzing the learning environment atmosphere.

[0056] Using the width learning algorithm model to solve the sound and image collected by the panoramic camera includes the following steps:

[0057] S1. Based on the data of teachers and students in the smart classroom, i.e. user data, a width learning algorithm model is trained;

[0058] S2, simultaneously transmitting user data, environmental sound data and image data to the width learning algorithm model and HMM emotion calculation model that have completed model training;

[0059] S3. Predict the teaching and learning behaviors of teachers and students through a breadth learning algorithm model, and understand the language and action intentions of teachers and students;

[0060] S4. The understood language and action intention data of teachers and students are passed into the HMM emotion computing model for data preprocessing. The user data, smart classroom environment data, and preprocessed intention data are converted into data expression information that can be understood by the HMM emotion computing model.

[0061] S5. Extract feature data related to teachers and students to perform emotion recognition of teachers and students;

[0062] S6. Establish an HMM emotion computing model, use the smart classroom environment data and intention data as sensitive factor indicators, and calibrate the HMM emotion computing model;

[0063] S7. Output emotional expression data that meets the needs of teachers and students.

[0064] As shown in Figures 1 and 2, a multimodal smart classroom edge computing control terminal includes a device housing 8, a printed circuit board and a controller are arranged inside the device housing 8, the printed circuit board is connected to the controller, a microphone 1 and an audio processing chip 9 are arranged on the printed circuit board, 4-8 microphones 1 are provided, and the microphone 1 array is arranged around the printed circuit board, an external device interface 3 and a keypad 2 are provided on one side of the device housing 8, a wireless network interface 4 is provided on the side of the device housing 8 adjacent to the keypad 2, a wired network interface 6 and a power input interface 7 are provided on the side of the device housing 8 opposite to the wireless network interface 4, a panoramic camera 5 is provided on the side of the device housing 8 opposite to the external device interface 3 and the keypad 2, and the microphone 1, audio processing chip 9, external device interface 3, keypad 2, wireless network interface 4, wired network interface 6, power input interface 7, and panoramic camera 5 are all connected to the controller.

[0065] When the microphone array is in use, the system can automatically call the ambient sound picked up by the microphone array from multiple directions, and send it to the audio processing chip 9 for processing to filter out the noise. Multiple microphones 1 can obtain sound information more sensitively and clearly.

[0066] When using the keypad 2, the user can manually operate the system key menu on the multimodal smart classroom edge computing control terminal through the keypad 2 to configure various parameters. The user can also use the keypad 2 to manually control selected IoT devices, such as curtains, lights and other devices in the smart classroom.

[0067] External device interface 3: When installing this terminal device, you can connect a suitable external device through a specific port. External device interface 3 provides a variety of different communication protocols for accessing and controlling multiple devices in the smart classroom and communicating with external devices.

[0068] Wireless network antenna interface 4 is used to connect to the wireless network antenna. The wireless network antenna can not only be directly installed on the terminal body itself, but also be installed at various points in the smart classroom through the antenna cable, so that the smart classroom can achieve multi-point network coverage.

[0069] Panoramic camera 5 can automatically capture images of the smart classroom according to the system program. After being sent to the system for processing, the AI ​​deep learning algorithm can be used to identify teachers and students in the image, thereby providing data resources for learning situation analysis and emotional computing.

[0070] The wired network interface 6 can be connected to optical fiber and Ethernet. The high data bandwidth and transmission speed of the optical fiber network are conducive to the transmission of continuous high-quality multimedia resources. The Ethernet network is used for data exchange and network input between devices.

[0071] The power input interface 7 is used to connect the terminal to the power supply. The device housing 8 is used to install various unit devices and provide a stable operating environment for the entire hardware system.

[0072] Therefore, the present invention adopts the above-mentioned multimodal smart classroom edge computing control system, and enables the interconnection between various devices by adding voice control, gesture control, and multimodal control methods, which saves time and is more convenient than using manual buttons alone.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal smart classroom edge computing control system, characterized by: It includes an audio data acquisition unit, a panoramic image acquisition and AI calculation unit, a network connection unit and a device control unit, which are connected to the controller; the audio data acquisition unit includes a microphone assembly, which includes a microphone, an audio processing chip and a printed circuit board for welding and assembly, and is used to obtain target sounds; the panoramic image acquisition and AI calculation unit includes a panoramic camera, which includes multiple high-definition ordinary cameras with no blind spots, and is used to obtain panoramic image data inside the smart classroom; the network connection unit includes a WIFI6 wireless network unit, a fiber optic access unit and an RJ45 wired network unit, which are used for access of ordinary computing devices, IoT devices and other wireless devices; the device control unit controls the IoT device through the RS232 bus interface.

2. The multimodal smart classroom edge computing control system according to claim 1, characterized in that: The audio data acquisition unit collects sounds from multiple different directions and performs noise reduction on them, providing a noise reduction processing method for a speech signal, the method comprising the following steps: S1, picking up sound data through a multi-microphone array composed of microphones and pre-storing it in the system memory; S2. Preprocess the speech data to be processed to obtain audio data of the first feature; the preprocessing uses a deep learning algorithm to perform preliminary processing on the speech data, including removing environmental background noise in the speech data to retain a clearer human voice; S3. Calling a preset second-order noise reduction algorithm to perform secondary processing on the audio data of the first feature, filtering out the stationary noise in the voice data, and obtaining second feature data; S4. Inputting the first feature data and the second feature data into a preset noise reduction deep learning algorithm, filtering transient noise in the speech data to be processed, and obtaining third feature data; S5. Determine the speech data after noise reduction processing according to the first feature data, the second feature data, and the third feature data.

3. The multimodal smart classroom edge computing control system according to claim 1, characterized in that: The panoramic image acquisition and AI calculation unit solves the sound and image collected by the panoramic camera according to the width learning algorithm model to analyze the students' concentration and emotions and the learning environment atmosphere.

4. The multimodal smart classroom edge computing control system according to claim 3, characterized in that: The width learning algorithm model solves the sound and image collected by the panoramic camera, including the following steps: S1. Based on the data of teachers and students in the smart classroom, i.e. user data, a width learning algorithm model is trained; S2, simultaneously transmitting user data, environmental sound data and image data to the width learning algorithm model and HMM emotion calculation model that have completed model training; S3. Predict the teaching and learning behaviors of teachers and students through a breadth learning algorithm model, and understand the language and action intentions of teachers and students; S4. The understood language and action intention data of teachers and students are passed into the HMM emotion computing model for data preprocessing. The user data, smart classroom environment data, and preprocessed intention data are converted into data expression information that can be understood by the HMM emotion computing model. S5. Extract feature data related to teachers and students to perform emotion recognition of teachers and students; S6. Establish an HMM emotion computing model, use the smart classroom environment data and intention data as sensitive factor indicators, and calibrate the HMM emotion computing model; S7. Output emotional expression data that meets the needs of teachers and students.

5. The multimodal smart classroom edge computing control system according to claim 1, characterized in that: The device control unit controls the IoT device through the RS232 bus interface, including controlling multiple infrared receiving remote control devices in the smart classroom and accelerating through deep learning algorithms, using a multi-microphone array and a panoramic camera to generate control instructions for voice or gesture control.

6. A multimodal smart classroom edge computing control terminal according to any one of claims 1 to 5, characterized in that: The device comprises a housing, a printed circuit board and a controller are arranged inside the housing, the printed circuit board is connected to the controller, a microphone and an audio processing chip are arranged on the printed circuit board, an external device interface and a keypad are arranged on one side of the device housing, a wireless network interface is arranged on the side of the device housing adjacent to the keypad, a wired network interface and a power input interface are arranged on the side of the device housing opposite to the wireless network interface, a panoramic camera is arranged on the side of the device housing opposite to the external device interface and the keypad, and the microphone, audio processing chip, external device interface, keypad, wireless network interface, wired network interface, power input interface, and panoramic camera are all connected to the controller.