Multi-modal smart classroom edge computing control system

Abstract

Disclosed in the present invention is a multi-modal smart classroom edge computing control system. The system comprises an audio data collection unit, a panoramic image collection and AI computing uni

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of education informatization technologies, and in particular, to a multi-modal edge computing control system for a smart classroom. BACKGROUND

[0002] In recent years, with the continuous development of big data and information technology, and the advent of the Internet era, people's lifestyles and production methods have changed dramatically. Intelligent and information-based education modes have also become a daily part of people's learning life, and the optimization and innovation of education means need to start from the design of smart classrooms in colleges and universities. Currently, most of the primary and secondary schools in China have begun to apply the smart classroom technology. The functional divisions of an Internet of Things (loT) application system for a smart classroom are linked through the control logic. So far, a smart classroom control method is usually manifested in two forms: (I) The smart classroom is mainly controlled by a plurality of subsystems respectively, which requires a plurality of operations each time; and (2) The smart classroom can also function independently and is operated and controlled independently through devices, i.e., controlled through buttons, switches, and remote controls of the corresponding devices.

[0003] Therefore, the existing smart classroom control system or control method has the following disadvantages:

[0004] (1) The control method is extensive, and the single system function and control method is unitary. Due to the independent control of a plurality of systems and the non-communication between the systems, or the few communication modes and communication methods, it requires manual operation and control on each device or subsystem to switch on and off devices each time, which is time-consuming, labor-intensive, and inefficient.

[0005] (2) The existing smart classroom independent device control method requires a specific remote control or keypad to control or use specific devices, resulting in inconveniency and inefficiency, and a failure in matching with “smart classrooms”. The control method is a simple patchwork of single-function devices.

[0006] Therefore, it is necessary to provide a multi-modal smart classroom control system that is efficient and easy to manage and use. SUMMARY

[0007] An objective of the present disclosure is to provide a multi-modal edge computing control system for a smart classroom, to achieve interconnection between devices by adding voice control, gesture control, and a multi-modal control method, which is more time-saving and convenient than using manual buttons alone.

[0008] To achieve the above objective, the present disclosure provides a multi-modal edge computing control system for a smart classroom, including an audio data acquisition unit, a panoramic image acquisition and artificial intelligence (Al) computing unit, a network connection unit, and a device control unit. The audio data acquisition unit, the panoramic image acquisition and AI computing unit, the network connection unit, and the device control unit are connected to a controller. The audio data acquisition unit includes a microphone assembly, including a microphone, an audio processing chip, and a printed circuit board for soldering and assembly, to acquire a target sound. The panoramic image acquisition and AI computing unit includes a panoramic camera, including a plurality of dead-angle-free high-definition ordinary cameras, to acquire panoramic image data inside the smart classroom. The network connection unit includes a WIFI6 wireless network unit, a fiber optic access unit, and an RJ45 wired network unit, to access an ordinary computing device, an loT device, and other wireless devices. The device control unit is configured to control the loT device through an RS232 bus interface.

[0009] Preferably, the audio data acquisition unit is configured to acquire a plurality of sounds in different directions and perform noise reduction on the sounds, and provide a noise reduction method for a voice signal. The method includes the following steps:

[0010] SI. picking up sound data through a multi-microphone array formed by microphones, and pre-storing the sound data to a memory of the system;

[0011] S2. preprocessing voice data to be processed, to obtain audio data of a first feature; and calling a deep learning algorithm to perform preliminary processing on the voice data, including removing environmental background noise in the voice data and leaving clear human voice;

[0012] S3, calling a preset second-order noise reduction algorithm to perform secondary processing on the audio data of the first feature, and filtering out smooth noise in the voice data, to obtain data of a second feature ;

[0013] S4. inputting the data of the first feature and the data of the second feature to a preset noise reduction deep learning algorithm, and filtering out transient noise in the voice data to be processed, to obtain data of a third feature; and

[0014] S5. determining noise-reduced voice data based on the data of the first feature, the data of the second feature, and the data of the third feature.

[0015] Preferably, the panoramic image acquisition and AI computing unit is configured to solve the sound and images acquired by the panoramic camera based on a broad learning algorithm model, to analyze concentration and emotion of students and a learning atmosphere.

[0016] Preferably, the solving the sound and images acquired by the panoramic camera based on a broad learning algorithm model includes the following steps:

[0017] SI. training the broad learning algorithm model based on data of teachers and students, i.e., user data, of the smart classroom;

[0018] S2. transferring the user data, environmental sound data, and image data simultaneously to the trained broad learning algorithm model and a Hidden Markov Model (HMM) emotion computing model;

[0019] S3, predicting behaviors of teachers and students in teaching and learning processes based on the broad learning algorithm model, to understand language and action intentions of teachers and students;

[0020] S4. transferring understood language and action intention data of teachers and students into the HMM emotion computing model for data preprocessing, and converting the user data, smart classroom environment data, and preprocessed intention data into data expression information understandable by the HMM emotion computing model;

[0021] S5. extracting feature data related to teachers and students for emotion recognition of teachers and students;

[0022] S6. establishing the HMM emotion computing model, and correcting the HMM emotion computing model by using the smart classroom environment data and intention data as indicators of sensitive factors; and

[0023] S7. outputting emotion expression data that meets needs of teachers and students.

[0024] Preferably, the device control unit is configured to control the loT device through the RS232 bus interface in ways of controlling a plurality of infrared-receiving remote control devices in the smart classroom and accelerating the devices based on a deep learning algorithm, and generating a control command using a multi-microphone array and the panoramic camera, for voice or gesture control.

[0025] The above multi-modal edge computing control terminal for a smart classroom includes a device housing, where the device housing is internally provided with a printed circuit board and a controller, the printed circuit board is connected to the controller, the printed circuit board is provided with a microphone and an audio processing chip, one side of the device housing is provided with a peripheral interface and a keypad, a side surface of the device housing adjacent to the keypad is provided with a wireless network interface, a side surface of the device housing opposite to the wireless network interface is provided with a wired network interface and a power input interface, a side surface of the device housing opposite to the peripheral interface and the keypad is provided with a panoramic camera, and the microphone, the audio processing chip, the peripheral interface, the keypad, the wireless network interface, the wired network interface, the power input interface, and the panoramic camera are all connected to the controller.

[0026] The advantages and positive effects of the multi-modal edge computing control system for a smart classroom according to the present disclosure are as follows:

[0027] 1. While the manual button control is retained, the voice control, gesture control, and multi-modal control method are added, which is more time-saving and convenient than step-by-step control using manual buttons alone.

[0028] 2. The previous control subsystems are changed, and the interconnection between the devices is achieved through the multi-modal edge computing control system for a smart classroom.

[0029] 3. Voice is put in a local terminal for edge computing processing instead of cloud processing, which reduces the dependence on the network, improves the control delay, and makes the response quicker.

[0030] 4. The gesture recognition is fixed for real-time edge computing processing by a local terminal based on a deep learning algorithm.

[0031] The technical solutions of the present disclosure will be further described in detail below with reference to drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG. 1 is a schematic structural diagram of an embodiment of a multi-modal edge computing control system for a smart classroom according to the present disclosure;

[0033] FIG. 2 is a schematic structural diagram of an embodiment of the multi-modal edge computing control system for a smart classroom according to the present disclosure from another angle;

[0034] FIG. 3 is a flowchart of a voice noise reduction process of a multi-microphone array of an embodiment of the multi-modal edge computing control system for a smart classroom according to the present disclosure;

[0035] FIG. 4 is a schematic composition diagram of a panoramic image recognition function of an embodiment of the multi-modal edge computing control system for a smart classroom according to the present disclosure; and

[0036] FIG. 5 is a control workflow diagram of an embodiment of the multi-modal edge computing control system for a smart classroom according to the present disclosure.

[0037] Reference Numerals:

[0038] 1, microphone; 2, keypad; 3, peripheral interface; 4, wireless network antenna interface; 5, panoramic camera; 6, wired network interface; 7, power input interface; 8, device housing; 9, audio processing chip. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The technical solutions of the present disclosure will be further described below with reference to the drawings and embodiments.

[0040] Unless otherwise defined, the technical or scientific terms used herein should have the usual meanings understood by a person of ordinary skill in the field to which the present disclosure belongs. Terms such as "first" and "second" used herein do not denote any order, quantity or importance, but are merely intended to distinguish between different constituents. "Comprising", "containing", and similar words mean that elements or articles appearing before "comprising" or "containing" include the elements or articles and their equivalent elements appearing behind "comprising" or "containing", not excluding any other elements or articles. Terms such as "connected to" and "connected with" are not restricted to physical or mechanical connections, but may also include electrical connections, whether direct or indirect. "Upper", "lower", "left", "right", and the like are used only to indicate a relative positional relationship, and when the absolute position of the described object is changed, the relative positional relationship is also changed accordingly. Embodiment

[0041] A multi-modal edge computing control system for a smart classroom includes an audio data acquisition unit, a panoramic image acquisition and AI computing unit, a network connection unit, and a device control unit. The audio data acquisition unit, the panoramic image acquisition and AI computing unit, the network connection unit, and the device control unit are connected to a controller. The audio data acquisition unit includes a microphone assembly. The microphone assembly includes four to eight silicon chip microphones, which are ZTS6032M devices, an XMOS-XVF3610 voice processing chip, and a fiberglass PCB substrate for soldering and assembly, to acquire a target sound. The panoramic image acquisition and AI computing unit includes a panoramic camera. The panoramic camera includes a plurality of dead-angle-free high-definition ordinary cameras, to acquire a panoramic image data inside the smart classroom. The network connection unit includes a WIFI6 wireless network unit, a fiber optic access unit, and a RJ45 wired network unit, to access an ordinary computing device, an loT device, and other wireless devices. The device control unit is configured to control the loT device through an RS232 bus interface.

[0042] A multi-modal edge computing control terminal for a smart classroom is equipped with a plurality of microphones, which form a sound pickup microphone array that can effectively extract sounds in all directions from an environment. Due to the existence of uncontrollable noise in the smart classroom, the multi-modal edge computing control terminal for a smart classroom needs to be capable of automatically extracting the target sound, process the sound extracted by the sound pickup microphone array through the audio processing chip, filter out useless noise, and extract the useful target sound, and be capable of ensuring that the target sound is clear enough. Therefore, a multi-microphone noise reduction algorithm needs to be introduced to ensure the effective acquisition of audio data.

[0043] FIG. 3 is a flowchart of voice noise reduction for a multi-microphone array. As shown in the figure, to resolve the microphone noise reduction problem, a noise reduction method for a voice signal is provided. The method includes the following steps:

[0044] S1. Pick up sound data through a multi-microphone array formed by microphones, and pre-store the sound data to a memory of the system.

[0045] S2. Preprocess voice data to be processed, to obtain audio data of a first feature; and call a deep learning algorithm to perfonn preliminary processing on the voice data, including removing environmental background noise in the voice data and leaving clear human voice.

[0046] S3. Call a preset second-order noise reduction algorithm to perform secondary processing on the audio data of the first feature, and filter out smooth noise in the voice data, to obtain data of a second feature.

[0047] S4. Input the data of the first feature and the data of the second feature to a preset noise reduction deep learning algorithm, and filter out transient noise in the voice data to be processed, to obtain data of a third feature.

[0048] S5. Determine noise-reduced voice data based on the data of the first feature, the data of the second feature, and the data of the third feature.

[0049] FIG. 5 is a control workflow diagram of an embodiment of the multi-modal edge computing control system for a smart classroom. The device control unit is configured to control the loT device through an RS232 bus interface. As shown in FIG. 5, voice control and gesture control of the smart classroom can be implemented through multi-modal voice and panoramic image recognition.

[0050] In the multi-modal voice control, the multi-microphone array can pick up sound data from a large scene of the smart classroom and filter the sound data automatically. The multi-microphone array can acquire and recognize sounds from different directions through the microphones at different positions, which makes the sounds picked up more stereo, and can control all types of devices in the smart classroom through voice recognition, such as voice control of the switch on and off of a large electronic touch screen, voice control of the switch on and off of lights and curtains in the smart classroom, and voice control of a working status of a smart recording system in the smart classroom.

[0051] In the panoramic image recognition control, the panoramic camera is mounted in the smart classroom, to obtain panoramic image data inside the smart classroom. By recognizing body movements of anyone in the panoramic image based on an image edge computing algorithm, and preseting various device control actions in the smart classroom in the multi-modal edge computing control terminal system for a smart classroom, the terminal system can control various types of devices in the smart classroom through the body movements or local movements made by teachers or students in the smart classroom, such as gesture control of the switch on and off of the large electronic touch screen.

[0052] As shown in FIG. 4, a panoramic image recognition function is implemented by a panoramic camera, an image processor of the multi-modal edge computing control terminal for a smart classroom, a voice or image prompter, i.e., a voice-prompted speaker for controlling to confirm a command, and a controlled target device.

[0053] The multi-modal edge computing control system for a smart classroom can obtain an image in the smart classroom through the panoramic camera in the smart classroom, and calculate concentration and emotion of students and a learning atmosphere.

[0054] The multi-modal edge computing control system for a smart classroom acquires real-time expressions, body movements, and eye movements of students in class through the panoramic camera in the smart classroom, to train a broad learning algorithm model. An AI computing unit equipped in the multi-modal edge computing control terminal for a smart classroom is used to solve the images acquired by the panoramic camera based on the broad learning algorithm model, and then transfer a solving result into a learning situation analysis system of the smart classroom to analyze concentration and emotion of students in class, thereby helping teachers to analyze learning outcomes of the students.

[0055] The multi-modal edge computing control terminal for a smart classroom acquires image data in the smart classroom through the panoramic camera in the smart classroom and uses the multi-microphone array to acquire voice information in the smart classroom. The AI computing unit equipped in the multi-modal edge computing control terminal for a smart classroom is used to analyze the voice data and image data in the smart classroom based on the broad learning algorithm model. An analysis result includes voice semantics, voice emotion, and body movements and facial expressions of students and teachers, to obtain a learning atmosphere, thereby helping teachers to analyze the learning atmosphere to find out intrinsic factors affecting learning effects of students.

[0056] The solving the sound and images acquired by the panoramic camera based on a broad learning algorithm model includes the following steps:

[0057] SI. training the broad learning algorithm model based on data of teachers and students, i.e., user data, of the smart classroom;

[0058] S2. transferring the user data, environmental sound data, and image data simultaneously to the trained broad learning algorithm model and an HMM emotion computing model;

[0059] S3, predicting behaviors of teachers and students in teaching and learning processes based on the broad learning algorithm model, to understand language and action intentions of teachers and students;

[0060] S4. transferring understood language and action intention data of teachers and students into the HMM emotion computing model for data preprocessing, and converting the user data, smart classroom environment data, and preprocessed intention data into data expression information understandable by the HMM emotion computing model;

[0061] S5. extracting feature data related to teachers and students for emotion recognition of teachers and students;

[0062] S6. establishing the HMM emotion computing model, and correcting the HMM emotion computing model by using the smart classroom environment data and intention data as indicators of sensitive factors; and

[0063] S7. outputting emotion expression data that meets needs of teachers and students.

[0064] As shown in FIG. 1 and FIG. 2, a multi-modal edge computing control terminal for a smart classroom includes a device housing 8. The device housing 8 is internally provided with a printed circuit board and a controller. The printed circuit board is connected to the controller. The printed circuit board is provided with a microphone 1 and an audio processing chip 9. Four to eight microphone 1 are provided. A microphone 1 array is arranged around the printed circuit board. One side of the device housing 8 is provided with a peripheral interface 3 and a keypad 2. A side surface of the device housing 8 adjacent to the keypad 2 is provided with a wireless network interface 4. A side surface of the device housing 8 opposite to the wireless network interface 4 is provided with a wired network interface 6 and a power input interface 7. A side surface of the device housing 8 opposite to the peripheral interface 3 and the keypad 2 is provided with a panoramic camera 5. The microphone 1, the audio processing chip 9, and the peripheral interface 3, keypad 2, the wireless network interface 4, the wired network interface 6, the power input interface 7, and the panoramic camera 5 are all connected to the controller.

[0065] When the microphone array is used, the system can automatically call the environmental sound picked up by the microphone array from a plurality of directions, and transfer the sound to the audio processing chip 9 to filter out the noise. The plurality of microphones 1 can acquire more sensitive and clear sound information.

[0066] When the keypad 2 is used, the user can manually operate a key menu of the system in the multi-modal edge computing control terminal for a smart classroom through the keypad 2, to configure various parameters, or can use the keypad 2 to manually control selected loT devices, such as curtains or lights in the smart classroom.

[0067] During the mounting of the terminal device, the peripheral interface 3 can be connected to an appropriate peripheral through a specific port. The peripheral interface 3 provides various different communication protocols for accessing and controlling a plurality of devices in the smart classroom and communicating with peripherals.

[0068] When the wireless network antenna interface 4 is used, the wireless network antenna is accessed. The wireless network antenna can not only be mounted directly on the terminal body, but also be mounted at points in the smart classroom through antenna cables, so that the smart classroom can achieve multi-point network coverage.

[0069] The panoramic camera 5 can automatically shoot images in the smart classroom based on the system program, transfer the images to the system to recognize teachers and students in the images based on an AI deep learning algorithm, to provide data resources for the learning situation analysis and emotion computing.

[0070] The wired network interface 6 can be connected to a fiber optic network and an Ethernet network. A high data bandwidth and a transmission speed of the fiber optic network facilitate the transmission of continuous high-quality multimedia resources. The Ethernet network is used for data exchange and network input between devices.

[0071] The power input interface 7 is connected to a power supply for the terminal. The device housing 8 is used to mount units and devices and provide a stable operating environment for the entire hardware system.

[0072] Therefore, the present disclosure adopts the above multi-modal edge computing control system for a smart classroom, to achieve interconnection between devices by adding voice control, gesture control, and a multi-modal control method, which is more time-saving and convenient than using manual buttons alone.

[0073] Finally, it should be noted that the foregoing embodiments are only intended to describe, rather than to limit the technical solutions of the present disclosure. Although the present disclosure is described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that modifications or equivalent replacements may be made to the technical solutions of the present disclosure without departing from the spirit and scope of the technical solutions of the present disclosure.

Claims

1. A multi-modal edge computing control system for a smart classroom, comprising an audio data acquisition unit, a panoramic image acquisition and artificial-intelligence (AI) computing unit, a network connection unit, and a device control unit, wherein the audio data acquisition unit, the panoramic image acquisition and AI computing unit, the network connection unit, and the device control unit are connected to a controller; the audio data acquisition unit comprises a microphone assembly, comprising a microphone, an audio processing chip, and a printed circuit board for soldering and assembly, to acquire a target sound; the panoramic image acquisition and AI computing unit comprises a panoramic camera, comprising a plurality of dead-angle-free high-definition ordinary cameras, to acquire panoramic image data inside the smart classroom; the network connection unit comprises a WIFI6 wireless network unit, a fiber optic access unit, and an RJ45 wired network unit, to access an ordinary computing device, an loT device, and other wireless devices; and the device control unit is configured to control the loT device through an RS232 bus interface.

2. The multi-modal edge computing control system for a smart classroom according to claim 1, wherein the audio data acquisition unit is configured to acquire a plurality of sounds in different directions and perform noise reduction on the sounds, and provide a noise reduction method for a voice signal, and the method comprises the following steps:SI, picking up sound data through a multi-microphone array formed by microphones, and pre-storing the sound data to a memory of the system;S2, preprocessing voice data to be processed, to obtain audio data of a first feature; and calling a deep learning algorithm to perform preliminary processing on the voice data, comprising removing environmental background noise in the voice data and leaving clear human voice;S3, calling a preset second-order noise reduction algorithm to perform secondary processing on the audio data of the first feature, and filtering out smooth noise in the voice data, to obtain data of a second feature;S4, inputting the data of the first feature and the data of the second feature to a preset noise reduction deep learning algorithm, and filtering out transient noise in the voice data to be processed, to obtain data of a third feature; andS5, determining noise-reduced voice data based on the data of the first feature, the data of the second feature, and the data of the third feature.

3. The multi-modal edge computing control system for a smart classroom according to claim 1, wherein the panoramic image acquisition and AI computing unit is configured to solve the sound and images acquired by the panoramic camera based on a broad learning algorithm model, to analyze concentration and emotion of students and a learning atmosphere.

4. The multi-modal edge computing control system for a smart classroom according to claim 3, wherein the solving the sound and images acquired by the panoramic camera based on a broad learning algorithm model comprises the following steps:SI, training the broad learning algorithm model based on data of teachers and students, namely, user data, of the smart classroom;S2, transferring the user data, environmental sound data, and image data simultaneously to the trained broad learning algorithm model and a Hidden Markov Model (HMM) emotion computing model;S3, predicting behaviors of teachers and students in teaching and learning processes based on the broad learning algorithm model, to understand language and action intentions of teachers and students;S4, transferring understood language and action intention data of teachers and students into the HMM emotion computing model for data preprocessing, and converting the user data, smart classroom environment data, and preprocessed intention data into data expression information understandable by the HMM emotion computing model;S5, extracting feature data related to teachers and students for emotion recognition of teachers and students;S6, establishing the HMM emotion computing model, and correcting the HMM emotion computing model by using the smart classroom environment data and intention data as indicators of sensitive factors; andS7, outputting emotion expression data that meets needs of teachers and students.

5. The multi-modal edge computing control system for a smart classroom according to claim 1, wherein the device control unit is configured to control the loT device through the RS232 bus interface in ways of controlling a plurality of infrared-receiving remote control devices in the smart classroom and accelerating the devices based on a deep learning algorithm, and generating a control command using a multi-microphone array and the panoramic camera, for voice or gesture control.

6. The multi-modal edge computing control terminal for a smart classroom according to any one of claims 1 to 5, comprising a device housing, wherein the device housing is internally provided with the printed circuit board and the controller, the printed circuit board is connected to the controller, the printed circuit board is provided with the microphone and the audio processing chip, one side of the device housing is provided with a peripheral interface and a keypad, a side surface of the device housing adjacent to the keypad is provided with a wireless network interface, a side surface of the device housing opposite to the wireless network interface is provided with a wired network interface and a power input interface, a side surface of the device housing opposite to the peripheral interface and the keypad is provided with the panoramic camera, and the microphone, the audio processing chip, the peripheral interface, the keypad, the wireless network interface, the wired network interface, the power input interface, and the panoramic camera are all connected to the controller.

Images