Interactive picture book system for teaching intervention of children with autism

Abstract

The invention discloses an interactive picture book system for teaching intervention of children with autism, and belongs to the crossing field of education science and technology and auxiliary medical technology. Comprising an entity picture book module, a hardware interaction module and a software interaction module. The entity picture book is provided with a special material touch area and a physical button; the hardware interaction module adopts an Arduino development board to realize input and output control, and comprises a touch sensing unit, an audio output unit, an LED lighting effect unit and a Bluetooth communication unit; and the software interaction module is developed based on Procesing, and plays a corresponding teaching video after receiving a signal through Bluetooth. According to the system, collaborative feedback of touch, hearing and vision is achieved through multi-mode interactive design, action cognition can be enhanced through touch-audio-visual linkage in one-to-one teaching, the participation degree can be synchronously improved through Bluetooth animation in a collective class, meanwhile, the operation advantage of an entity picture book is kept, and the system is more practical. And the learning concentration and the imitation ability of the autism children are effectively improved.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of educational technology and assistive medical technology, specifically involving an interactive picture book system for teaching intervention of children with autism that combines computer software, hardware interaction and multimodal feedback technology. Background Technology

[0002] In the field of autism education, the effectiveness of teaching tools is crucial to children's learning and development. However, existing autism teaching tools have many problems:

[0003] 1. Single-sensory interaction mode: Traditional autism teaching tools, such as ordinary picture books and electronic tablets, often focus only on stimulating a single sense. Ordinary picture books mainly present content through visual means and lack auditory and tactile feedback; although electronic tablets can provide some audiovisual experience, they also lack tactile interaction, making it difficult to stimulate children's multi-sensory learning, resulting in poor learning outcomes.

[0004] 2. Poor adaptability to different scenarios: Existing systems struggle to meet the needs of various teaching scenarios simultaneously. In one-on-one teaching, there is a need to strengthen children's motor cognition, but existing tools cannot provide effective means; in group classes, it is difficult to increase group participation and fully motivate children's learning enthusiasm.

[0005] 3. Lack of physical interaction: Purely digital tools, such as tablet applications, although feature-rich, cannot provide the tactile experience of physical picture books, which can easily cause children to lose focus; while ordinary physical picture books lack dynamic feedback and cannot effectively interact with children, making it difficult to meet the needs of modern education. Summary of the Invention

[0006] The purpose of this invention is to address the problems in existing technologies, such as the difficulty in stimulating multi-sensory learning due to a single interaction mode, poor adaptability to different scenarios and inability to meet diverse teaching needs, and poor learning experience due to the lack of physical interaction. This invention provides an interactive picture book system for the educational intervention of children with autism, which achieves tactile, auditory, and visual synergistic feedback through multimodal interaction design, effectively meeting the needs of one-on-one teaching and group classes, and combining physical and digital elements. This system achieves tactile, auditory, and visual synergistic feedback through multimodal interaction design, meeting the needs of different teaching scenarios and effectively improving the learning outcomes of children with autism.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solutions.

[0008] An interactive picture book system for educational intervention in children with autism includes:

[0009] The physical picture book module is equipped with a special material touch area and physical buttons. The special material touch area detects touch operations through a press-triggered circuit; the physical buttons are used to trigger interactive signals.

[0010] The hardware interaction module includes a main control unit, a touch sensing unit, an audio output unit, an LED light effect unit, and a Bluetooth communication unit. The main control unit is used for input / output control and signal processing. The touch sensing unit connects to the physical picture book module and is used to detect the operation of the special material touch area and physical buttons, converting pressure signals into electrical signals. The audio output unit is used to store and play teaching audio. The LED light effect unit is used to provide visual feedback. The Bluetooth communication unit is used to transmit signals to the software interaction module.

[0011] The software interaction module is used to receive operation signals via the Bluetooth communication unit and play the corresponding teaching video. At the same time, it sends audio and LED control commands to the hardware module to drive the audio output unit and the LED light effect unit to provide coordinated feedback.

[0012] The system achieves teaching intervention through a multimodal feedback loop of tactile (physical picture book manipulation), auditory (voice playback), and visual (LED light effects + teaching videos).

[0013] Furthermore, the special material touch area uses safe materials with different textures, such as basketball and soccer ball textures; the physical buttons use highly durable microswitches. Both the special material touch area and the physical buttons are made of materials that meet child safety standards.

[0014] Furthermore, the main control unit uses an Arduino development board, with its digital input pins connected to the touch sensing unit, digital output pins connected to the LED lighting unit, and serial port pins connected to the Bluetooth communication unit and audio output unit. The Arduino development board has abundant digital and analog input / output pins, and can perform filtering, amplification, and other preprocessing operations on the electrical signals transmitted by the touch sensing unit through built-in pre-programmed signal processing algorithms.

[0015] Furthermore, the touch sensing unit employs a membrane switch. The highly sensitive membrane switch is attached to the underside of the touch area made of a special material, converting pressure signals into electrical signals in real time.

[0016] Furthermore, the audio output unit uses an MP3 module.

[0017] Furthermore, the LED light effect unit uses yellow LED beads and a driving circuit. The driving circuit is controlled by the level signal output by the main control unit to realize the on / off state, blink frequency adjustment, etc., and provide visual feedback.

[0018] Furthermore, the Bluetooth communication unit uses the HC-05 module, which enables bidirectional signal transmission between the hardware and software modules based on the standard Bluetooth protocol; it achieves data transmission between the hardware and software interaction modules at a stable and interference-resistant transmission rate. The hardware interaction module transmits operation signals to the software interaction module through this module, and the software interaction module sends audio playback commands and LED control commands to the hardware interaction module through this module.

[0019] Furthermore, the software interaction module, based on the Processing development environment, receives Bluetooth signals through the computer's Bluetooth adapter, parses and processes the signals, retrieves and plays corresponding teaching videos from the teaching resource library on the local storage device, and sends control commands to the hardware interaction module to achieve coordinated control of the audio output unit and the LED light effect unit.

[0020] Preferably, the software interaction module has a built-in instruction parsing program that accurately parses the received Bluetooth signal according to preset instruction rules, and the teaching videos are in high-definition format, including various action demonstrations and knowledge explanations suitable for children with autism.

[0021] Preferably, the working logic of the audio output unit is as follows: the main control unit receives audio instructions from the software interaction module; sends playback instructions to the MP3 module via the serial port; the MP3 module retrieves the audio file from the storage device and drives the speaker to play it through the amplification circuit.

[0022] Preferably, the working logic of the LED light effect unit is as follows: the main control unit outputs a PWM signal according to the instructions of the software interaction module; the driving circuit adjusts the brightness and flashing frequency of the LED beads to synchronize with the video content and voice playback.

[0023] Preferably, the program logic of the software interaction module includes: listening for Bluetooth commands via a serial port; retrieving the corresponding teaching video from the local resource library after parsing the command; and sending the audio number and LED control parameters to the hardware module via Bluetooth while playing the video.

[0024] Throughout the system's operation, the physical picture book module serves as the user's entry point, triggering operations through a special material touch area and physical buttons. The touch sensing unit of the hardware interaction module captures these signals and transmits them to the main control unit. After filtering and amplifying, the main control unit transmits the signals to the software interaction module via Bluetooth. Upon receiving the signals, the software interaction module plays local teaching videos and simultaneously controls the audio output unit of the hardware interaction module to play corresponding audio, while the LED light effect unit provides visual feedback. This achieves multi-modal interaction with multiple modules, completing a full closed loop from user operation to system feedback.

[0025] The system of this invention includes a one-on-one teaching mode and a group class mode, wherein:

[0026] One-on-one teaching mode: When a user touches the special material touch area of ​​the picture book, the membrane switch detects the signal, and the Arduino development board sends the command to the software interaction module via Bluetooth. After receiving the command, the software interaction module plays the corresponding audio, such as "clapping," while simultaneously lighting up the LED beads in the LED light effect unit and displaying a demonstration video of the gesture. Teachers can guide users to imitate the actions in the video, and the system reinforces children's understanding of the actions through repeated feedback.

[0027] Group lesson mode: The teacher sends a "group instruction" through the main control unit, which is broadcast to the computer in the group class via Bluetooth communication unit; the computer screen plays relevant animations (such as "kicking a ball animation"), and users operate physical buttons on the picture book (such as "football button") to trigger interaction, thereby improving the consistency of participation of all children and enhancing the effectiveness of group teaching.

[0028] Compared with the prior art, the present invention has the following outstanding technical effects and advantages:

[0029] 1. Multimodal Interaction Revolutionizes the Learning Experience: Traditional teaching tools have a single interaction mode, making it difficult to engage the multiple senses of children with autism in learning. This invention provides tactile perception through a special material touch area, auditory feedback through an audio output unit, and visual stimulation through an LED light effect unit and teaching videos, constructing a synergistic feedback mechanism of touch, hearing, and vision. This multimodal interactive design can fully activate children's sensory system, greatly improving children's attention and acceptance of learning content compared to a single mode, and significantly enhancing learning outcomes.

[0030] 2. Flexible Adaptation to Diverse Teaching Scenarios: Existing teaching systems cannot simultaneously meet the different needs of one-on-one teaching and group classes. This invention utilizes Bluetooth communication technology. In one-on-one teaching, children's touch operations can instantly trigger personalized audiovisual feedback, helping teachers to specifically reinforce children's motor cognition. In group classes, teachers can broadcast commands via Bluetooth to synchronize large-screen animations with children's picture book operations, effectively improving the classroom participation of all children and solving the problem of poor adaptability of traditional tools.

[0031] 3. Integrating the advantages of both physical and digital: Purely digital tools lack the tactile experience of physical interaction, easily leading to children's distraction; while ordinary physical picture books lack dynamic interaction. This invention retains the authentic tactile experience of physical picture books while providing them with digital dynamic feedback functions. This allows children to receive timely audio and video responses as they touch and press the physical picture book, combining the immersive experience of physical interaction with the richness of digital content to create a higher-quality, more focused learning tool for children with autism. Attached Figure Description

[0032] Figure 1 This is a structural block diagram of the interactive picture book system of the present invention.

[0033] Figure 2 This is a circuit diagram of the hardware interaction module of the present invention.

[0034] Figure 3 This is a schematic diagram of the audio output unit of the present invention.

[0035] Figure 4 This is a schematic diagram of the structure of the LED light effect unit of the present invention.

[0036] Figure 5 This is a schematic diagram of the structure of the Bluetooth communication unit of the present invention.

[0037] Figure 6 This is a schematic diagram of the interactive scenario of the one-on-one teaching mode of the present invention.

[0038] Figure 7 This is a schematic diagram of the interactive scenario of the group class mode of the present invention.

[0039] Figure 8 This is a perspective view of the interactive picture book system of the present invention.

[0040] Figure 9 This is the bottom view of the interactive picture book system of the present invention.

[0041] Figure 10 This is a diagram showing the internal structure of the interactive picture book system of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the following embodiments will be used in conjunction with the accompanying drawings to further illustrate the invention. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0043] like Figure 1 As shown in the figure, an embodiment of an interactive picture book system for teaching intervention for children with autism according to the present invention includes a physical picture book module, a hardware interaction module, and a software interaction module. The operation signals of the physical picture book module are processed by the hardware interaction module (including a main control unit, a touch sensing unit, etc.) and then transmitted to the software interaction module via a Bluetooth communication unit. The software module feeds back control commands to the hardware module to achieve audio and LED light effect output, forming a multimodal teaching feedback closed loop.

[0044] The physical picture book module features a touch area made of a special material (such as a material simulating the surface texture of a basketball or soccer ball) and physical buttons, both made of materials that meet child safety standards. The touch area, triggered by pressing, activates the internal circuitry to detect touch operations. The physical buttons use highly durable microswitches to trigger specific interactive signals, providing children with diverse operation methods. The electrical signals generated by both are transmitted via wires to the digital input pins of the Arduino development board (main control unit).

[0045] The hardware interaction module includes a main control unit and functional units. The main control unit uses an Arduino development board (model Nano), which has abundant digital input / output pins and analog input / output pins, stable operating performance, and an open-source programming environment, enabling convenient implementation of the input / output control logic of the entire hardware interaction module. The functional units include a touch sensing unit, an audio output unit, an LED light effect unit, and a Bluetooth communication unit.

[0046] Figure 2 This is a circuit connection diagram of the hardware interaction module in this invention. It includes an Arduino main control board (Nano model), a high-sensitivity membrane switch, an MP3 module, an LED driver circuit, and a Bluetooth module (HC-05). Each module is connected through a digital I / O interface, a TX / RX serial port, and control pins to realize instruction processing, signal transmission, and feedback output.

[0047] The battery provides DC power. The main control board connects to the physical buttons of the physical picture book via digital input pins to detect press signals, and digital output pins control LED beads to provide visual feedback. Serial port pins (TX / RX) are cross-connected to a Bluetooth module (such as HC-05) for bidirectional Bluetooth communication, transmitting operation signals and control commands. Audio output pins connect to a speaker to play educational audio. During operation, children can operate the buttons or touch areas... Figure 2 (Illustrated on the right) The Arduino Nano detects and encodes changes in electrical levels, sending them via Bluetooth to the software module. The software parses the data and provides feedback commands, which the main control board then uses to drive the speaker and LEDs, creating auditory and visual feedback. This design features multi-input expansion (10 buttons supporting rich interactions), modular integration (unified component management for easy assembly and maintenance), and low-power stability (5V battery adaptability for mobile teaching, and the Arduino open-source ecosystem ensuring reliable signal processing and communication), providing hardware support for the system's physical implementation and multimodal teaching functions (such as haptic-audiovisual linkage).

[0048] The touch sensing unit uses a high-sensitivity membrane switch, which is thin, durable and fast-responding. It can accurately detect the operation of touch areas made of special materials and physical buttons, and convert pressure signals into electrical signals in real time and transmit them to the Arduino development board.

[0049] Figure 3 This is a schematic diagram of the audio output unit of the present invention, showing the connection relationship between the audio output unit and the Arduino main control board in the hardware interaction module:

[0050] Left side pins: Correspond to the interfaces of the Arduino development board, including analog input pin (AG), digital pins (13, 12), power supply pin (5V, providing +5V DC power) and ground pin (GND, negative power supply).

[0051] Right-side audio output unit: Taking the MP3 module as an example, its interface includes:

[0052] TXO (transmit pin): Connects to the RXO (receive pin) of the Arduino and is used by the audio module to send status information (such as a playback completion signal) back to the main control board.

[0053] RXO (Receive Pin): Connects to the TXO (Transmit Pin) of the Arduino to receive audio playback control commands (such as "Play the first audio segment") sent by the main control board.

[0054] VCC (Positive Power Supply): Connected to the Arduino's 5V pin to obtain operating power;

[0055] GND (Ground Pin): Shares the same ground as the Arduino's GND pin to ensure a stable electrical circuit.

[0056] Working principle: When the Arduino main control board receives an audio playback command (such as "play 'clapping hands' voice") from the software interaction module, it sends a serial port command to the RXO pin of the audio module via the TXO pin. After parsing the command, the audio module retrieves the corresponding audio file (such as MP3 format) from its built-in storage (or external storage), processes it through the signal amplification circuit, and drives the speaker to play the teaching audio, realizing auditory feedback function (such as action commands, encouraging sound effects, etc.). This, in conjunction with visual feedback (LED light effects, video playback), constructs a multimodal interactive experience.

[0057] Figure 4 This is a schematic diagram of the LED light effect unit of the present invention. The Arduino Nano is connected to the LED light effect unit through a digital output pin, providing power (5V) and ground (GND), and outputting control signals. The LED light effect unit includes yellow LED beads and a driving circuit, which receives commands from the Arduino Nano (main control board) to turn the LED beads on and off, providing children with intuitive visual feedback (such as synchronization with video playback and voice commands, strengthening the memory association of tactile operation).

[0058] Figure 5 This is a schematic diagram of the Bluetooth communication unit of the present invention. The Bluetooth communication unit adopts the HC-05 module, including a power supply pin, a TX / RX serial port pin, and a status indicator light. It supports the standard Bluetooth communication protocol, has stable data transmission performance and strong anti-interference capability, and realizes bidirectional signal transmission between the hardware interaction module and the software interaction module.

[0059] The Arduino development board serves as the core, with its digital input pins receiving electrical signals from the touch sensing unit. These signals undergo pre-processing, including filtering and amplification, by the onboard processor and pre-programmed algorithms to remove environmental interference and signal noise, ensuring signal accuracy and effectiveness. The processed signal is then transmitted via the TX pin of the Arduino development board to the RX pin of the Bluetooth communication unit (HC-05 module). Simultaneously, the TX pin of the Bluetooth communication unit is connected to the RX pin of the Arduino development board, enabling bidirectional data transmission. The Bluetooth communication unit follows the standard Bluetooth communication protocol, sending the processed signal at a stable transmission rate to the Bluetooth adapter of the computer hosting the software interaction module.

[0060] In the audio output unit circuit, the MP3 module is connected to the Arduino development board. When the Arduino development board receives an audio playback command from the software interaction module, it sends a control signal to the MP3 module through its digital output pin. The MP3 module reads the teaching audio data from the storage device, processes it through the amplification circuit, and then drives the speaker to play the teaching audio. In the LED light effect unit circuit, the yellow LED is connected to the digital output pin of the Arduino development board through a driver circuit. The Arduino development board outputs different level signals to control the driver circuit, realizing the adjustment of the LED's on / off state, flashing frequency, and brightness, providing visual feedback.

[0061] The software interaction module runs on the computer, receiving Bluetooth signals through the computer's Bluetooth adapter. After processing by the instruction parsing program, it retrieves the corresponding teaching videos from the local teaching resource library. On one hand, the software interaction module controls the computer's display device to play the videos; on the other hand, it sends control commands to the Arduino development board via Bluetooth to achieve coordinated control of the audio output unit and the LED lighting unit, ultimately achieving multimodal interactive teaching functions involving touch, hearing, and vision.

[0062] The software interaction module is developed using the Processing development environment, possessing excellent cross-platform compatibility and a rich library of graphics and multimedia processing tools. After receiving a Bluetooth signal via the computer's Bluetooth adapter, the software interaction module performs the following operations:

[0063] 1) The received signal is parsed and processed. According to the preset instruction rules, the corresponding teaching video is quickly retrieved from the teaching resource library of the local storage device and played. The teaching video is in high-definition format and covers a variety of action demonstrations and knowledge explanations suitable for children with autism.

[0064] 2) Based on teaching needs and user operations, send control commands to the hardware interaction module to achieve coordinated control of the audio output unit and the LED light effect unit, and build a multimodal interactive experience.

[0065] Example 1: One-on-one teaching scenario

[0066] Figure 6 This is a schematic diagram of the interactive scenario of the 1-on-1 teaching mode of the present invention. In the 1-on-1 teaching scenario, all modules of the interactive picture book system work closely together to form a complete and efficient teaching interaction system. The child interacts with the physical picture book module. When the child places their hand on the "material touch area" of the picture book (such as an area simulating the texture of a basketball surface), the membrane switch below the special material touch area (as the core component of the touch sensing unit in the hardware interaction module) changes pressure, causing its internal circuit to conduct and generating an electrical signal. This electrical signal is transmitted from the physical picture book module to the touch sensing unit of the hardware interaction module through pre-laid and tightly connected wires, and then stably input to the digital input pins of the Arduino development board (main control unit, model Nano).

[0067] After receiving a signal, the Arduino development board uses its pre-programmed signal processing algorithms to perform pre-processing operations such as filtering and amplification. The filtering process effectively removes noise signals generated by environmental electromagnetic interference, while the amplification process ensures that weak, effective signals reach an accurately identifiable strength, thus guaranteeing signal accuracy and effectiveness. The processed signal is then transmitted via the TX pin of the Arduino development board to the RX pin of the Bluetooth communication unit (HC-05 module). The Bluetooth communication unit strictly adheres to the standard Bluetooth communication protocol, transmitting the signal at a stable and interference-resistant transmission rate to the Bluetooth adapter of the computer hosting the Processing-based software interaction module, achieving reliable data transmission between the hardware and software interaction modules.

[0068] After receiving the signal, the software interaction module uses its built-in instruction parsing program to accurately analyze the signal instruction and quickly retrieve the corresponding "clapping" instructional video and audio file from the locally stored structured teaching resource library. The software interaction module sends precise audio playback instructions to the MP3 audio output unit of the hardware interaction module through the computer's audio output interface, controlling it to play the "clapping" audio at an appropriate volume and rhythm. Simultaneously, it sends control instructions to the Arduino development board, outputting specific level signals through digital output pins to control the yellow LED beads in the LED light effect unit, causing them to flash at a frequency simulating the clapping rhythm. Finally, it plays a high-definition demonstration video of the "clapping" action on the computer display device; the video is vividly colored and the movements are broken down in detail, making it easy for children to observe and learn.

[0069] In this complete interactive process, the physical picture book module serves as the intuitive entry point for user operation, playing a crucial role in receiving children's touch input. The hardware interaction module is responsible for the entire process of signal acquisition, processing, and transmission, as well as the precise execution of control commands from the software interaction module. The software interaction module focuses on the efficient retrieval of teaching resources and multimedia playback control. These three modules cooperate and work together to achieve multimodal interaction involving touch and audiovisual stimulation, effectively enhancing children's cognitive understanding of actions. Teachers observe children's reactions in real time, guiding them to carefully observe and imitate the action demonstrations in the video. If a child makes a clapping motion, the teacher provides timely encouragement and guides the child to touch the "material touch area" again. The system then repeats the entire interactive process, continuously reinforcing children's cognition and memory of the clapping motion through repeated feedback. If a child fails to imitate immediately, the teacher provides hands-on assistance to help the child complete the action, while the system continuously provides feedback until the child can complete the action independently.

[0070] Example 2: Group Lesson Scenario

[0071] Figure 7 This diagram illustrates an interactive scenario in the group lesson mode of this invention. In the group lesson scenario, the various modules of the system construct an orderly collaborative network through Bluetooth communication, achieving efficient group teaching interaction. Before the lesson begins, the teacher uses the main control device to pair and connect with the computer used for teaching in the group class and the interactive picture books in the hands of all the children via Bluetooth. Through a strict pairing and authentication process, a stable communication network for the entire system is established.

[0072] During classroom teaching, the teacher sends "collective instructions" (such as "Let's learn how to kick a ball together") through the main control terminal according to a carefully designed teaching plan. This instruction is broadcast via the RF transmission function of the Bluetooth communication unit (HC-05 module) and stably transmitted to the hardware interaction module of each children's picture book and the group teaching computer. Upon receiving the instruction, the group teaching computer retrieves a high-definition "ball-kicking animation" from its local large-capacity animation resource library and plays it smoothly on the large screen. The animation content is professionally designed, including detailed teaching details such as the standard kicking posture, the way force is applied, and the trajectory of the ball. Simultaneously, the computer sends an initialization signal to each child's picture book via Bluetooth, putting the picture book's hardware interaction module into a ready state to receive children's operation signals. At this time, the Arduino development board's signal processing program is in standby mode, waiting for input signals from the touch sensing unit.

[0073] After watching the animation on the large screen, children operate the "soccer button" on the picture book. The touch-sensing unit (membrane switch) of the hardware interaction module, with its sensitive pressure-sensing characteristics, detects the button press and generates an electrical signal. This signal is transmitted through wires to the signal conditioning circuit of the touch-sensing unit, where it undergoes preliminary shaping and amplification before being input to the digital input pin of the Arduino development board. The Arduino development board encodes the signal, adds a unique device identification code, and transmits it via the TX pin to the Bluetooth communication unit. From there, it is sent to the Bluetooth adapter of the classroom computer via the modulation and transmission function of the HC-05 module.

[0074] After receiving signals from children, the computer in the group teaching session quickly and accurately identifies the specific child performing the action based on the device identification code. Then, it invokes a pre-set animation enhancement program to personalize the "kicking animation," for example, making the virtual character of the corresponding child in the animation perform more exaggerated and guiding kicking movements. Simultaneously, it plays a customized encouraging sound effect specific to that child (such as "[Child's Name], great kick!") through the computer's high-fidelity audio output interface. In this group teaching interaction, the physical buttons of the physical picture book module serve as the trigger source for the operation signals. The hardware interaction module completes the signal acquisition, processing, encoding, and transmission, while the software interaction module (running on the group teaching computer) is responsible for receiving signals, accurately processing them, and intelligently controlling the playback of animation and audio. All modules are closely connected via stable Bluetooth communication, enabling active interactive participation from all children in group teaching and effectively improving the consistency and enthusiasm of group learning. Teachers continuously observe children's participation in the classroom, providing targeted individual guidance to children who are not proficient in operation or have low participation, ensuring that the system functions fully and achieves good teaching results. This invention retains the tactile advantages of physical picture books while enhancing dynamic interactive functions, thereby improving the concentration and imitation abilities of children with autism.

[0075] Figures 8-10 This is a schematic diagram of the interactive picture book's appearance structure according to the present invention. It includes a main control module box, a switch controller, a Micro-USB charging port, and the interactive picture book. The main control module box and the interactive picture book are connected by wires. The switch controller and the Micro-USB charging port are located on the main control module box.

[0076] like Figure 10 The HC-05 Bluetooth module is used for Bluetooth communication between the hardware interaction module and the software interaction module (computer) to transmit operation signals and control commands (such as signals from children pressing buttons and feedback commands from the software module).

[0077] The Arduino Nano motherboard is the core control unit, used to receive operation signals from the physical picture book (such as connection signals from the button module and LED module), process them, and coordinate the work of each module (such as driving the speaker to play voice and controlling the LED to blink).

[0078] The 5V battery module provides the system's operating power (+5V DC), ensuring that the hardware modules can still operate normally when disconnected from an external power source (such as in mobile teaching scenarios). It is connected to the Arduino and other modules (such as speakers and LEDs) via wires.

[0079] The speaker module connects to the audio output pin of the Arduino to play teaching voice (such as action commands and encouraging sound effects), realizing auditory feedback function, and working in conjunction with visual feedback (LED) and tactile operation (physical picture books).

[0080] The switch controller is used for starting and stopping the system power supply (such as turning on / off the power supply to hardware modules), which makes it convenient for teachers or operators to manage the equipment.

[0081] The Micro-USB charging port charges the 5V battery module and supports the USB standard charging protocol to ensure power supply capability (such as the system can work with an external power source while charging).

[0082] Internal wires connect the various modules (such as the electrical connection between Arduino and LEDs, buttons, and speakers), while external wires connect to the touch area / button module of the physical picture book. Figure 10 The right-hand side of the middle area is connected to realize the acquisition and feedback output of operation signals.

[0083] The following is a description of the program control logic of this invention (implemented in Arduino code):

[0084] To realize the interactive function of this invention, the following main control logic can be programmed onto the Arduino main control board in the hardware interaction module:

[0085] 1. Initialization settings: In the setup() function, the system is configured with 11 button input ports and 4 LED output ports. At the same time, the audio module (GD5800) and serial communication rate (9600bps) are initialized, and the playback volume is set.

[0086] 2. User Input Detection: In the `loop()` main loop, the program continuously checks the state of all buttons. When any button is pressed, after confirming that its trigger is valid, the button number `buttonFlag` is recorded.

[0087] 3. Corresponding response trigger: The switch-case statement executes a preset operation for each button number, including: sending a character to the serial port (e.g., sending "A" for button 1) for Bluetooth module communication; calling mp3.playFileByIndexNumber(n) to play the corresponding audio number; controlling the on / off state of the corresponding LED for visual cues; setting a polling condition for waiting for the button to be released; and finally resetting the button state to enter the next round of detection.

[0088] 4. Bluetooth Communication Design: Serial communication sends letters (AK) representing the operation content to the Bluetooth module (HC-05) via Serial.print(). The software interaction module receives and parses the command and then starts video playback and feedback control.

[0089] 5. LED and MP3 Synergistic Output: In addition to voice feedback, some buttons also use LED lights to flash, improving the synchronization and user experience of multimodal feedback.

[0090] The control program has a clear overall logic and a stable structure, and runs on Arduino Nano. Combined with the hardware design of this system, it achieves a complete teaching loop of button input—feedback output—Bluetooth communication—software linkage.

[0091] The following is a description of the program logic of the software interaction module of this invention (Processing side):

[0092] To realize the teaching video playback and serial communication functions of the software interaction module in the system of this invention, the following program logic was written based on the Processing platform to realize the dynamic calling of teaching video resources and serial port command parsing.

[0093] 1. Serial communication initialization: Use the Serial class in Processing to create a serial port connection object myPort, specify the serial port number (COM3) and baud rate (9600), and establish communication with the Arduino main control module.

[0094] 2. Preloading of teaching videos: When the program starts, 11 teaching video animation files (paths such as / data / animation1.mp4 to / animation11.mp4) are preloaded through the setup() function and stored in the animations array of type Movie[] to support quick access later.

[0095] 3. Serial Port Event Handling and Command Parsing: Use the serialEvent() event function to listen for Bluetooth serial port data; when a character (such as "A" ~ "K") is received, convert it into the corresponding array index number (0~10); call animations[n].play() to play local teaching videos.

[0096] 4. The main drawing function draw(): Clears the background and displays the status information (currently received data); if there is a selected animation (i.e., currentAnimation ≠ -1), the corresponding video is played on the canvas to provide visual feedback.

[0097] 5. Video frame processing: The movieEvent(Movie m) callback function is used to read video frames one by one to ensure continuous picture.

[0098] The program logic works in conjunction with the button triggering logic in the Arduino main control board. When a child operates the touch button of the picture book, the Bluetooth module sends a command to the Processing program, and the system can play the corresponding teaching video, thus building a multimodal closed-loop interactive experience of "tactile input → voice feedback + video playback".

[0099] This invention triggers a dynamic audiovisual feedback loop through tactile manipulation of physical picture books, achieving multimodal coordinated intervention involving touch, hearing, and vision. The special material touch area and physical buttons serve as input media, and the hardware module detects and transmits signals. The software module synchronously drives the output of teaching videos, customized voice, and LED light effects, forming a real-time linkage of "physical operation - digital feedback," which effectively improves the sensory participation and learning concentration of children with autism.

[0100] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. An interactive picture book system for educational intervention in children with autism, characterized in that... It includes a physical picture book module, a hardware interaction module, and a software interaction module; The physical picture book module is equipped with a special material touch area and physical buttons. The special material touch area detects touch operations through a press-triggered circuit; the physical buttons are used to trigger interactive signals. The hardware interaction module includes a main control unit, a touch sensing unit, an audio output unit, an LED light effect unit, and a Bluetooth communication unit. The main control unit is used to implement input / output control and signal processing. The touch sensing unit connects to the physical picture book module and is used to detect the operation of the special material touch area and physical buttons, converting pressure signals into electrical signals. The audio output unit is used to store and play teaching voice. The LED light effect unit is used to provide visual feedback. The Bluetooth communication unit is used to transmit signals to the software interaction module. The software interaction module is used to receive operation signals through the Bluetooth communication unit and play the corresponding teaching video, while sending audio and LED control commands to the hardware module to drive the audio output unit and LED light effect unit to provide coordinated feedback. The system achieves teaching intervention through a multimodal feedback loop involving touch, hearing, and vision.

2. The interactive picture book system for teaching intervention for children with autism as described in claim 1, characterized in that... The special material touch area uses safety materials with different textures; the physical buttons use highly durable micro switches.

3. The interactive picture book system for teaching intervention for children with autism as described in claim 1, characterized in that... The main control unit uses an Arduino development board. Its digital input pins are connected to the touch sensing unit, its digital output pins are connected to the LED light effect unit, and its serial port pins are connected to the Bluetooth communication unit and the audio output unit.

4. The interactive picture book system for teaching intervention for children with autism as described in claim 1, characterized in that... The touch sensing unit uses a membrane switch, which is attached to the underside of the touch area made of a special material to convert pressure signals into electrical signals in real time.

5. The interactive picture book system for teaching intervention for children with autism as described in claim 1, characterized in that... The audio output unit uses an MP3 module.

6. The interactive picture book system for teaching intervention for children with autism as described in claim 1, characterized in that... The LED light effect unit uses yellow LED beads and a driving circuit. The driving circuit is controlled by the electrical signal output by the main control unit to realize the on / off state of the yellow LED beads and the adjustment of the flashing frequency, providing visual feedback.

7. The interactive picture book system for teaching intervention for children with autism as described in claim 1, characterized in that... The Bluetooth communication unit uses the HC-05 module, which realizes bidirectional signal transmission between the hardware and software modules based on the standard Bluetooth protocol. The hardware interaction module transmits operation signals to the software interaction module through it, and the software interaction module sends audio playback commands and LED control commands to the hardware interaction module through it.

8. The interactive picture book system for teaching intervention for children with autism as described in claim 1, characterized in that... The software interaction module is based on the Processing development environment.

9. The interactive picture book system for teaching intervention for children with autism as described in claim 1, characterized in that... It features a 1-on-1 teaching mode. In this mode, after the special material touch area is activated, the computer software interaction module plays teaching videos, the audio output unit plays teaching voices, and the LED light effect unit simultaneously turns on and off or flashes to enhance children's understanding of the movements.

10. The interactive picture book system for teaching intervention for children with autism as described in claim 1, characterized in that... It features a group class mode where the picture book device sends instructions via Bluetooth to play teaching animations. Children can activate physical buttons to trigger feedback, and the screen displays personalized teaching animations and encouraging sound effects, thereby increasing group participation.

Images