Classroom large-screen device interaction method and device, equipment and readable storage medium

Abstract

The application discloses a classroom large-screen device interaction method and device, equipment and a readable storage medium. The classroom large-screen device comprises a touch screen and a vibration sensor for monitoring a vibration signal generated by the touch screen when the touch screen is touched by an external action. The method comprises: acquiring the vibration signal monitored by the vibration sensor; determining a target action type of the external action based on the vibration signal; acquiring prompt information corresponding to the target action type, and outputting the prompt information in the form of voice broadcast, wherein the prompt information is prompt content related to a teacher's teaching scene. Compared with the existing teaching scene, the application can automatically detect different external actions of the teacher through the classroom large-screen device, and then automatically broadcast the corresponding prompt information, without the need for the teacher to interrupt the teaching content to orally deliver the prompt content, which can better assist the teacher to complete the teaching work.

Description

Technical Field

[0001] This application relates to the field of smart classroom technology, and more specifically, to a classroom large-screen device interaction method, apparatus, equipment, and readable storage medium. Background Technology

[0002] With the development of smart classrooms, teachers are increasingly using smart devices to assist in teaching.

[0003] Taking classroom large-screen devices as an example, classroom large-screen devices can also be called smart blackboards, educational all-in-one machines, etc. They use a large screen for teaching demonstrations, blackboard writing, etc., which facilitates teachers' teaching work and improves students' viewing experience.

[0004] However, existing technologies are still insufficient for the development of classroom large-screen devices. Some basic tasks in teaching scenarios still require teachers to do them themselves. Therefore, how to further develop classroom large-screen devices to better assist teachers in completing their teaching work has become an urgent problem to be solved by those in the field. Summary of the Invention

[0005] In view of the above problems, this application is made to provide a classroom large-screen device interaction method, apparatus, device, and readable storage medium, so as to achieve the goal of better assisting teachers' teaching work with the help of classroom large-screen devices. The specific solution is as follows:

[0006] In a first aspect, a classroom large-screen device interaction method is provided, wherein the classroom large-screen device includes a touch screen and a vibration sensor for monitoring the vibration signal generated by the touch screen when touched by an external action; the method includes:

[0007] Acquire the vibration signal detected by the vibration sensor;

[0008] Based on the vibration signal, the target action type of the external action is determined;

[0009] Obtain prompt information corresponding to the target action type and output the prompt information in the form of voice broadcast, wherein the prompt information is prompt content related to the teacher's teaching scenario.

[0010] In one possible design, in another implementation of the first aspect of this application, if the target action type of the external action is an action type unrelated to the touch position, the process of obtaining the prompt information corresponding to the target action type includes:

[0011] Obtain pre-configured prompt information corresponding to the target action type.

[0012] In one possible design, in another implementation of the first aspect of this application, if the target action type of the external action is an action type related to the touch position, the process of obtaining the prompt information corresponding to the target action type includes:

[0013] The touch position coordinates of the external action on the touch screen are detected, and the screen partition position to which the touch position coordinates belong is determined;

[0014] Obtain prompt information corresponding to the target action type and the screen partition position. The prompt information includes location information, which includes the screen partition position or guidance information that includes the screen partition position.

[0015] In one possible design, in another implementation of the first aspect of this application, the process of obtaining the prompt information corresponding to the target action type and the screen partition position includes:

[0016] Query the pre-configured set of prompt information corresponding to the target action type, wherein each prompt information in the set contains different location information;

[0017] Select the target prompt information from the set of prompt information found. The location information contained in the target prompt information includes the screen partition location, or includes guidance information for the screen partition location.

[0018] In one possible design, in another implementation of the first aspect of this application, the process of obtaining the prompt information corresponding to the target action type and the screen partition position includes:

[0019] Obtain a pre-configured prompt information template corresponding to the target action type, wherein the prompt information template contains position information slots to be filled;

[0020] The screen partition location or the guidance information of the screen partition location is used as location information and filled into the location information slot in the prompt information template to obtain the complete prompt information after filling.

[0021] In one possible design, in another implementation of the first aspect of the embodiments of this application, if the target action type of the external action is an action type related to the touch position, then in addition to outputting the prompt information in the form of voice broadcast, the method further includes:

[0022] The coordinates of the touch position on the touchscreen are displayed in the form of visual markers to indicate the touchscreen position that students should pay attention to.

[0023] In one possible design, in another implementation of the first aspect of the embodiments of this application, the vibration sensor is an acoustic sensor, and the process of determining the target action type of the external action based on the vibration signal includes:

[0024] Based on the magnitude of the vibration signal, determine the target action type of the external action;

[0025] or,

[0026] The distribution pattern of the sound waves is determined based on the vibration signal, and the target action type of the external action is determined based on the distribution pattern of the sound waves.

[0027] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of determining the target action type of the external action based on the magnitude of the vibration signal includes:

[0028] When the vibration magnitude of the vibration signal exceeds a first threshold, the external action is determined to be a slapping type; when the vibration magnitude of the vibration signal exceeds a second threshold but does not exceed the first threshold, the external action is determined to be a knocking type, wherein the second threshold is less than the first threshold.

[0029] The process of determining the target action type of an external action based on the distribution pattern of the sound waves includes:

[0030] When the sound wave distribution pattern belongs to the first mode, the external action is determined to be a slapping type. When the sound wave distribution pattern belongs to the second mode, the external action is determined to be a knocking type. The sound wave distribution in the second mode is more concentrated than that in the first mode.

[0031] In one possible design, in another implementation of the first aspect of the embodiments of this application, the target action type of the external action includes a slapping action type and a knocking action type;

[0032] The prompts corresponding to the types of slapping actions are used to restrain students' bad classroom habits;

[0033] The prompt information corresponding to the tapping action type is used to remind students to pay attention to the content specified by the teacher on the touch screen.

[0034] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of acquiring the vibration signal monitored by the vibration sensor includes:

[0035] Start the SonarSensorService service to activate the vibration sensor and monitor the vibration signal generated by the touchscreen when it is touched by an external action.

[0036] In one possible design, in another implementation of the first aspect of the embodiments of this application, in addition to outputting the prompt information in the form of voice broadcast, the method further includes:

[0037] Prompts are provided on the touchscreen via visual markers.

[0038] Secondly, embodiments of this application provide an interactive device for a classroom large-screen device. The classroom large-screen device includes a touch screen and a vibration sensor disposed on the back of the touch screen. The vibration sensor is used to monitor the vibration signal generated by the touch screen when it is touched by an external action. The device includes:

[0039] A vibration signal acquisition unit is used to acquire the vibration signal detected by the vibration sensor;

[0040] An action type determination unit is used to determine the target action type of the external action based on the vibration signal;

[0041] The prompt information acquisition unit is used to acquire prompt information corresponding to the target action type, wherein the prompt information is prompt content related to the teacher's teaching scenario;

[0042] The prompt information broadcasting unit is used to output the prompt information in the form of voice broadcast.

[0043] Thirdly, embodiments of this application provide a classroom large-screen device, including: a touch screen, a vibration sensor, a memory, and a processor;

[0044] The vibration sensor is used to monitor the vibration signal generated by the touch screen when it is touched by an external action, and transmit it to the processor;

[0045] The memory is used to store programs;

[0046] The processor is used to execute the program to implement the classroom large screen device interaction described in any of the first aspects of this application.

[0047] Fourthly, embodiments of this application provide a readable storage medium storing a computer program thereon, which, when executed by a processor, implements the classroom large-screen device interaction described in any of the first aspects of this application.

[0048] Fifthly, a computer program product is provided, including a computer program that, when executed by a processor, implements the classroom large-screen device interaction described in any of the first aspects of this application.

[0049] Using the above technical solution, this application provides an improved classroom large-screen device and its interaction method. The classroom large-screen device includes a touch screen and a vibration sensor. The vibration sensor can monitor the vibration signal generated by the touch screen when it is touched by an external action. Teachers can trigger different types of actions on the touch screen according to different teaching scenarios, such as patting and tapping. Different types of actions generate different vibration signals. Therefore, this application can analyze the target action type of the external action based on the vibration signal monitored by the vibration sensor. Furthermore, it can obtain the prompt information corresponding to the target action type. The prompt information is related to the teacher's teaching scenario. For example, the prompt information corresponding to the patting action may be "Please pay attention to classroom discipline," and the prompt information corresponding to the tapping action may be "Please pay attention to the content in the middle of the blackboard," etc. Furthermore, the prompt information can be output by voice to remind students to pay attention. Therefore, compared to the existing teaching methods that require teachers to interrupt their lectures to verbally provide prompts, the proposed solution can automatically detect different external actions of the teacher through the classroom's large screen device and then automatically broadcast the corresponding prompts. This eliminates the need for teachers to interrupt their lectures to verbally provide prompts, ensuring the continuity of the teaching content, increasing teaching interactivity, and better assisting teachers in completing their teaching tasks. Attached Figure Description

[0050] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0051] Figure 1 A flowchart illustrating a classroom large-screen device interaction method provided in an embodiment of this application;

[0052] Figure 2 An example of a screen partitioning diagram is provided;

[0053] Figure 3 A schematic diagram of the structure of an interactive device for a classroom large-screen device provided in this application embodiment;

[0054] Figure 4 This is a schematic diagram of the structure of the classroom large-screen device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] Traditionally, classroom large-screen devices are used in school classrooms for teachers to conduct teaching demonstrations or write on the blackboard. However, some basic tasks in this teaching scenario still require teachers to perform manually. For example, when classroom discipline needs to be emphasized, the teacher needs to tap the podium and verbally remind students to pay attention. Similarly, when the teacher needs to emphasize a particular part of the screen content as important, the teacher also needs to speak to emphasize it. These basic tasks interrupt the teacher's teaching content and consume extra energy. Therefore, this application aims to use classroom large-screen devices to enhance teaching interaction, assist teachers in completing some basic tasks during classroom teaching, free up some of their labor, and ensure the continuity of teaching content.

[0057] Therefore, this application provides an improved classroom large-screen device, which includes a touch screen and a vibration sensor. When an external action touches the touch screen, a vibration signal is generated, and the vibration sensor is used to monitor this vibration signal.

[0058] The vibration sensor can take various forms, such as an acoustic sensor, which generates vibrations when the touchscreen is touched by an external force. These vibrations propagate through the air, and the acoustic sensor identifies the air vibration signal. Other types of sensors can also be used, such as piezoelectric sensors. This embodiment uses an acoustic sensor as an example; it can be placed on the back of the touchscreen, such as in the center, to effectively detect vibration signals generated by vibrations at various locations across the entire touchscreen.

[0059] As described in the prior art, in some teaching scenarios, teachers need to verbally provide prompts to remind students. In this embodiment, to replace verbal prompts, different types of actions and corresponding prompts can be pre-set according to different teaching scenarios. Examples of different types of actions include tapping a location on a touchscreen or patting a location on a touchscreen with your palm. These actions generate vibration signals that are detected by a vibration sensor. Different types of actions produce different vibration signals. This application can analyze the vibration signal to determine the target action type of the teacher's action, and then, according to pre-set rules, obtain the corresponding prompts for the target action type and automatically broadcast them verbally, replacing the teacher's verbal prompts.

[0060] This solution can be implemented using a large classroom screen device, which can be equipped with an audio component for broadcasting prompts. Alternatively, the large classroom screen device can communicate with a third-party audio device to broadcast prompts.

[0061] Next, combined Figure 1 The interactive method for classroom large-screen devices described in this application includes the following steps:

[0062] Step S100: Obtain the vibration signal detected by the vibration sensor.

[0063] Specifically, after the classroom large screen device is powered on and the system is started, the system services related to the vibration sensor can be activated, and then the vibration sensor can be activated through the relevant system services to monitor the vibration signal generated by the touch screen when it is touched by external actions.

[0064] Taking a classroom large-screen device using a Windows system as an example, this embodiment can start the SonarSensorService service. The SonarSensorService service is a Windows service used to process sonar sensor data. It provides a mechanism for managing and configuring sonar sensors, and offers related APIs and functions to integrate and use sonar sensors on the Windows operating system. In this embodiment, the vibration sensor can be started using the SonarSensorService service to monitor the vibration signals generated by the touchscreen when it is touched by external actions.

[0065] Step S110: Based on the vibration signal, determine the target action type of the external action.

[0066] Specifically, as mentioned earlier, teachers may have different needs in different teaching scenarios to remind students in different situations. In order to distinguish different prompts, different types of external actions can be set, with different types of external actions corresponding to different prompts.

[0067] Furthermore, different types of external actions produce different vibration signals after touching the touchscreen. Therefore, in this step, the target action type of the external action received by the touchscreen can be analyzed based on the vibration signal detected by the vibration sensor.

[0068] Step S120: Obtain the prompt information corresponding to the target action type, and output the prompt information in the form of voice broadcast, wherein the prompt information is prompt content related to the teacher's teaching scenario.

[0069] Specifically, this application can collect various prompting needs from teachers in different teaching scenarios. For example, in some scenarios, teachers may need to remind students to pay attention to classroom discipline, focus on key content on the screen, speak up, participate in group discussions, randomly select a student to answer a question, or announce the end of class. For each prompt, a corresponding target action type can be pre-set. It is understood that the target action type for each prompt can be pre-set, or teachers can freely set or modify the target action type according to their own habits. For example, the first prompt might be used to regulate students' bad classroom habits, such as "Please pay attention to classroom discipline," "One, two, three, sit up straight," or "Small eyes, look at the blackboard." For this first prompt, the corresponding target action type could be a patting action. The teacher can then gently pat the touchscreen with their palm or part of their palm, and according to this application, the system can automatically recognize the teacher's action type and automatically read the first prompt aloud. For example, the second prompt message is used to remind students to pay attention to content specified by the teacher on the touchscreen, such as "Please pay attention to the content on the blackboard." The corresponding target action type for this second prompt message could be a tapping action. The teacher can tap the touchscreen with their finger or a stylus, and according to this application's solution, the teacher's action type can be automatically recognized, and the second prompt message can be automatically read aloud.

[0070] The above embodiments use slapping and tapping action types as examples for illustration. These slapping and tapping action types can further include specific slapping and tapping counts; that is, different slapping counts correspond to different slapping action types, and different tapping counts correspond to different tapping action types. The slapping and tapping action types referred to herein can be distinguished by attributes such as the magnitude and distribution pattern of the vibration signals generated by the actions. For example, the vibration signal generated by a slapping action is larger than that generated by a tapping action.

[0071] Different action types can be set by default when the device is shipped from the factory, or users can set different action types themselves later. Of course, in addition to the two action types shown in the examples above, teachers can also set other action types, depending on their personal habits.

[0072] This application provides an improved classroom large-screen device and its interaction method. The classroom large-screen device includes a touch screen and a vibration sensor. The vibration sensor can monitor the vibration signal generated by the touch screen when touched by external actions. Teachers can trigger different types of actions on the touch screen according to different teaching scenarios, such as patting and tapping. Different types of actions generate different vibration signals. Therefore, this application can analyze the target action type of the external action based on the vibration signal detected by the vibration sensor, and further obtain the prompt information corresponding to the target action type. This prompt information is related to the teacher's teaching scenario. For example, the prompt information corresponding to the patting action could be "Please pay attention to classroom discipline," and the prompt information corresponding to the tapping action could be "Please pay attention to the content in the middle of the blackboard," etc. Furthermore, the prompt information can be output verbally to remind students to pay attention. It can be seen that compared with the existing teaching scenario where teachers need to interrupt the lecture to verbally pronounce the prompt information, the solution of this application can automatically detect different external actions of the teacher through the classroom large-screen device, and then automatically pronounce the corresponding prompt information, without requiring the teacher to interrupt the lecture to verbally pronounce the prompt information. This ensures the continuity of the teaching content, increases the interactivity of teaching, and can better assist teachers in completing their teaching work.

[0073] In step S110 above, the target action type of the external action is determined. There can be multiple action types for external actions. In this embodiment, an optional classification method is provided, that is, the action types of external actions can be divided into two categories according to whether the reminder intention that the teacher expects to achieve when performing the corresponding type of external action is related to the touch position of the external action on the touch screen. The first category is the action type that is unrelated to the touch position, and the second category is the action type that is related to the touch position.

[0074] For example, a teacher might want to use a tapping motion to remind students to pay attention to important content on the screen where the teacher is tapping. This reminder intention includes specific location information, meaning that the tapping motion touches the location on the touchscreen where the teacher wants the students to focus. Therefore, this reminder intention is an action type related to the user's tapping motion touches the location on the touchscreen, and the tapping motion type can be classified as the second category.

[0075] For example, if a user wants to remind students to pay attention to classroom discipline by patting, the intention of this reminder is obviously unrelated to the patting position on the touchscreen. Therefore, the patting action can be classified as the first type.

[0076] Based on the above classification of action types, after determining the target action type of the external action in step S110, it can be confirmed whether the target action type belongs to the first category or the second category. If it belongs to the first category, that is, the target action type of the external action is an action type unrelated to the touch position, then the process of obtaining the prompt information corresponding to the target action type in step S120 may include:

[0077] Retrieve pre-configured prompts corresponding to the target action type.

[0078] Specifically, for each action type belonging to the first category, this application can pre-configure the prompt information corresponding to each action type, as shown in Table 1 below:

[0079] Table 1

[0080]

[0081]

[0082] When it is determined in step S110 that the target action type of the external action belongs to the first category, the pre-configured information can be queried to obtain the prompt information corresponding to the target action type.

[0083] In another scenario, if the target action type of the external action determined in step S110 belongs to the second category, that is, the target action type of the external action is an action type related to the touch position, then the process of obtaining the prompt information corresponding to the target action type in step S120 may include:

[0084] S1. Detect the touch position coordinates of the external action on the touch screen and determine the screen partition to which the touch position coordinates belong.

[0085] Combination Figure 2 As shown, it illustrates a schematic diagram of a touchscreen partition.

[0086] Figure 2 The example divides the entire touchscreen screen into nine areas: "top left," "top," "top right," "left," "center," "right," "bottom left," "bottom right," and "bottom right." The areas of each section can be the same or different.

[0087] certainly, Figure 2 This is just one option for partitioning the touchscreen; other partitioning strategies can also be used, such as dividing it into a different number of sub-regions.

[0088] The touchscreen can detect the coordinates of external actions on the screen, and then determine the screen partition to which the coordinates of the touch position belong according to the partition.

[0089] S2. Obtain prompt information corresponding to the target action type and the screen partition position. The prompt information includes location information, which includes the screen partition position or guidance information that includes the screen partition position.

[0090] Specifically, the prompt information obtained in this step corresponds to the target action type and is also related to the screen partition location determined in the previous step. The prompt information contains location information, which can be the screen partition location itself or guidance information indicating the screen partition location. Since there may be multiple ways to define the screen partition location, the prompt information can directly contain the screen partition location or it can be another way of expressing the screen partition location, namely, the guidance information for the screen partition location in this article.

[0091] For example, if a teacher taps the center of the screen to prompt students to pay attention to the content in that area, the resulting message could be, "Please pay attention to the content in the center of the blackboard."

[0092] There are several ways to obtain the prompt information in step S2 above. This embodiment introduces several optional implementation methods:

[0093] The first type

[0094] This application can pre-configure a set of prompt information corresponding to each action type, and the location information contained in each prompt information in the set is different.

[0095] Taking the aforementioned tapping action type as an example, its corresponding set of prompt messages can include the following:

[0096] Message 1: Please pay attention to the content in the upper left corner of the blackboard.

[0097] Message 2: Please pay attention to the content above the blackboard.

[0098] Message 3: Please pay attention to the content in the upper right corner of the blackboard.

[0099] Message 4: Please pay attention to the content on the left side of the blackboard.

[0100] Message 5: Please pay attention to the content in the middle of the blackboard.

[0101] Message 6: Please pay attention to the content on the right side of the blackboard.

[0102] Message 7: Please pay attention to the content at the bottom left of the blackboard.

[0103] Message 8: Please pay attention to the content below the blackboard.

[0104] Message 9: Please pay attention to the content at the bottom right of the blackboard.

[0105] As can be seen from the above examples, each prompt in the set of prompts reminds students to pay attention to the content on the blackboard. The only difference is that different prompts contain different location information, thus reminding students to pay attention to the content at different locations on the blackboard.

[0106] Furthermore, a target prompt is selected from the set of prompts retrieved. The location information contained in the target prompt includes the screen partition location, or guidance information that includes the screen partition location.

[0107] That is, in the set of prompt information corresponding to the target action type, further query the target prompt information whose location information matches the screen partition position.

[0108] For example, if the screen partition is located at the top, then the target prompt information queried from the above prompt information set is prompt information 2.

[0109] In the method provided in this embodiment, it is necessary to pre-generate a set of prompt information containing various location information corresponding to each action type. Based on this, the target prompt information can be directly queried, making the processing logic simpler.

[0110] The second type

[0111] In the second implementation, in order to reduce the amount of stored data, this embodiment can store only the corresponding prompt information template for each action type. There can be only one prompt information template, which contains the position information slot to be filled.

[0112] Taking the above-mentioned knocking action type as an example, the corresponding prompt message template could be: Please pay attention to the content of the blackboard [].

[0113] The brackets "[]" represent location information slots.

[0114] Based on this, a prompt information template corresponding to the target action type can be obtained. Furthermore, the screen partition position or the guidance information of the screen partition position is used as position information and filled into the position information slot in the prompt information template to obtain the complete prompt information after filling.

[0115] For example, if the screen partition is located at the top, then this screen partition can be filled into the above prompt information template to obtain the complete prompt information: Please pay attention to the content above the blackboard.

[0116] The method provided in this embodiment can significantly reduce data storage requirements compared to the first implementation.

[0117] In some embodiments of this application, while the prompting information is output via voice broadcast in step S120, in order to further enhance the reminder effect on students, prompts can also be displayed simultaneously on the touchscreen using visual markers. Here, visual markers refer to processing methods that enhance the display effect through visual markings, such as using visual changes like text or color for prompts.

[0118] Specifically, when the target action type of the identified external action is an action type related to the touch location, the teacher's reminder intention is strongly correlated with the current touch location. It is necessary to remind the student to pay attention to the content on the screen at the current touch location. Therefore, in addition to outputting the prompt information via voice broadcast, the touch location coordinates can also be displayed on the touchscreen using visual markers to indicate the touchscreen location the student should focus on. For example, the text, patterns, etc., in the area near the touch location coordinates can be enlarged, or highlighted with a color, to visually and intuitively prompt the student to pay attention to the screen content in conjunction with the voice broadcast.

[0119] In some embodiments of this application, an acoustic wave sensor is used as an example of a vibration sensor to describe the process of determining the target action type of an external action based on the vibration signal of the air collected by the acoustic wave sensor.

[0120] Since different types of external actions produce different vibration signals after touching a touchscreen, the target action type of the external action can be determined based on the vibration signal. This embodiment introduces several optional implementation methods:

[0121] Firstly,

[0122] Considering that the magnitude of the vibration signal generated after different types of external actions touch the touch screen may be different, this embodiment can determine the target action type of the external action based on the magnitude of the vibration signal.

[0123] Taking slapping and tapping external actions as examples:

[0124] Considering that the vibration signal generated by a slapping action is generally greater than that of a knocking action, when the vibration magnitude of the vibration signal exceeds a first threshold, the external action can be identified as a slapping action. When the vibration magnitude of the vibration signal exceeds a second threshold but does not exceed the first threshold, the external action is identified as a knocking action. The second threshold is less than the first threshold. The second and first thresholds can be set according to the actual situation. By setting the second threshold as a lower limit, it is possible to avoid the teacher's normal blackboard writing or slight accidental touch being mistaken for a knocking action, thus improving the robustness of the solution.

[0125] In another alternative implementation, a neural network model can be used to predict the target action type of an external action based on the magnitude of the vibration signal. That is, sample vibration signals generated by various types of actions can be collected in advance to train the first action classification model. Once the trained first action classification model is obtained, the currently acquired vibration signal can be fed into the first action classification model to obtain the target action type of the external action output by the first action classification model.

[0126] Since the first action classification model has learned the mapping relationship between different vibration signals and action types on large-scale training data, it can accurately predict the target action type of external actions in the inference stage.

[0127] Secondly,

[0128] Considering that the contact area between the tapping action and the touchscreen is relatively larger, while the contact area between the striking action and the touchscreen is relatively smaller, when reflected in the vibration signal, the sound wave distribution of the vibration signal generated by the larger contact area is more concentrated, while the sound wave distribution of the vibration signal generated by the smaller contact area is more dispersed, that is, the sound wave distribution patterns are different.

[0129] Therefore, in this embodiment, the distribution pattern of the sound wave can be determined based on the vibration signal, and then the target action type of the external action can be determined based on the distribution pattern of the sound wave.

[0130] Taking slapping and tapping external actions as examples:

[0131] When the distribution pattern of the sound waves is determined to belong to the first mode, the external action can be determined to be a slapping type. When the distribution pattern of the sound waves belongs to the second mode, the external action can be determined to be a knocking type. The sound wave distribution in the second mode is more concentrated than that in the first mode.

[0132] In this embodiment, the contact area between different types of external actions and the touch screen is different, which leads to different sound wave distribution patterns of vibration signals. Analyzing the types of external actions from this perspective can improve the detection accuracy of external action types.

[0133] In another optional implementation, a neural network model can be used to predict the target action type of an external action based on the distribution pattern of sound waves. That is, sample sound wave distribution patterns corresponding to vibration signals generated by various types of actions can be collected in advance to train the second action classification model. Once the trained second action classification model is obtained, for the currently acquired vibration signal, its corresponding sound wave distribution pattern is determined and further fed into the second action classification model to obtain the target action type of the external action output by the second action classification model.

[0134] Since the second action classification model has learned the mapping relationship between the sound wave distribution patterns of different vibration signals and action types on large-scale training data, it can accurately predict the target action type of external actions in the inference stage.

[0135] It is understood that this embodiment can also combine the first and second methods of determining the target action type of an external action, and fuse the detection results of the two methods to determine the final detection result. Taking the determination of the target action type of an external action through the first and second action classification models as an example, the confidence scores of the same action type predicted by the first and second action classification models can be added together, and the action type with the highest confidence score after addition can be taken as the target action type of the external action.

[0136] By integrating two different implementation schemes, the accuracy of external action type prediction can be further improved.

[0137] The classroom large screen device interaction device provided in the embodiments of this application is described below. The classroom large screen device interaction device described below and the classroom large screen device interaction method described above can be referred to in correspondence.

[0138] See Figure 3 , Figure 3 This is a schematic diagram of the structure of a classroom large-screen interactive device disclosed in an embodiment of this application.

[0139] like Figure 3 As shown, the device may include:

[0140] Vibration signal acquisition unit 11 is used to acquire the vibration signal detected by the vibration sensor;

[0141] Action type determination unit 12 is used to determine the target action type of the external action based on the vibration signal;

[0142] The prompt information acquisition unit 13 is used to acquire prompt information corresponding to the target action type, wherein the prompt information is prompt content related to the teacher's teaching scenario;

[0143] The prompt information broadcasting unit 14 is used to output the prompt information in the form of voice broadcast.

[0144] Optionally, if the target action type of the external action is an action type unrelated to the touch position, then the process by which the above-mentioned prompt information acquisition unit acquires the prompt information corresponding to the target action type includes:

[0145] Obtain pre-configured prompt information corresponding to the target action type.

[0146] Optionally, if the target action type of the external action is an action type related to the touch position, then the process by which the above-mentioned prompt information acquisition unit acquires the prompt information corresponding to the target action type includes:

[0147] The touch position coordinates of the external action on the touch screen are detected, and the screen partition position to which the touch position coordinates belong is determined;

[0148] Obtain prompt information corresponding to the target action type and the screen partition position. The prompt information includes location information, which includes the screen partition position or guidance information that includes the screen partition position.

[0149] Optionally, the above-mentioned prompt information acquisition unit acquires the prompt information corresponding to the target action type and the screen partition position in two ways:

[0150] The first method involves querying a pre-configured set of prompts corresponding to the target action type, wherein each prompt in the set contains different location information.

[0151] Select the target prompt information from the set of prompt information found in the query. The location information contained in the target prompt information includes the screen partition location, or includes guidance information for the screen partition location.

[0152] The second method involves obtaining a pre-configured prompt information template corresponding to the target action type, wherein the prompt information template contains a position information slot to be filled.

[0153] The screen partition location or the guidance information of the screen partition location is used as location information and filled into the location information slot in the prompt information template to obtain the complete prompt information after filling.

[0154] Optionally, the apparatus of this application may further include:

[0155] The first visual prompting unit is used to display a visual marker at the coordinates of the touch position on the touch screen when the target action type of the external action is an action type related to the touch position, so as to prompt the student to pay attention to the touch screen position.

[0156] Optionally, the apparatus of this application may further include:

[0157] The second visual prompting unit is used to provide prompts on the touchscreen in the form of visual markings.

[0158] Optionally, if the vibration sensor is an acoustic sensor, then the process by which the action type determination unit determines the target action type of the external action based on the vibration signal includes:

[0159] Based on the magnitude of the vibration signal, determine the target action type of the external action;

[0160] or,

[0161] The distribution pattern of the sound wave is determined based on the vibration signal, and the target action type of the external action is determined based on the distribution pattern of the sound wave.

[0162] Optionally, the process by which the action type determination unit determines the target action type of the external action based on the magnitude of the vibration signal includes:

[0163] When the vibration magnitude of the vibration signal exceeds a first threshold, the external action is determined to be a slapping type; when the vibration magnitude of the vibration signal exceeds a second threshold but does not exceed the first threshold, the external action is determined to be a knocking type, wherein the second threshold is less than the first threshold.

[0164] The process by which the action type determination unit determines the target action type of the external action based on the distribution pattern of the sound waves includes:

[0165] When the sound wave distribution pattern belongs to the first mode, the external action is determined to be a slapping type. When the sound wave distribution pattern belongs to the second mode, the external action is determined to be a knocking type. The sound wave distribution in the second mode is more concentrated than that in the first mode.

[0166] Optionally, the target action type of the external action includes slapping action type and knocking action type; the prompt information corresponding to the slapping action type is used to restrain students' bad classroom habits; the prompt information corresponding to the knocking action type is used to remind students to pay attention to the content specified by the teacher on the touch screen.

[0167] Optionally, the process by which the vibration signal acquisition unit acquires the vibration signal monitored by the vibration sensor includes:

[0168] Start the SonarSensorService service to activate the vibration sensor and monitor the vibration signal generated by the touchscreen when it is touched by an external action.

[0169] The interactive device for classroom large-screen devices provided in this application embodiment can be applied to classroom large-screen devices. Optionally, Figure 4 The hardware structure block diagram of the classroom large screen device is shown below. Figure 4 The hardware structure of the classroom large screen device may include: touch screen 1, vibration sensor 2, processor 3, communication interface 4, memory 5 and communication bus 6.

[0170] In this embodiment, the number of touch screen 1, vibration sensor 2, processor 3, communication interface 4, memory 5 and communication bus 6 are all at least one, and touch screen 1, vibration sensor 2, processor 3, communication interface 4 and memory 5 communicate with each other through communication bus 6.

[0171] Touchscreen 1 is used to display information;

[0172] Vibration sensor 2 is used to monitor the vibration signal generated by touch screen 1 when it is touched by an external action, and transmit it to processor 3;

[0173] Processor 3 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0174] Memory 4 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0175] The memory stores a program, which the processor can call. The program is used for:

[0176] Acquire the vibration signal detected by the vibration sensor;

[0177] Based on the vibration signal, the target action type of the external action is determined;

[0178] Obtain prompt information corresponding to the target action type and output the prompt information in the form of voice broadcast, wherein the prompt information is prompt content related to the teacher's teaching scenario.

[0179] Optionally, the refined and extended functions of the program can be found in the description above.

[0180] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0181] Acquire the vibration signal detected by the vibration sensor;

[0182] Based on the vibration signal, the target action type of the external action is determined;

[0183] Obtain prompt information corresponding to the target action type, and output the prompt information in the form of voice broadcast, wherein the prompt information is prompt content related to the teacher's teaching scenario.

[0184] Optionally, the refined and extended functions of the program can be found in the description above.

[0185] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the various steps of the aforementioned classroom large-screen device interaction method.

[0186] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0187] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0188] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A classroom large-screen device interaction method, characterized in that, The classroom large-screen device includes a touch screen and a vibration sensor for monitoring the vibration signal generated by the touch screen when it is touched by an external action; the method includes: Acquire the vibration signal detected by the vibration sensor; Based on the vibration signal, the target action type of the external action is determined; Obtain prompt information corresponding to the target action type, and output the prompt information in the form of voice broadcast, wherein the prompt information is prompt content related to the teacher's teaching scenario; If the target action type of the external action is an action type related to the touch position, the process of obtaining prompt information corresponding to the target action type includes: detecting the touch position coordinates of the external action on the touch screen, and determining the screen partition position to which the touch position coordinates belong; obtaining prompt information corresponding to the target action type and the screen partition position, wherein the prompt information contains position information, and the position information includes the screen partition position, or includes guidance information for the screen partition position; Accordingly, in addition to outputting the prompt information in the form of voice broadcast, the method further includes: displaying the touch position coordinates on the touch screen in the form of visual markers, so as to visually and intuitively prompt the student to pay attention to the screen content at the touch screen position in conjunction with the voice broadcast.

2. The method according to claim 1, characterized in that, If the target action type of the external action is an action type unrelated to the touch position, the process of obtaining the prompt information corresponding to the target action type includes: Obtain pre-configured prompt information corresponding to the target action type.

3. The method according to claim 1, characterized in that, The process of obtaining prompt information corresponding to the target action type and the screen partition position includes: Query the pre-configured set of prompt information corresponding to the target action type, wherein each prompt information in the set contains different location information; Select the target prompt information from the set of prompt information found. The location information contained in the target prompt information includes the screen partition location, or includes guidance information for the screen partition location.

4. The method according to claim 1, characterized in that, The process of obtaining prompt information corresponding to the target action type and the screen partition position includes: Obtain a pre-configured prompt information template corresponding to the target action type, wherein the prompt information template contains position information slots to be filled; The screen partition location or the guidance information of the screen partition location is used as location information and filled into the location information slot in the prompt information template to obtain the complete prompt information after filling.

5. The method according to claim 1, characterized in that, The vibration sensor is an acoustic sensor. The process of determining the target action type of the external action based on the vibration signal includes: Based on the magnitude of the vibration signal, determine the target action type of the external action; or, The distribution pattern of the sound wave is determined based on the vibration signal, and the target action type of the external action is determined based on the distribution pattern of the sound wave.

6. The method according to claim 5, characterized in that, The process of determining the target action type of the external action based on the magnitude of the vibration signal includes: When the vibration magnitude of the vibration signal exceeds a first threshold, the external action is determined to be a slapping type; when the vibration magnitude of the vibration signal exceeds a second threshold but does not exceed the first threshold, the external action is determined to be a knocking type, wherein the second threshold is less than the first threshold. The process of determining the target action type of an external action based on the distribution pattern of the sound waves includes: When the sound wave distribution pattern belongs to the first mode, the external action is determined to be a slapping type. When the sound wave distribution pattern belongs to the second mode, the external action is determined to be a knocking type. The sound wave distribution in the second mode is more concentrated than that in the first mode.

7. The method according to claim 1, characterized in that, The target action types of the external actions include slapping action types and knocking action types; The prompts corresponding to the types of slapping actions are used to restrain students' bad classroom habits; The prompt information corresponding to the tapping action type is used to remind students to pay attention to the content specified by the teacher on the touch screen.

8. The method according to claim 1, characterized in that, The process of acquiring the vibration signal detected by the vibration sensor includes: Start the SonarSensorService service to activate the vibration sensor and monitor the vibration signal generated by the touchscreen when it is touched by an external action.

9. An interactive device for a classroom large-screen display, characterized in that, The classroom large-screen device includes a touch screen and a vibration sensor disposed on the back of the touch screen. The vibration sensor is used to monitor the vibration signal generated by the touch screen when it is touched by an external action; the device includes: A vibration signal acquisition unit is used to acquire the vibration signal detected by the vibration sensor; An action type determination unit is used to determine the target action type of the external action based on the vibration signal; The prompt information acquisition unit is used to acquire prompt information corresponding to the target action type, wherein the prompt information is prompt content related to the teacher's teaching scenario; A prompt message broadcasting unit is used to output the prompt message in the form of voice broadcast; If the target action type of the external action is an action type related to the touch position, the process of the prompt information acquisition unit acquiring the prompt information corresponding to the target action type includes: detecting the touch position coordinates of the external action on the touch screen and determining the screen partition position to which the touch position coordinates belong; acquiring the prompt information corresponding to the target action type and the screen partition position, wherein the prompt information contains position information, and the position information includes the screen partition position or guidance information including the screen partition position; Correspondingly, it also includes: a first visual prompting unit, used to display the touch position coordinates on the touch screen in the form of a visual mark when the target action type of the external action is an action type related to the touch position, so as to visually and intuitively prompt the student to pay attention to the screen content of the touch screen position in conjunction with the voice broadcast.

10. A classroom large-screen device, characterized in that, include: Touchscreen, vibration sensor, memory, and processor; The vibration sensor is used to monitor the vibration signal generated by the touch screen when it is touched by an external action, and transmit it to the processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the classroom large screen device interaction method as described in any one of claims 1 to 8.

11. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the classroom large screen device interaction method as described in any one of claims 1 to 8.

Images