Intention understanding method and system for intelligent pen

Abstract

The invention discloses an intention understanding method and system for a smart pen, and mainly relates to the technical field of multi-modal fusion. Comprising the steps that a multi-modal information and perception model is established, and the model comprises a visual channel, a sensor channel and a voice channel; constructing a plane preliminary selection algorithm according to the established multi-modal information and perception model; constructing a multi-modal information fusion algorithm according to the constructed plane preliminary selection algorithm; and constructing an active collaborative equivalence model according to a multi-modal information fusion algorithm. The solid geometry teaching aid has the beneficial effects that the solid geometry teaching aid can provide support for solid geometry classroom teaching of teachers, so that the solid geometry teaching aid can clearly and conveniently display the composition of geometries and the spatial relationship of geometric elements.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal fusion, and in particular to an intention understanding method and system for a smart pen. Background Art

[0002] In middle school mathematics instruction, the transition from plane geometry to solid geometry presents a significant challenge for many students. In plane geometry, students primarily engage with and understand two-dimensional figures, which is relatively straightforward for most students. Solid geometry, on the other hand, requires students to visualize and manipulate three-dimensional objects and master the construction of geometric elements such as three-view drawings and auxiliary planes. This represents a significant leap for many students. Because of its inherent complexity and cognitive demands, solid geometry also requires students to possess advanced spatial reasoning and abstract thinking. However, a significant number of students have limited spatial imagination and struggle to understand the necessary spatial relationships and perspective shifts, making it difficult to master these three-dimensional structures. This transition can be overwhelming for students.

[0003] Due to students' learning difficulties, in order to help students understand the concept of solid geometry more clearly, teachers need to spend more time explaining geometric concepts and repeatedly drawing geometric figures from multiple angles. Even so, traditional solid geometry teaching still has problems such as difficulty in coordinating explanations and demonstrations, and difficulty in drawing. While increasing the workload of teachers, it ultimately leads to a decline in teaching efficiency.

[0004] Therefore, there is an urgent need for an intention understanding method and system for a smart pen to solve the above problems. Summary of the Invention

[0005] The purpose of the present invention is to provide an intention understanding method for a smart pen, which can provide support for teachers' solid geometry classroom teaching, enabling them to clearly and conveniently demonstrate the composition of geometric bodies and the spatial relationship of geometric elements.

[0006] To achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions:

[0007] In one aspect, a method for understanding intentions of a smart pen is provided, comprising the following steps:

[0008] S1: Establishing a multimodal information and perception model, wherein the model includes a visual channel, a sensor channel, and a speech channel;

[0009] S2: Constructing a preliminary plane selection algorithm based on the multimodal information and perception model established in step S1;

[0010] S3: constructing a multimodal information fusion algorithm based on the plane preliminary selection algorithm constructed in step S2;

[0011] S4: Based on the multimodal information fusion algorithm in step S4, an active collaborative equivalence model is constructed.

[0012] Preferably, in step S1, establishing a visual channel includes:

[0013] A YOLOv8-based smart pen real-time tracking module is constructed. Based on the projection distance from the pen tip coordinate point to the plane, the probability of the plane being selected on the spatial level is calculated. The calculation process is as follows:

[0014] V vi =F vi (P,Dis[distance1,distance2,…,distance i ])(1)

[0015] Among them, F vi represents the feature extraction operation under the visual channel, P represents the coordinate of the smart pen tip at this time, and the vector Dis records the projection distance between the pen tip coordinate P and each plane in the virtual space at this time, V vi is the sub-intention set under the visual channel;

[0016] According to the coordinates of the smart pen tip P obtained in the real-time tracking module, the coordinates of the projection point P′ of P on the i-th plane in the virtual environment are calculated:

[0017]

[0018] Among them, P i is a point on the plane, N is the normal vector of the plane, and · represents the dot product operation;

[0019] Calculate P and P i The Euclidean distance between:

[0020] distance i =‖PP i ‖(3)

[0021] Among them, distance i That is, the distance between the pen tip and the i-th plane in the virtual space at this moment;

[0022] The standard sigmoid function is used to calculate the probability of the plane being selected to obtain the main semantic information under the visual channel:

[0023]

[0024] Among them, P i Select represents the probability of the i-th plane being selected, α and β are offsets;

[0025] The secondary semantics of the visual channel is set as the user's action information, and the primary and secondary semantics are fused to obtain the sub-intention vector V under the visual channel. vi .

[0026] Preferably, in step S1, establishing a sensor channel includes:

[0027] The gesture sensor is used to obtain motion data from the smart pen, extracting the acceleration change sequence. This change sequence is then fed into the LSTM network for analysis to determine whether the characteristics of the acceleration sequence reflect the user's intention to select a plane, i.e., the action information in the intention. Combined with the secondary intention of the sensor channel, a sub-intention set under the sensor channel is generated. The calculation process is as follows:

[0028] V s =F s (LSTM(Acc[acceleration1,acceleration2,...,acceleration j ]))(5)

[0029] Among them, F s Represents the feature extraction operation under the sensor channel, and the vector Acc records t d The sequence of acceleration changes of the smart pen in time, V s It is the sub-intent set under the sensor channel.

[0030] Preferably, in the LSTM network, a binary cross entropy loss function is used:

[0031]

[0032] Among them, N is the number of samples, y j is the actual type of the sample, is the predicted value of the model;

[0033] The prediction result of the LSTM network output layer is:

[0034] P k =σ(W out ·h t +b out ),k∈{"Select","Not Select"} (7)

[0035] Among them, P k is the probability that the predicted acceleration change sequence is classified as k, where k represents the type of sequence and reflects whether the user has the intention to select a plane. W out is the output layer weight, b out is the output layer bias, σ is the sigmoid activation function;

[0036] The obtained classification results are used as the main intention in the sensor channel sub-intention set, and the existing plane information in the current virtual environment is combined as the secondary intention to construct the sensor channel sub-intention set V s .

[0037] Preferably, in step S1, establishing a voice channel includes:

[0038] Convert the user's voice command into text information, and use Jieba word segmentation to operate on the text information, divide the actions and operation objects in the user's voice command, and construct the sub-intent set V of the voice channel a , the calculation process is:

[0039] V a =F a (∏(jieba(ToText(Info Acoustic ))))(8)

[0040] Among them, F a Represents the feature extraction operation in the speech channel, Info Acoustic Represents the voice information input by the user. The ToText operation converts voice into text. The jieba operation represents the word segmentation operation. The ∏ operation represents the concatenation of the word segmentation results in the format of "action + operation object" to obtain the complete intent information.

[0041] The obtained complete intent information is compared with the content in the speech intent database to calculate the text similarity. The probability value of this speech recognition is determined and stored in the position corresponding to the subscript of the vector. The probability values of the remaining positions are set to 0.01.

[0042] Preferably, specifically: a plane selection algorithm based on spatiotemporal information, including:

[0043] S21: When the sensor channel detects the user holding the smart pen, the system time t is recorded. n With the current coordinates P of the smart pen tip, formula (1) is called to use the information reflecting the geometric position relationship in the virtual environment, such as the spatial projection distance, to construct the sub-intention set V of the visual channel. vi , if in the next t d If the user intention detected by the sensor channel within the time is the same as the action information detected by the sensor channel that the user holds the smart pen, then the sub-intention set V under the sensor channel is constructed. vi ;

[0044] S22: In the data structure of the sub-intention set that constitutes the visual channel and the sensor channel, it is set as an m×1-dimensional vector, that is, it consists of m probability values, where m is the number of planes in the current virtual experimental environment. Each subscript position of the sub-intention vector represents a plane, and the probability value represents the probability of the plane corresponding to the subscript being selected. The sub-intention vectors of the two channels are fused using the method of calculating the Hadamard product. The calculation process is:

[0045]

[0046] Where, ⊙ represents the Hadamard product operation, I s,vi The predicted intent is obtained by fusing the sub-intention sets of the two channels of vision and sensor;

[0047] S23: Obtain the user's judgment information on the correctness of the target plane selected by the system, confirm the selected plane result, and perform the next operation or end the interaction based on the user's further feedback information.

[0048] Preferably, the multimodal information fusion algorithm is specifically: a multimodal fusion intention understanding algorithm based on an improved GFN, and the calculation process is:

[0049] I Pred =GFN(V vi ,V s ,V a )(10)

[0050] Among them, I Pred is the predicted intention obtained through multimodal fusion, V vi ,V s ,V a The sub-intent sets extracted from the visual channel, sensor channel and voice channel respectively;

[0051] The following steps are involved:

[0052] S31: Get the voice command input by the user and calculate its sub-intention set V according to formula (8) a , and the sub-intention vector V of the visual channel and sensor channel vi and V s Perform Hadamard product operations on both of them and calculate the similarity to obtain the fusion result vector of the two sub-intention vectors with high similarity. Assigning a higher influence coefficient, the calculation process is:

[0053]

[0054] in, Indicates the influence coefficient that needs to be added during the fusion process of the sub-intention vectors corresponding to channels C1 and C2;

[0055] The dual-channel fusion result D is obtained in the second layer of GFN vi.s ,D vi.a ,D s.a ;

[0056] In the third layer, the two-channel fusion result vectors are subjected to Hadamard product operations, and similarity calculation and influence coefficient assignment are performed in the same way as in the second layer to obtain three three-channel fusion result vectors T vi.s.vi.a ,T vi.a.s.a ,T vi.s.s.a Each dual-channel fusion result vector is also subjected to Hadamard product operation, similarity calculation and weighting operation in sequence with the single-channel feature vector that does not participate in the vector fusion in the second layer, and the other three three-channel information fusion results T are obtained. vi.s.a ,T vi.a.s ,T s.a.vi , a total of six result vectors of three-channel information fusion with different influence coefficients are obtained. The logarithmic sum of the six vectors is calculated to obtain the final feature vector:

[0057]

[0058] Among them, T i,j The value of the element with subscript j in the i-th three-channel fusion vector;

[0059] Pass Res through a Softmax classifier to get the final prediction result with the highest probability I Pred ;

[0060] S32: Convert the sub-intention vector into a planar scatter plot, and convert each key-value pair consisting of a subscript and a probability value into a coordinate point. That is, the problem is converted into calculating the similarity of two broken lines. The Hausdorff distance is used to reflect the distance between the point sets constituting the two broken lines. The method for calculating the similarity of the sub-intention vector is:

[0061]

[0062] in, Represents the Hausdorff distance between the sub-intention vectors corresponding to channels C1 and C2, Then it is the similarity score between channels C1 and C2;

[0063] Convert the similarity score into the influence coefficient of the fused vertex

[0064]

[0065] Preferably, the active collaborative equivalent model is specifically an active assisting face selection auxiliary equivalent model, including:

[0066] S41: For the selected plane Q given by the system Pred , by finding the time and plane Q Pred The plane Q with the maximum average depth value maxD and Q maxD Build an equivalent model for the frontal view and calculate the average depth distance:

[0067]

[0068] Among them, Q a ,Q b are two point sets that constitute the corresponding planes, m, n are the number of points that constitute the corresponding planes, d ij is the absolute value of the depth interpolation between two points in the point set, z a ,z b is the z-axis coordinate of the point;

[0069] S42: Active collaboration through generation of equivalent models.

[0070] On the other hand, a system for the method for understanding intention of a smart pen according to claim 1 is provided, comprising a smart pen body, a sensor, a signal transmission module and a remote processor.

[0071] The sensor is provided on the smart pen body and is used to: collect visual information and voice information;

[0072] The signal transmission module is used to transmit the collected data information to the remote processor;

[0073] The processor is used to: establish a multimodal information and perception model, construct a plane preliminary selection algorithm, construct a multimodal information fusion algorithm, and construct an active collaborative equivalence model.

[0074] Compared with the prior art, the beneficial effects of the present invention are:

[0075] 1. In response to the problem of accurately selecting a plane of a geometric body in the process of solid geometry teaching, the multimodal fusion intention understanding algorithm proposed in this invention is mainly to detect the intention of selecting a plane by extracting multimodal behavioral feature information in accordance with the teacher's behavioral habits, and to reduce the selection difficulty caused by inadequate depth information perception through a human-computer collaborative algorithm.

[0076] 2. In the intention understanding part, the visual channel and sensor channel are divided into primary semantics and secondary semantics. The reason for this is mainly to hope that each channel focuses on the information it is better at obtaining and processing, and each channel extracts independent behavioral features, thereby achieving complementarity between multimodal information and improving the accuracy of predicted intentions; in addition, for the voice channel, since the amount of voice information input in the actual teaching environment is too large, in order to avoid useless information affecting intention understanding or meaningless interaction, the voice information is used as the input information determined by the teacher in the fusion process. Because voice information can relatively clearly and intuitively reflect the user's intention, this design ensures that the intention implied by the voice information still has a high degree of credibility in the fusion stage, and can serve as a good reference for the system to correct intentions; in addition, voice information can also play a role in division and refinement when selecting sub-planes that constitute a certain plane.

[0077] 3. In the human-computer collaboration part, the present invention mainly studies the problem of inaccurate plane selection caused by teachers' difficulty in perceiving depth information in actual processes. It effectively utilizes the ability of the smart pen system to effectively perceive depth information, and generates equivalent models through active collaboration to provide users with a geometric state with an opposite perspective, so that teachers can more easily select the target plane without changing the current geometric state. While eliminating the time consumed by teachers to manually adjust the geometric angle, the cognitive load of teachers is reduced. At the same time, the use of complementary equivalent models can also more intuitively display the spatial relationship of geometric elements in the teaching process, thereby improving the effect of geometry teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 is a flow chart of the method of the present invention;

[0079] Figure 2 It is a distribution diagram of the motion trajectory of the smart pen and the area of interest of the present invention;

[0080] Figure 3 (a), (b), (c), and (d) are four representative trajectory division diagrams in the present invention;

[0081] Figure 4 It is a schematic diagram of the GFN layered fusion structure of the present invention;

[0082] Figure 5 It is a schematic diagram of the algorithm combination operation process of the present invention;

[0083] Figure 6 (a) and (b) are schematic diagrams of the equivalent complementary model of the present invention;

[0084] Figure 7 This is an auxiliary schematic diagram of the equivalent model collaborative surface selection of the present invention;

[0085] Figure 8 is a schematic structural diagram of the smart pen of the present invention;

[0086] Figure 9 It is a functional schematic diagram of the smart pen of the present invention. DETAILED DESCRIPTION

[0087] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0088] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention, and should not be understood as limiting the present invention.

[0089] In the present invention, terms such as "fixed connection," "connected," and "connection" should be interpreted broadly to mean a fixed connection, an integral connection, or a detachable connection; a direct connection or an indirect connection through an intermediary. Relevant researchers or technicians in this field may determine the specific meanings of these terms in the present invention based on specific circumstances, and they should not be construed as limitations of the present invention.

[0090] Example:

[0091] To address interdisciplinary human-computer interaction, we should explore possible ways to effectively transmit data and information between humans and computers. This focus is on developing input and output devices that support vision, touch, and language to enable more efficient computer use. Furthermore, considering the behavioral characteristics of teachers in actual teaching, we understand that they are accustomed to interacting with the system through modalities such as voice, vision, and tactile pressure. Therefore, this example selects vision, sensors, and voice as sources of external interaction information, focusing on two key issues: information perception in each modality and the integration of the three modalities.

[0092] like Figure 1 As shown, this embodiment provides a method for understanding intentions of a smart pen, comprising the following steps:

[0093] S1: Establishing a multimodal information and perception model, wherein the model includes a visual channel, a sensor channel, and a speech channel;

[0094] S2: Constructing a preliminary plane selection algorithm based on the multimodal information and perception model established in step S1;

[0095] S3: constructing a multimodal information fusion algorithm based on the plane preliminary selection algorithm constructed in step S2;

[0096] S4: Based on the multimodal information fusion algorithm in step S4, an active collaborative equivalence model is constructed.

[0097] The main task of the visual channel is to detect which plane has the highest probability of being selected in the current spatial relationship. Under the visual channel, a real-time tracking module of the smart pen based on YOLOv8 is first constructed, and the probability of a plane being selected in the spatial layer is calculated based on the projection distance from the pen tip coordinate point to the plane. In the task of smart pen tracking, this embodiment uses a depth camera to acquire real-time video information and passes it to the YOLOv8 target detection network to track the pen tip. The motion information of the smart pen in the real space is mapped to the virtual space in real time through the above-mentioned smart pen hardware system to enable the user to operate the geometric objects in the virtual space. Next, its calculation process is shown in formula (1):

[0098] V vi =F vi (P,Dis[distance1,distance2,…,distance i ]) (1)

[0099] Among them F vi represents the feature extraction operation under the visual channel, P represents the coordinate of the smart pen tip at this time, and the vector Dis records the projection distance between the pen tip coordinate P and each plane in the virtual space at this time, V vi is the sub-intention set under the visual channel. Next, this embodiment further describes the calculation process of the probability of each plane being selected in the space. In this embodiment, first, according to the coordinates of the smart pen tip P obtained in the real-time tracking module, the coordinates of the projection point P′ of P on the i-th plane in the virtual environment are calculated using formula (2):

[0100]

[0101] Among them, P i is a point on the plane, N is the normal vector of the plane, and · represents the dot product operation. Next, use formula (3) to calculate P and P i The Euclidean distance between:

[0102] distance i =‖PP i ‖(3)

[0103] The distance obtained i That is, the distance between the pen tip and the i-th plane in the virtual space at this time. This is used to obtain the ascending sort sequence of the distances between the pen tip and all planes at this time. Then, the probability of the plane being selected is calculated using the standard sigmoid function according to formula (4):

[0104]

[0105] Among them, P i Select represents the probability of the i-th plane being selected, α and β are offsets, and this is used to obtain the main semantic information under the visual channel, that is, the analysis and processing of the user's intention to select a plane at the spatial level, focusing on reflecting which plane in the space has the highest probability of being selected by the user, that is, determining the operated object or the object in the operation instruction; at the same time, the secondary semantics of the visual channel are set as the user's action information, that is, the predicate in the operation instruction. Since this article is mainly aimed at the selection operation of the plane, the predicate in the visual sub-intention set is set to "Select". The main and secondary semantics are fused to obtain the sub-intention vector V under the visual channel vi .

[0106] The main task of the sensor channel is to capture the user's motion data within a certain period of time, analyze the characteristics of the motion data, and infer whether the user has the intention to select a certain plane. In the sensor channel, this embodiment first uses the smart pen motion data information obtained by the posture sensor (MPU6050) to extract the acceleration change sequence from it, and sends it to the LSTM network for analysis. Finally, it is determined whether the characteristics of the acceleration sequence reflect the user's intention to select a plane, that is, the action information in the intention (that is, the predicate verb, Select or Not Select in this article), and combines the channel's secondary intention (operation object, object) to generate a sub-intention set under the sensor channel, as shown in the following formula (5):

[0107] V s =F s (LSTM(Acc[acceleration1,acceleration2,...,acceleration j ])) (5)

[0108] Among them, F s Represents the feature extraction operation under the sensor channel, and the vector Acc records t d The sequence of acceleration changes of the smart pen in time, V sIt is a sub-intention set under the sensor channel. Since individuals tend to perform operations within their focus area, and visual attention can reflect the intention of the operation, similar to the task of visual attention requiring distinction between the gaze area and the area where the eyes sweep, in the pen-based interaction process in this embodiment, it is also necessary to distinguish between operations that do not contain intentions, such as "passing by" or "sweeping" an area with the smart pen, and operations that may contain intention information, such as "staying" or "hovering" in an area. By analyzing the behavioral characteristics of teachers in the actual teaching process, this embodiment combines the performance characteristics of visual attention, extracts the acceleration change characteristics when the user holds the smart pen and moves, and analyzes the characteristics of the motion data to reflect whether the user is interested in the plane corresponding to the area, such as Figure 2 As shown, it shows the visualization result of the user's interest in a plane, which in the GeoPen system described in this article indicates whether the user intends to select the plane or perform other operations.

[0109] In the LSTM network, this embodiment summarizes the problem of identifying the motion trajectory of the smart pen into a two-class classification problem, namely, d Whether the sequence of changes in the motion acceleration of the smart pen within a certain time period reflects the user's intention to select a plane. In this embodiment, during the sampling process, the following Figure 3 The four types of motion trajectory information shown are representative, among which the stationary trajectory (such as Figure 3 (a)) and circling trajectory (as shown in Figure 3 (b)) as the behavioral characteristics of the intention to "choose", and Figure 3 (c) with Figure 3 The two motion trajectories shown in (d) do not contain the behavioral characteristics of the "selection" intention. In addition to the obvious difference in path length that can be visually observed, the change information of the acceleration on the three coordinate axes implied in the trajectory is the input information of the LSTM network in the sensor channel, which serves as the basis for binary classification reasoning and subsequent modal fusion.

[0110] For this binary classification problem, this embodiment uses a binary cross entropy loss function in the network, as shown in formula (6):

[0111]

[0112] Among them, N is the number of samples, y j is the actual type of the sample, is the predicted value of the model. The prediction result of the LSTM network output layer is shown in the following formula (7):

[0113] P k =σ(W out ·h t +b out),k∈{"Select","Not Select"} (7)

[0114] Among them, P k is the probability that the predicted acceleration change sequence is classified as k, where k represents the type of sequence and reflects whether the user has the intention to select a plane. W out is the output layer weight, b out is the output layer bias, and σ is the sigmoid activation function.

[0115] After classifying the acceleration change sequence of the smart pen within the time period, the obtained classification result is used as the main intention in the sub-intention set of the sensor channel. Combined with the existing plane information in the current virtual environment (i.e., the operation object in the sub-intention set) as the secondary intention (object), the sub-intention set V of the sensor channel is constructed. s .

[0116] The main work of the voice channel is different from the above two channels. It is not used to construct the semantic structure of the intention, but as a clear input information, it is used for intention error correction and intention refinement. In the voice channel, since the semantic information contained in the voice information is relatively complete, this embodiment no longer divides the main and secondary semantic information, but directly extracts a complete sub-intention set containing action information and operation objects (ie, predicate and object), and hopes to use the voice information as a sub-intention set with clear information to participate in the fusion process in the fusion stage. First, in this embodiment, the existing speech recognition technology is used to convert the user's voice instructions into text information, and the jieba segmentation is used to operate on the text information, dividing the action (predicate) and operation object (object) in the user's voice instructions, and constructing the sub-intention set V of the voice channel. a , the calculation process is shown in formula (8):

[0117] V a =F a (∏(jieba(ToText(Info Acoustic ))))(8)

[0118] Among them, F a Represents the feature extraction operation in the speech channel, Info AcousticRepresents the voice information input by the user, the ToText operation is voice-to-text conversion, jieba represents the word segmentation operation, and the ∏ operation means splicing the word segmentation results in the format of "predicate + object" to obtain complete intent information. Next, the text similarity between this information and the content in the voice intent database is calculated (the content in the intent database is determined by the plane information existing in the virtual environment at this time), and the probability value of this speech recognition should be stored in the position corresponding to the subscript of the vector. The probability values of the remaining positions are set to 0.01. The purpose of this is to maintain the sub-intent vector V of the voice channel. a The formal consistency with the sub-intention vectors of the other two channels is helpful for subsequent multimodal fusion operations. At the same time, in this embodiment, the voice channel is expected to be an information channel that more intuitively reflects the user's intention. When the fusion results of the visual channel and the sensor channel are inaccurate, it can provide semantic information that is more credible and more clearly reflects the user's intention for the system's multimodal information fusion operation. At the same time, it allows users to more naturally correct the results of the predicted intention when the system's intention understanding is inaccurate. For example, if the user's voice information is: "In this embodiment, the next step is to select plane ABCD", and the credibility of this voice recognition is 0.98, then the sub-intention set V composed of the operation information and the plane to be operated extracted by the voice channel is a ={0.98,0.01……,0.01} (assuming that the subscript corresponding to the semantics “select plane ABCD” in the sub-intention vector is 0).

[0119] Furthermore, the spatiotemporal information acquired by the visual and trajectory channels can only preliminarily identify a single plane in space, but cannot accurately partition it, meaning it cannot select sub-planes consisting of only a few vertices. Considering the actual geometry and the need for explanation, this embodiment leverages the intuitive nature of the voice channel. By extracting information about the vertices that make up a sub-plane, the voice channel can be used to divide and slice planes composed of multiple vertices. This allows for the selection of sub-planes consisting of any three points within a complex plane, thus completing the division of a complex plane.

[0120] 1. Plane selection algorithm based on spatiotemporal information:

[0121] This algorithm is mainly used to make a preliminary selection of a certain face in the target geometric body, and provide a basis for the subsequent fusion algorithm to carry out multimodal information fusion based on the user's feedback results. In the teaching scenario, this embodiment hopes to improve the teacher's teaching efficiency, and the plane selection process will not take too long, and ensure the naturalness of the operation and conform to the teacher's routine. Therefore, this embodiment hopes to combine multimodal input information, use spatial information to reflect the information of the user's operation object (object), and use the speed change sequence within a certain time to reflect the user's action information (predicate), and use the information of each channel complementarily, that is, integrate spatiotemporal information to predict the target plane selected by the user and understand his intention;

[0122] First, when the sensor channel detects the motion characteristics of the user holding the smart pen reflecting the "selection" intention, the system time t at this time is recorded. n With the current coordinates of the smart pen tip P, we then call formula (1) in the visual channel and use the information reflecting the geometric position relationship in the virtual environment, such as the spatial projection distance, to construct the sub-intention set V of the visual channel. vi , if in the next t d If the user intention detected by the sensor channel is still “select” within the time period, then the sub-intention set V under the sensor channel is constructed. vi Due to the behavioral characteristics of teachers, a secondary detection mechanism for the smart pen motion trajectory is adopted in this embodiment. By further confirming the user's selection intention, the extracted user intention is ensured to be more accurate, and the target plane that the user wants to select is judged more accurately;

[0123] In the data structure of the sub-intention set that constitutes the visual channel and the sensor channel, in this embodiment, it is set as an m×1-dimensional vector, that is, it consists of m probability values, where m is the number of planes in the current virtual experimental environment. Each subscript position of the sub-intention vector represents a plane, and the probability value represents the probability that the plane corresponding to the subscript is selected. Since the two channels are divided into primary and secondary semantics in this embodiment, the consistency of the semantic information represented by the sub-intention sets of the two channels can be ensured. Therefore, this embodiment considers finding the subscript that obtains the maximum probability value in both vectors on this basis to reflect the sub-intention with the highest probability;

[0124] After obtaining the sub-intention sets of the two channels, this embodiment adopts the method of calculating the Hadamard product to fuse the sub-intention vectors of the two channels to preliminarily extract the user's intention in this interaction. The calculation of sub-intention fusion is shown in the following formula (9):

[0125]

[0126] Where ⊙ represents the Hadamard product operation, I s,vi The predicted intent is obtained by fusing the sub-intention sets of the two channels of vision and sensor;

[0127] Finally, the system obtains the user's judgment on the correctness of the target plane selected by the system, confirms the selected plane result, and performs the next operation or ends the interaction based on the user's further feedback information.

[0128] 2. Multimodal fusion intent understanding algorithm based on improved GFN

[0129] Because PSABSI only performs a preliminary selection of planes and ensures simple and time-saving operation, inaccurate selection may occur in some cases. Therefore, when the information from the visual channel and sensor channel alone cannot accurately understand the user's intention, the system uses the user's voice command as a new information input channel and combines the information from the visual and sensor channels through multimodal fusion to further predict the user's intention.

[0130] To address the problem of intent understanding in PSABSI, this embodiment uses a construction method similar to that of the graph fusion network (GFN) to perform hierarchical fusion of the input information of the three modalities, further compare the information of each channel, and then perform information fusion by logarithmic summation. Finally, a Softmax classifier is used to obtain the probabilistic predicted intent. The hierarchical fusion structure is as follows: Figure 4 As shown, the calculation process of this algorithm is shown in formula (10):

[0131] I Pred =GFN(V vi ,V s ,V a )(10)

[0132] Among them I Pred is the predicted intention obtained through multimodal fusion, V vi ,V s ,V a The sub-intent sets extracted by the above three channels respectively;

[0133] First, when PSABSI does not understand the user's intention accurately, it obtains the user's input voice command and calculates its sub-intention set V according to the formula (8) mentioned above. a , and the sub-intention vector V of the visual channel and sensor channel vi and V s Perform Hadamard product operations on both of them and calculate the similarity to obtain the fusion result vector of the two sub-intention vectors with high similarity. Assign a higher influence coefficient, as shown in formula (11):

[0134]

[0135] in, Indicates the influence coefficient that needs to be added during the fusion process of the sub-intention vectors corresponding to channels C1 and C2, thereby obtaining the dual-channel fusion result D in the second layer of GFN vi.s ,D vi.a ,D s.a In the calculation at this time, the voice channel plays the role of emphasizing the user's intention; in the third layer, in this embodiment, the two-channel fusion result vectors are subjected to Hadamard product operations, and similarity calculations and influence coefficient assignments are performed in the same way as in the second layer to obtain three three-channel fusion result vectors T vi.s.vi.a ,T vi.a.s.a ,T vi.s.s.a At the same time, each dual-channel fusion result vector is also subjected to Hadamard product operation, similarity calculation and weighting operation in sequence with the single-channel feature vector that does not participate in the vector fusion in the second layer, and the other three three-channel information fusion results T are obtained. vi.s.a ,T vi.a.s ,T s.a.vi , thus a total of 6 result vectors of three-channel information fusion with different influence coefficients are obtained. Next, the logarithmic sum of these 6 vectors is calculated, as shown in formula (12):

[0136]

[0137] Where T i,j The element value of the subscript j of the i-th three-channel fusion vector is obtained, and the final feature vector Res is obtained. The log function can also play a role in further filtering low-probability sub-intentions and improving the difference between different sub-intentions and probabilities. Finally, Res is passed through a Softmax classifier to obtain the final prediction result with the highest probability I Pred ;

[0138] In this embodiment, the distribution of the sub-intention vector with a higher probability intention is reflected by calculating the similarity, that is, whether the peak values in the two vectors appear at the same subscript. Therefore, the sub-intention vector is converted into a plane scatter line graph, and the key-value pair consisting of each subscript and probability value is converted into a coordinate point, that is, the problem is converted into calculating the similarity of the two broken lines. In this embodiment, the distance between the point sets constituting the two broken lines can be reflected by the Hausdorff distance. The method for calculating the similarity of the sub-intention vector is shown in the following formula (13):

[0139]

[0140] in, Represents the Hausdorff distance between the sub-intention vectors corresponding to channels C1 and C2, Then it is the similarity score between channels C1 and C2, and the similarity score is converted into the influence coefficient of the fusion vertex by the following formula (14):

[0141]

[0142] The purpose is to ensure that the sub-intention vectors with relatively concentrated probability position distribution can maintain a larger influence coefficient and pass the high-credibility information implied by the factor value to the lower layer, ensuring that clear intentions with a higher probability can be retained and play a role in the final fusion stage.

[0143] According to the above algorithm description, the operation process of combining PSABSI with improved GFN is simulated in this embodiment, such as Figure 5 As shown in the figure, two main application scenarios of introducing interaction are respectively shown, namely error intent correction and selection intent refinement. The reason for doing so is to clarify the purpose of the voice channel and enable it to play a role in necessary tasks;

[0144] First, the current system gives the plane V6V7V8V9V 10 The selection result of , and the plane required in the experiment is its sub-plane V6V7V 13 V9. At this time, since the visual channel and the sensor channel cannot further divide the selected plane, the voice channel input command is used here to further clarify the intention, and the GFN network is used to complete the fusion of the three channel sub-intention vectors to obtain the refined intention and complete the division and selection of the sub-plane. The second case is when the plane predicted by PSABSI is inaccurate. It can be seen that the surface V8V3V4V9 of the geometric body in the figure is selected instead of the auxiliary plane V9V 13 V 14 V4 was correctly selected, and the two are relatively close in space. Using only PSABSI often leads to such selection difficulties. For this reason, users can correct errors more intuitively through the voice channel, while the other two channels can only be corrected by re-executing PSABSI, and the results may still be wrong. Therefore, after supplementing the reasoning information required by the system through the voice channel and passing through GFN, the wrong prediction results were corrected, and the target auxiliary plane V9V required for the experiment was correctly selected. 13 V 14 V4.

[0145] 3. Actively assist in selecting an equivalent model

[0146] This algorithm mainly assists users in plane selection operations by establishing an equivalent complementary model when users have difficulty selecting a plane, that is, performing active collaboration. In a virtual experimental environment, users can accurately perceive two-dimensional information by moving the tip of the smart pen on the screen, but their perception of depth information is relatively insufficient. In plane selection operations, many selection difficulties arise because users find it difficult to accurately perceive changes in depth information. In addition, in actual teaching, the spatial relationship of certain geometric elements may only be more intuitively observed from certain specific perspectives. If the teacher rotates the geometric body before performing a plane selection operation, it will take more time. Therefore, in this embodiment, it is hoped that the GeoPen system can give full play to its ability to accurately perceive depth information and actively collaborate with the user to avoid the problem of users having difficulty perceiving depth information.

[0147] When the user thinks that the system's predicted intention Pred When it is inaccurate, the system will correct the system's prediction results in an active collaborative manner, that is, the system actively provides assistance in selecting surfaces to complete the teaching task. This embodiment tests the user's intention by calculating average depth information. Considering that the user may rotate the geometric body at any angle in actual operation, there will always be some planes that are not easy to observe from the current perspective. Therefore, this embodiment dynamically obtains and calculates depth information based on the actual situation of the geometric body, and generates an equivalent model with the plane as the main view for the plane that is difficult to select under the current situation. This allows the user to select the target plane by operating the equivalent model while maintaining the current perspective. Similarly, the plane selection situation on the original geometric body model will also be reflected in the equivalent model. At the same time, the equivalent model of the geometric body from another perspective also forms a visual complementary relationship with the original geometric body model, which is conducive to the teacher's explanation and the students' mastery and understanding of the relationship between spatial geometric elements.

[0148] For the system given the selected plane Q Pred , by finding the time and plane Q Pred The plane Q with the maximum average depth value maxD and Q maxD Build an equivalent model for the frontal view, e.g. Figure 6 As shown, the two present a visually complementary relationship, in which Figure 6 (b) Figure 6 The equivalent model of (a), the surface selection results in the equivalent model will also be reflected on the original geometry;

[0149] The formula for calculating the average depth distance is shown in the following formula (15):

[0150]

[0151] where Q a ,Q b are two point sets that constitute the corresponding planes, m, n are the number of points that constitute the corresponding planes, d ij is the absolute value of the depth interpolation between two points in the point set, z a ,z b is the z-axis coordinate of the point, by finding the minimum The plane of values is used to generate equivalent models;

[0152] This embodiment describes the face selection auxiliary equivalent model in combination with actual conditions. At this time, the initial viewing angle of the geometric body is as follows: Figure 7 As shown, for the current viewing angle, the target plane V7V to be selected in the experiment is 13 V 14 V2 is selected incorrectly due to occlusion interference and difficulty in determining depth distance, such as Figure 7 As shown in part (b) of the figure; at this time, after the error is detected, the system will actively analyze the plane depth information under the current perspective to generate an equivalent model of the current perspective, providing users with active face selection assistance, such as Figure 7 As shown in part (c), Figure 7 (c)(i) and Figure 7 (c)(ii) shows the two geometric bodies in the current and reverse perspectives of the equivalent model. It can be seen that in the reverse perspective equivalent model, the target plane V7V 13 V 14 V2 is in a position that is easier to select, and there is less occlusion interference from other geometric elements such as planes around this position. By reselecting in the equivalent model of the reverse perspective, the correct plane is successfully selected and reflected on the geometry of the initial perspective at the same time, such as Figure 7 As shown in part (d), active collaboration through the generation of equivalent models allows teachers to accurately select difficult-to-select planes without changing their current perspective, saving time and avoiding the increased interaction load caused by difficulties in perceiving depth information. Results from subsequent experiments also demonstrate that equivalent models are not simply transformations of geometric objects from the current perspective, but rather dynamic models that combine depth perception and user interaction.

[0153] On the other hand, this embodiment provides a system for the above-mentioned method for understanding intentions of a smart pen, including a smart pen body, a sensor, a signal transmission module, and a remote processor.

[0154] The sensor is provided on the smart pen body and is used to: collect visual information and voice information;

[0155] The signal transmission module is used to transmit the collected data information to the remote processor;

[0156] The processor is used to: establish a multimodal information and perception model, construct a plane preliminary selection algorithm, construct a multimodal information fusion algorithm, and construct an active collaborative equivalence model.

[0157] Among them, the hardware equipment of the smart pen is the basis for teachers to interact with the virtual geometry teaching experiment platform. The smart pen we designed consists of a 3D printed pen body, three touch sensors (including one TTP233 and two resistive film pressure sensors FSR402, respectively denoted as TS1, TS2 and TS3) and a posture sensor (MPU-6050). TS1 is used as a function button, that is, it is used to perform operations on geometric bodies in space, including executing the face selection algorithm in this article (long press), and performing transformation operations such as scaling (single click) and rotating (double click) on geometric bodies; TS2 and TS3 are used as "OK" and "Cancel" function buttons respectively, which are used to confirm\negate the results of the user's own operations on the geometric body or the system's understanding of the user's intentions. The schematic diagram of the smart pen structure and the physical prototype are shown in the figure below. Figure 8 shown.

[0158] With the above hardware as the basis for interaction, the sensor buttons on the smart pen body correspond to the functions in the virtual experiment platform, such as Figure 9 As shown,

[0159] By clicking TS1 and moving the smart pen upward or downward, the virtual geometric body can be enlarged or reduced. Figure 9 (a) with Figure 9 (b) As shown; short press TS1, the virtual geometric body will rotate accordingly with the rotation angle of the smart pen, as shown in Figure 9 (c) As shown; after confirming the scaling or rotation angle of the geometric body, click TS2 to confirm the current change result; long press TS1 to draw and write. In order to maintain the consistency between the teacher's virtual experimental platform and the common blackboard teaching in reality, we set up the writing and drawing function. The trajectory of the teacher's holding the smart pen will be visualized in the virtual geometry experimental platform, as shown in Figure 9 (d) As shown; Finally, click TS3 to restore the virtual geometry to the size and angle before the change, or clear the existing handwriting, as shown in Figure 9 (e) shown.

[0160] Furthermore, to integrate virtual and real life, a RealSense SR300 depth camera and the YOLOv8 object detection network are used to track the smart pen's movements in real-world 3D space. The camera's capture range is sufficient to support the teacher's movements at the podium, allowing the trajectory of the smart pen held by the teacher to be instantly and accurately reflected in the virtual geometry teaching experiment platform.

[0161] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art can make various equivalent modifications or substitutions to the transaction features between nodes without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for understanding intentions of a smart pen, characterized in that: The following steps are involved: S1: Establishing a multimodal information and perception model, wherein the model includes a visual channel, a sensor channel, and a speech channel; S2: Constructing a preliminary plane selection algorithm based on the multimodal information and perception model established in step S1; S3: constructing a multimodal information fusion algorithm based on the plane preliminary selection algorithm constructed in step S2; S4: Based on the multimodal information fusion algorithm in step S4, an active collaborative equivalence model is constructed.

2. The method for understanding intention of a smart pen according to claim 1, characterized in that: In step S1, establishing a visual channel includes: A YOLOv8-based smart pen real-time tracking module is constructed. Based on the projection distance from the pen tip coordinate point to the plane, the probability of the plane being selected on the spatial level is calculated. The calculation process is as follows: V vi =F vi (P,Dis[distance1,distance2,……,distance i ])(1) Among them, F vi represents the feature extraction operation under the visual channel, P represents the coordinate of the smart pen tip at this time, and the vector Dis records the projection distance between the pen tip coordinate P and each plane in the virtual space at this time, V vi is the sub-intention set under the visual channel; According to the coordinates of the smart pen tip P obtained in the real-time tracking module, the coordinates of the projection point P′ of P on the i-th plane in the virtual environment are calculated: Among them, P i is a point on the plane, N is the normal vector of the plane, and · represents the dot product operation; Calculate P and P i The Euclidean distance between: distance i =‖P-P i ‖(3) Among them, distance i That is, the distance between the pen tip and the i-th plane in the virtual space at this moment; The standard sigmoid function is used to calculate the probability of the plane being selected to obtain the main semantic information under the visual channel: Among them, P i Select represents the probability of the i-th plane being selected, α and β are offsets; The secondary semantics of the visual channel is set as the user's action information, and the primary and secondary semantics are fused to obtain the sub-intention vector V under the visual channel. vi .

3. The method for understanding intention of a smart pen according to claim 1, characterized in that: In step S1, establishing a sensor channel includes: The gesture sensor is used to obtain motion data from the smart pen, extracting the acceleration change sequence. This change sequence is then fed into the LSTM network for analysis to determine whether the characteristics of the acceleration sequence reflect the user's intention to select a plane, i.e., the action information in the intention. Combined with the secondary intention of the sensor channel, a sub-intention set under the sensor channel is generated. The calculation process is as follows: V s =F s (LSTM(Acc[acceleration1,acceleration2,……,acceleration j ]))(5) Among them, F s Represents the feature extraction operation under the sensor channel, and the vector Acc records t d The sequence of acceleration changes of the smart pen in time, V s It is the sub-intent set under the sensor channel.

4. The method for understanding intention of a smart pen according to claim 3, characterized in that: In the LSTM network, the binary cross entropy loss function is used: Among them, N is the number of samples, y j is the actual type of the sample, is the predicted value of the model; The prediction result of the LSTM network output layer is: P k =σ(W out ·h t +b out ),k∈{"Select","Not Select"}(7) Among them, P k is the probability that the predicted acceleration change sequence is classified as k, where k represents the type of sequence and reflects whether the user has the intention to select a plane. W out is the output layer weight, b out is the output layer bias, σ is the sigmoid activation function; The obtained classification results are used as the main intention in the sensor channel sub-intention set, and the existing plane information in the current virtual environment is combined as the secondary intention to construct the sensor channel sub-intention set V s .

5. The method for understanding intention of a smart pen according to claim 1, characterized in that: In step S1, establishing a voice channel includes: Convert the user's voice command into text information, and use Jieba word segmentation to operate on the text information, divide the actions and operation objects in the user's voice command, and construct the sub-intent set V of the voice channel a , the calculation process is: V a =F a (∏(jieba(ToText(Info Acoustic ))))(8) Among them, F a Represents the feature extraction operation in the speech channel, Info Acoustic Represents the voice information input by the user. The ToText operation converts voice into text. The jieba operation represents the word segmentation operation. The ∏ operation concatenates the word segmentation results in the format of "action + operation object" to obtain complete intent information. The obtained complete intent information is compared with the content in the speech intent database to calculate the text similarity. The probability value of this speech recognition is determined and stored in the position corresponding to the subscript of the vector. The probability values of the remaining positions are set to 0.

01.

6. The method for understanding intention of a smart pen according to claim 1, characterized in that: The preliminary plane selection algorithm is specifically a plane selection algorithm based on spatiotemporal information, including: S21: When the sensor channel detects the user holding the smart pen, the system time t is recorded. n With the current coordinates P of the smart pen tip, formula (1) is called to use the spatial projection distance and other information reflecting the geometric position relationship in the virtual environment to construct the sub-intention set V of the visual channel. vi , if in the next t d If the user intention detected by the sensor channel within the time period is the same as the action information detected by the sensor channel that the user holds the smart pen, then the sub-intention set V under the sensor channel is constructed. vi ; S22: In the data structure of the sub-intention set that constitutes the visual channel and the sensor channel, it is set as an m×1-dimensional vector, that is, it consists of m probability values, where m is the number of planes in the current virtual experimental environment. Each subscript position of the sub-intention vector represents a plane, and the probability value represents the probability of the plane corresponding to the subscript being selected. The sub-intention vectors of the two channels are fused using the method of calculating the Hadamard product. The calculation process is: Where, ⊙ represents the Hadamard product operation, I s,vi The predicted intent is obtained by fusing the sub-intention sets of the two channels of vision and sensor; S23: Obtain the user's judgment information on the correctness of the target plane selected by the system, confirm the selected plane result, and perform the next operation or end the interaction based on the user's further feedback information.

7. The method for understanding intention of a smart pen according to claim 1, characterized in that: The multimodal information fusion algorithm is specifically: a multimodal fusion intention understanding algorithm based on an improved GFN, and the calculation process is: I Pred =GFN(V vi ,V s ,V a )(10) Among them, I Pred is the predicted intention obtained through multimodal fusion, V vi ,V s ,V a The sub-intent sets extracted from the visual channel, sensor channel and voice channel respectively; The following steps are involved: S31: Get the voice command input by the user and calculate its sub-intention set V according to formula (8) a , and the sub-intention vector V of the visual channel and sensor channel vi and V s Perform Hadamard product operations on both of them and calculate the similarity to obtain the fusion result vector of the two sub-intention vectors with high similarity. Assigning a higher influence coefficient, the calculation process is: in, Indicates the influence coefficient that needs to be added during the fusion process of the sub-intention vectors corresponding to channels C1 and C2; The dual-channel fusion result D is obtained in the second layer of GFN vi.s ,D vi.a ,D s.a ; In the third layer, the two-channel fusion result vectors are subjected to Hadamard product operations, and similarity calculation and influence coefficient assignment are performed in the same way as in the second layer to obtain three three-channel fusion result vectors T vi.s.vi.a ,T vi.a.s.a ,T vi.s.s.a Each dual-channel fusion result vector is also subjected to Hadamard product operation, similarity calculation and weighting operation in sequence with the single-channel feature vector that does not participate in the vector fusion in the second layer, and the other three three-channel information fusion results T are obtained. vi.s.a ,T vi.a.s ,T s.a.vi , a total of six result vectors of three-channel information fusion with different influence coefficients are obtained. The logarithmic sum of the six vectors is calculated to obtain the final feature vector: Among them, T i,j The value of the element with subscript j in the i-th three-channel fusion vector; Pass Res through a Softmax classifier to get the final prediction result with the highest probability I Pred ; S32: Convert the sub-intention vector into a planar scatter plot, and convert each key-value pair consisting of a subscript and a probability value into a coordinate point. That is, the problem is converted into calculating the similarity of two broken lines. The Hausdorff distance is used to reflect the distance between the point sets constituting the two broken lines. The method for calculating the similarity of the sub-intention vector is: in, Represents the Hausdorff distance between the sub-intention vectors corresponding to channels C1 and C2, Then it is the similarity score between channels C1 and C2; Convert the similarity score into the influence coefficient of the fused vertex 8. The method for understanding intention of a smart pen according to claim 1, characterized in that: The active collaborative equivalent model is specifically an active assistance face selection auxiliary equivalent model, including: S41: For the selected plane Q given by the system Pred , by finding the time and plane Q Pred The plane Q with the maximum average depth value maxD and Q maxD Build an equivalent model for the frontal view and calculate the average depth distance: Among them, Q a ,Q b are two point sets that constitute the corresponding planes, m, n are the number of points that constitute the corresponding planes, d ij is the absolute value of the depth interpolation between two points in the point set, z a ,z b is the z-axis coordinate of the point; S42: Active collaboration through generation of equivalent models.

9. A system for the method for understanding intention of a smart pen according to claim 1, comprising a smart pen body, a sensor, a signal transmission module, and a remote processor. The sensor is provided on the smart pen body and is used to: collect visual information and voice information; The signal transmission module is used to transmit the collected data information to the remote processor; The processor is used to: establish a multimodal information and perception model, construct a plane preliminary selection algorithm, construct a multimodal information fusion algorithm, and construct an active collaborative equivalence model.

Images