Man-machine cooperation method and system for multi-dimensional intelligent interaction pen

Abstract

The invention discloses a man-machine cooperation method and system for a multi-dimensional intelligent interaction pen, and mainly relates to the technical field of man-machine interaction cooperation. Comprising the following steps: synchronously collecting user interaction information through a visual channel, a voice channel and a sensor channel; calculating the matching degree of the tangent plane and the to-be-selected plane by using the bidirectional Hausdorff distance and the normal vector included angle; selecting the intention with the maximum probability of each channel as a candidate intention, and if the intentions of the channels are consistent, calculating a channel expected value by using information entropy and performing linear weighted fusion; if the intentions conflict, prompting the user to input again; and generating a corresponding operation prompt or navigation type selection guide. The method has the beneficial effects that the problems of time waste and low efficiency caused by repeated selection are effectively reduced, the reliability of the system is improved, and a better method and technical support are provided for classroom geometry teaching.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction and collaboration technology, specifically a human-computer collaboration method and system for a multi-dimensional intelligent interactive pen. Background Technology

[0002] In current mathematics teaching, especially solid geometry, conventional blackboard teaching often fails to intuitively demonstrate the spatial characteristics of geometric figures, and the repeated use of the blackboard for drawing is time-consuming. Moreover, for students with weak spatial imagination, the learning effect of this method may be relatively low. To address this issue, the industry has applied computer technology to the field of education, such as the use of teaching aids and virtual reality technology, to improve teachers' teaching effectiveness and enable students to observe geometric figures more intuitively.

[0003] Among current educational software, GeoGebra is a widely used mathematics teaching software. It uses conventional input devices such as a mouse, keyboard, or touchpad for interaction. While it allows for comprehensive observation of geometric shapes, its interaction mode is relatively simple and lacks high intelligence and intent understanding capabilities, presenting a certain learning curve. With the continuous development and evolution of human-computer interaction, the advancements in virtual reality (VR) and augmented reality (AR) have brought new approaches to geometry teaching, transforming traditional two-dimensional drawing methods into three-dimensional graphics. Geo+, designed by Rossano et al., uses AR technology to teach three-dimensional sets, enhancing students' engagement and immersion in the learning process.

[0004] However, the Geo+ system uses fingers to interact with virtual graphics, which cannot precisely select and change the constituent elements of geometric shapes. Yet, such precise selection is indispensable in actual teaching, and considering current teaching habits, chalk still plays a crucial role in the teaching process. Therefore, this embodiment considers pen-based interaction as the basic interaction method for the virtual-real integrated teaching platform as a feasible solution. The challenge this presents is that during the use of the smart pen, it is often necessary to perform operations on the points, lines, and surfaces that constitute geometric shapes. Therefore, correctly selecting the corresponding constituent elements of the geometric shape is challenging, especially the operation of "surfaces" within the geometric shape, which is more complex, often leading to incorrect or inaccurate selections, particularly when dealing with complex geometric shapes. Furthermore, commonly used interaction methods in conventional virtual experiments (such as physics and chemistry experiments) such as gestures and voice cannot meet the precise selection requirements of mathematical experiments. This raises another issue: designing an interaction method or device suitable for the precise selection needs in geometric experiments.

[0005] Therefore, there is an urgent need for a multi-dimensional intelligent interactive pen human-computer collaboration method and system to solve the above problems. SUMMARY

[0006] The present application aims to provide a multi-dimensional intelligent interactive pen human-computer collaboration method and system, which effectively reduces the time waste and low efficiency caused by repeated selection, improves the reliability of the system, and provides better methods and technical support for classroom geometry teaching.

[0007] To achieve the above-mentioned purpose, the present application realizes the following technical solutions: On the one hand, a multi-dimensional intelligent interactive pen human-computer collaboration method is provided, comprising the following steps: S1: synchronously collecting user interaction information through a visual channel, a voice channel and a sensor channel; S2: the user performs a cutting operation on a virtual geometric body through the intelligent pen to construct a virtual cutting surface, the system calculates the cutting point coordinates of the cutting surface and the edges of the geometric body and generates a cutting plane point set, and the matching degree of the cutting surface and the selected plane is calculated using the bidirectional Hausdorff distance and the normal vector angle; S3: selecting the intention with the maximum probability of each channel as the candidate intention, if the channel intention Figure One conflicts, prompting the user to re-enter; S4: according to the probability of selecting the plane and the probability interval threshold, combining the number of planes in the interval where the probability is the highest The interaction clarity is evaluated, and the corresponding operation prompt or navigation selection guide is generated.

[0008] Preferably, step S1 is specifically: The visual channel obtains the motion information of the intelligent pen tip in the real space through the depth camera and the YOLOv5 target detection network and maps it to the virtual space, and at the same time, the tip is expanded from a point model to a triangular surface model; the voice channel collects user voice instructions through a microphone and converts them into text information; the sensor channel collects three-dimensional motion parameters and operation instructions of the intelligent pen through a posture sensor and a touch sensor; The processing of the visual channel further includes: capturing the position of the intelligent pen tip in real time through the Realsense depth camera and projecting it to the three-dimensional virtual space constructed by Unity to obtain the coordinates of a point P in the triangular surface model; obtaining the direction of the normal vector N of the plane through the posture sensor, and calculating the plane equation combining the point P coordinates and the normal vector N; The processing of the voice channel includes: converting the voice instruction into Chinese text, performing word segmentation and removing mood words, generating a sentence vector by using a word2vec Chinese word vector model, calculating the cosine similarity of the sentence vector and the sentence vector in the voice intent library, and taking the result with a similarity greater than 80% as a voice sub-intent set; The hardware of the sensor channel includes: a posture sensor MOU-6050 for acquiring acceleration, angular velocity and angle information of the intelligent pen; a touch sensor TS1 for executing a surface selection algorithm by long pressing, executing geometric body scaling by single clicking, and executing geometric body rotation by double clicking; and touch sensors TS2 and TS3 for determining and canceling a function button, respectively.

[0009] Preferably, in step S2, the matching degree of the tangent plane and the candidate plane is calculated by using a bidirectional Hausdorff distance and a normal vector angle, and specifically is: ; wherein, is the Hausdorff distance of the tangent plane and the i-th candidate plane, is the angle between the normal vector of the tangent plane and the normal vector of the i-th candidate plane, and is the probability of the i-th plane being selected. ,

[0010] Preferably, step S2 further includes: before calculating the Hausdorff distance, according to the normal vector direction of the plane of the intelligent pen when the user presses the touch sensor, filtering out the planes corresponding to the normal vectors with an angle greater than 45°, and only performing Hausdorff distance calculation and probability sorting on the remaining planes.

[0011] Preferably, in step S3, the initial weight calculation process of the channel expected value by using information entropy and linear weighting fusion is: ; ; wherein, is the initial weight of the visual channel, is the initial weight of the voice channel, and when the user feeds back an understanding error, the weight is updated as follows: ; ; wherein, is the error number of the i-th plane, is a power index for controlling the weight reduction speed, is the updated visual channel weight, ​​​​​The updated speech channel weight.

[0012] Preferably, in step S3, the formula for calculating the information entropy is: ; wherein, is the expected value of the single-channel intention, is the intention extracted from a certain channel, is a certain intention in the set of set intentions; and the information entropy is normalized as: ; wherein, is the expected mean value, is the standard deviation.

[0013] Preferably, in step S4, the interval threshold is specifically: , , , and the specific rules for evaluating the interaction clarity are: if and , the evaluation is easy, and the user is automatically selected to the second highest plane after negation; if and , or , the evaluation is possible, and the user is prompted to input more detailed information or re-input; if , the evaluation is difficult, and the user is prompted to re-input and request confirmation.

[0014] On the other hand, a human-computer collaborative system for a multi-dimensional only interactive pen is provided, based on a human-computer collaborative method for a multi-dimensional only interactive pen as described above, comprising: An intelligent pen hardware module: composed of a 3D printed pen body, a posture sensor MOU-6050, touch sensors TS1, TS2 and TS3, for collecting physical parameters of user operations; A multi-modal information acquisition module: including a vision unit, a speech unit and a sensor unit, for synchronously acquiring vision, speech and sensor information; A data processing module: including a face selection sub-module, an intention fusion sub-module and a dynamic weight updating sub-module; An interaction feedback module: according to the output result of the data processing module, combined with the evaluation rules, generating operation prompts or navigation guides in the form of vision and speech.

[0015] Preferably, the smart pen hardware module also includes a Bluetooth serial communication unit, which is used to transmit the data collected by the attitude sensor and touch sensor to the computer in real time, so as to realize the mapping of the real motion trajectory to the virtual space; the vision unit in the multimodal information acquisition module uses virtual-real fusion technology to map the real-time motion tracking results of the smart pen in the real three-dimensional space to the virtual space.

[0016] Preferably, the data processing module also includes a historical interaction information database, which stores error information reported by users and corresponding planar selection records, providing statistical basis for the number of errors for the dynamic weight update submodule; when the user expresses the intention to reselect or abandon the interaction through voice command, the historical interaction information database does not record the interaction data for this interaction.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Historical interaction information is introduced into the system for error correction. When the initial feedback given by the system is inaccurate, the information evaluation mechanism and dynamic weight update algorithm mentioned in the paper can be used to refine the input information and combine it with the analysis and utilization of error information in the historical interaction records. This can correct and rectify misunderstandings, effectively reduce the time wastage and inefficiency caused by repeated selections, improve the reliability of the system, and provide better methods and technical support for classroom geometry teaching.

[0018] 2. Introducing the intelligent pen system into the classroom not only provides a more intuitive and flexible teaching method compared to traditional blackboard teaching, saving time spent on repeated blackboard teaching and drawing, but also provides teachers with a more convenient and innovative teaching tool. For students, it also enables them to observe geometric figures from all angles, which helps to improve their understanding of knowledge and spatial perception.

[0019] 3. It enables natural interaction between teachers and the intelligent pen system, achieves hand-eye coordination, reduces cognitive load, improves teaching effectiveness, and provides a new direction for the application of human-computer collaboration in traditional education. This allows pen-based interaction to be researched and applied in more fields of teaching, contributing to the development of education. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method of the present invention; Figure 2 These are simulation diagrams of the algorithm operation of the present invention, where (a), (b), (c), and (d) are flowcharts of the algorithm operation; Figure 3 This is a structural diagram of the smart pen of the present invention; Figure 4 This is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0021] The application will be further described below with reference to specific examples. It should be understood that these examples are only used to illustrate the application and not intended to limit the scope of the application. Furthermore, it should be understood that after reading the content of the present application, those skilled in the art can make various modifications or changes to the application, and these equivalent forms also fall within the scope defined by the present application.

[0022] In the present application, the terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is only a relationship word determined for the convenience of describing the structural relationship of the components or elements of the present application, and cannot be understood as a limitation on the present application.

[0023] Embodiment: As shown in the figure, the present embodiment provides a human-machine collaborative method for multi-dimensional interactive pen, comprising the following steps: Figure 1 S1: synchronously collecting user interaction information through visual channel, voice channel and sensor channel; S2: the user performs cutting operation on the virtual geometric body through the intelligent pen to construct a virtual cutting surface, the system calculates the cutting point coordinates of the cutting surface and each edge of the geometric body and generates a cutting plane point set, and the matching degree of the cutting surface and the selected plane is calculated by using bidirectional Hausdorff distance and normal vector angle; S3: selecting the intention with the maximum probability of each channel as the candidate intention, if the channel intention conflicts, calculating the channel expectation value by using information entropy and fusing by linear weighting; if the intention conflicts, prompting the user to re-input; Figure One S4: according to the probability of selecting the plane and the probability interval threshold, combining the number of planes in the interval where the probability is the highest to evaluate the interaction clarity, and generating corresponding operation prompt or navigation type selection guide.

[0024] Among them, in the process of system running, the acquisition and analysis processing of multi-channel information in the present embodiment are respectively carried out by the following ways: ​Firstly, the visual channel, in the visual aspect, is divided into two parts in this embodiment: smart pen tracking and probability calculation based on Hausdorff distance. In the task of smart pen tracking, this embodiment uses a depth camera to obtain real-time video information, which is handed over to the YOLOv5 target detection network for tracking the smart pen nib, and through the above-mentioned smart pen hardware system, the motion information of the smart pen in the real space is mapped to the virtual space in real time, so as to realize the operation of the user on the geometric body in the virtual space. The second part is the Hausdorff distance calculation in the virtual space. In this embodiment, the smart pen nib is expanded from a point model to a surface model, the selection operation is quantized as the Hausdorff distance calculation of two point sets in space, and the probability of selecting a certain plane is calculated based on this, and related weight information is provided for subsequent multi-modal fusion calculation. The operation characteristics of the user in the visual channel and the implicit semantics are shown in Table 1. Table 1 Implicit semantics of visual channel

[0025] Secondly, the information of the speech channel is input by the user through the microphone. For the speech instruction input by the user, first, the existing speech processing method is used to convert the user's speech information into Chinese text, then the Chinese text is split into words and the adverbs are removed, then the word2vec Chinese word vector model is used to convert it into a word vector, and finally a sentence vector is generated to represent the speech information. The generated sentence vector and the sentence vectors of the speech intent set in the speech intent library are calculated for cosine similarity, and the results with similarity greater than 80% are counted as the speech sub-intent set of this interaction. The operation characteristics of the user in the speech channel and the implicit semantics are shown in Table 2. Table 2 Implicit semantics of speech channel

[0026] Finally, the sensor channel is composed of a posture sensor MOU-6050 (denoted as AS) and three touch sensors (including one TTP233 and two resistive thin film pressure sensors FSR402 (denoted as TS1, TS2 and TS3 respectively). The sensor system works together and communicates with the computer in real time to realize the acquisition and processing of sensor information. Each sensor and its corresponding implicit speech information are shown in Table 3:

[0027] The user uses the smart pen to construct a virtual section surface by "cutting" the geometric body, and calculates the matching degree between the section surface and all the selected planes (here, the selected planes refer to all the planes constituting the geometric body and the auxiliary planes possibly existing in the geometric body) by using the bidirectional Hausdorff distance and the included angle of the plane normal vector.

[0028] In this embodiment, the pen tip is no longer regarded as a point for selection and other operations, but as a three-dimensional triangular plane in the three-dimensional space. For the convenience of observation, the system only sets the plane as a triangular plane of limited size, and calculates the plane equation at this time by using the coordinates of a point P in the plane and the plane normal vector. The coordinates of the point P are obtained by the vision channel of the system, the position of the pen tip in the real space is captured in real time by the Realsense depth camera and the YOLOv5 target detection network, and the coordinates of the point P are obtained by projecting the position of the pen tip in the real space to the three-dimensional virtual space constructed by Unity. In addition, the direction of the plane normal vector N is obtained by the attitude sensor in the smart pen hardware device, so as to calculate the plane equation at this time. After the user completes the selection of the position of the pen tip plane, the geometric body and the reverse "cutting" operation can be performed, that is, the calculation of the cut point is performed in combination with the plane equation at this time and the straight line equation constituting the geometric body. Since each cut point is obtained by "cutting" different straight lines by the same plane, all the cut points are in the same plane.

[0029] After the construction of the cut plane is completed, the matching degree between the cut plane and all the selected planes needs to be calculated. The driving function for calculating the plane matching degree will have two parts, one part is to use the Hausdorff distance, and the other part is to use the included angle of the normal vector.

[0030] After the cutting and the calculation of the cut point coordinates are completed, the normal vector direction of the smart pen plane is filtered out when the user presses the touch sensor, the normal vectors corresponding to the planes with an included angle greater than 45° are filtered out, and then the following Hausdorff distance calculation is performed. The coordinate point set of the cut plane is generated by using m (m≥3) cut points constituting the cut plane, the Hausdorff distance between the plane point set of the cut point generated by the user using "cutting" and the point set of all the existing planes is calculated, and the calculated distance is sorted in ascending order. The plane corresponding to the point set with the closest distance is regarded as the plane that the user wants to select with the highest probability in this time in the vision channel and the sensor channel. The formula for calculating the probability that the plane is selected by the distance is shown in equation (1): (1) wherein represents the Hausdorff distance between the plane drawn by the user and the i-th existing plane, is the normal vector N of the cut surface and the normal vector of the selected plane the angle between the i-th plane and the plane of the smart pen and , represents the probability that the i-th plane is selected.

[0031] Firstly, the following scenario is established in this embodiment, as shown in Figure 2 (a), which contains a quadrangular pyramid as the target geometry to be selected, and the right side is the simulated plane of the smart pen tip, which will be used to cut the target geometry in this embodiment; next, when the user clicks the touch sensor at the position selected by himself, the system will automatically calculate the plane equation of the smart pen plane at this time and calculate the coordinates of the intersection points of each edge of the target geometry (the number of intersection points is denoted as m), as shown in Figure 2 (b); since the m intersection points are cut by the same plane, the m points are coplanar, and the virtual cutting surface is shown in Figure 2 (c); finally, the bidirectional Hausdorff distance formula is used to calculate the distance between the cutting surface and all the preliminary candidate planes (the angle between the normal vector and the plane is less than 45°), and then the probability of each preliminary candidate plane being selected is calculated using formula (1), and the plane with the highest probability is marked, as shown in Figure 2 (d).

[0032] During the operation of the algorithm MFHII, if the system correctly understands the user's intention for the first time, the selection result is confirmed after the user indicates confirmation, and the interaction is completed; at this point, the use of a simple interaction and confirmation process can also improve the overall running efficiency of the algorithm; if the result of the first execution of the algorithm does not meet the user's intention, after the user indicates negation, the system will record the error information of this time, and the historical error information will be used in the IDBHINI algorithm described later, that is, the dynamic weight updating algorithm proposed in this embodiment, which further tries to obtain the user's intention and guides the user to modify and refine the input information; if the user's own intention changes, the user can also input the voice to reselect the instruction, if the user shows the intention information of reselecting or abandoning this interaction, the system will not record the error information of this time.

[0033] During the execution of the MFHII algorithm, if the tentative understanding of the user's intention by the system is incorrect, the system should analyze the last understanding result, i.e., evaluate the current interaction by using the context information of the current interaction, find the problem in the interaction process according to the effective degree of the user input information, and guide the user to finely process the input information, so as to improve the accuracy of the subsequent second interaction. The purpose of designing the algorithm is to provide timely feedback and dynamic optimization in the process of understanding the user's intention through interactive learning, so as to improve the success rate of the algorithm without causing the situation of low efficiency and rising interaction load due to too long interaction time. The IDBHINI algorithm proposed in the embodiment is divided into two parts: a dynamic weight updating algorithm based on context information and an information clarity evaluation mechanism based on human-computer interaction.

[0034] 1. Dynamic weight updating: Firstly, in each channel, the probability of the selected intention is the largest intention extracted by the channel. In the case of the largest probability intention extracted by different channels, Figure One the expected value of the corresponding channel is calculated by using information entropy. If the intentions extracted by two channels are inconsistent and conflict, the system gives up this fusion and prompts the user to re-input. The formula for calculating the expected value by using information entropy is as follows: (2) Where H(x) is the expected value of the single-channel intention, x is the intention extracted by a certain channel, and p(x) is the probability of the intention. The intention with the highest expected value is taken as the intention extracted by the channel. In the application of information entropy in the literature [9], the smaller the information entropy value, the higher the reasoning confidence.

[0035] For the intention probability information obtained by using information entropy, the embodiment first normalizes it by using formula (3) to ensure that the mean value is 0 and the standard deviation is 1, so as to ensure that the normalized information entropy value is closer to the range of 0 to 1, which is easier to explain and understand.

[0036] (3) Where μ is the mean value of the calculated expected value H(x), and σ is the standard deviation. Secondly, the normalized intention expected value is calculated by using the following formula : (4) Where w1 and w2 represent the weight of the visual channel (motion trajectory) and the weight of the speech channel, respectively. ​​with the expected value of the corresponding intention calculated by the visual channel and the speech channel.

[0037] In formula (4), the initial weights of the two channels are calculated by the following two formulas, respectively. According to the Hausdorff distance calculated in algorithm 1 and the normal vector cosine value obtained, the initial weights are further set in linear weighted combination by using the following formula: (5) wherein represents the Hausdorff distance between the to-be-determined plane and the virtual tangent plane, represents the angle between the normal vectors of the to-be-determined plane and the virtual tangent plane. At the same time, since the weights of the speech channel and the visual channel are dynamically complementary, the weight of the speech channel is set to in the embodiment, which is calculated by the following formula: (6) After calculating the weight values by the above formulas, in order to improve the accuracy of the algorithm, the embodiment uses human-computer interaction to explore the user's intention: if the selected plane initially fed back to the user by the system is not accurate, the user can directly express the intention of "selecting the next one", and the system will select the second nearest point set corresponding to the plane according to the distance obtained by sorting and selecting; if the to-be-selected plane given by the system this time is incorrect, the user needs to select again, then the system will record the error result this time, reduce the weight of the error plane in the next decision, and in the case that the user's intention does not change, the second time of the section construction will be more biased to the correct plane in the to-be-selected plane, and the tangent plane constructed in this way will also have a larger probability calculated according to formula (1) than before . In the embodiment, the number of errors is used as the independent variable, and the weight is dynamically and nonlinearly updated according to the number of errors, as shown in formula (7): (7) wherein represents the number of errors, and the power index y is used to control the descending speed of the weight, so as to prevent the second selection error from increasing due to the weight descending too fast; in addition, if the user wants to change the plane to be selected the second time, the number of errors can also be reset through the voice instruction. At the same time, the updated weight of the speech channel is calculated according to the complementary way of the double-channel weight, as shown in formula (8): (8) 2. Human-computer interaction information clarity evaluation mechanism: If the system's initial understanding of the user's intention is inaccurate, the user can directly indicate the error by pressing the touch sensor TS3 of the smart pen or by voice, and the system will analyze the error information and assist the user in gradually refining and accurately inputting the information, thereby improving the algorithm accuracy and interaction efficiency. Upon receiving the user's indication of the error, the system will evaluate the reliability of the selected point using the formula (1) mentioned above, set the parameter 、 、 to represent the probability interval threshold (where , , ), and to represent the number of planes in the probability interval with the highest probability of being selected; if (M represents the plane with the highest probability of being selected) and 0 < 3, it indicates that the distance between the user's constructed section and the target plane is relatively accurate, and the system is rated as easy, so the system automatically selects the plane with the second highest probability after the user expresses the denial. If > 3, it indicates that although the user's constructed section is relatively appropriate, the system cannot successfully obtain the intention information due to factors such as the structure of the geometric body or the viewing angle, so the system rates the current interaction as possible and requests the user to further input more refined information through voice prompts. If , it indicates that the distance between the user's constructed section and the target plane is slightly larger, and there is a certain error, so the system rates the current interaction as possible and requests the user to further input valuable confirmation information. The user can improve the weight of a certain plane by inputting voice commands, reselect after recalculation, or directly issue a re-input command. If , it indicates that the user's constructed section is not accurate, and the system rates it as difficult, making it difficult to infer the user's true intention, so the system prompts the user to re-input and requests the user to confirm.

[0038] In addition, the system can also provide a navigation selection method for the user to reduce the user's operation load. When the system's initial understanding of the user's selected intention is inaccurate, the system can find the optimal selection cutting direction of the second and third ranked planes in terms of probability size for the user, i.e., keeping the normal vector N of the smart pen parallel to the normal vector of one of the planes and drawing a vector from the pen tip pointing to the plane. The user can follow this trajectory to approach the plane to calculate a better probability result based on the distance and angle.

[0039] For example, Figures 3-4As shown, the embodiment also provides a multi-dimensional only-interaction pen-oriented human-computer collaborative system, comprising: The intelligent pen hardware module is composed of a 3D printing pen body, a posture sensor MOU-6050, touch sensors TS1, TS2 and TS3, and is used for collecting physical parameters of user operation; The multi-modal information acquisition module includes a vision unit, a voice unit and a sensor unit, and is used for synchronously acquiring vision, voice and sensor information; The data processing module includes a plane selection sub-module, an intention fusion sub-module and a dynamic weight updating sub-module; The interaction feedback module generates operation prompts or navigation guides in the form of vision and voice according to the output result of the data processing module and in combination with evaluation rules.

[0040] The intelligent pen hardware module further includes a Bluetooth serial communication unit, which is used for real-time transmission of data collected by the posture sensor and the touch sensor to the computer end, so as to realize mapping of real motion trajectories to virtual space; the vision unit in the multi-modal information acquisition module maps real-time tracking results of the intelligent pen in real three-dimensional space to virtual space through virtual-real fusion technology.

[0041] The data processing module further includes a historical interaction information library, which is used for storing error information and corresponding plane selection records fed back by the user, and provides a statistical basis for the number of errors for the dynamic weight updating sub-module; when the user expresses the intention of reselecting or abandoning interaction through a voice instruction, the historical interaction information library does not record the interaction data of this time.

[0042] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A human-machine collaboration method for a multi-dimensional intelligent interaction pen, characterized in that, The method comprises the following steps: S1: synchronously collecting user interaction information through a visual channel, a voice channel and a sensor channel; S2: a user performs a cutting operation on a virtual geometric body through an intelligent pen to construct a virtual cutting surface, the system calculates the coordinates of the cutting points of the cutting surface and each edge of the geometric body and generates a cutting plane point set, and a matching degree of the cutting surface and a selected plane is calculated by using a bidirectional Hausdorff distance and a normal vector angle; S3: selecting an intention with the maximum probability in each channel as a candidate intention, if the channel intentions are consistent, calculating a channel expectation value by using information entropy and fusing by linear weighting, and if the intentions conflict, prompting the user to re-input; S4: According to the plane being selected probability and the probability interval threshold, combining the plane number of the interval where the plane with the highest probability is located The interactive clarity is evaluated, and a corresponding operation prompt or navigation type selection guide is generated. 2.The method of claim 1, wherein, Step S1 is specifically: The visual channel obtains motion information of the tip of the intelligent pen in a real space by a depth camera and a YOLOv5 target detection network and maps the motion information to a virtual space, and simultaneously expands the tip of the intelligent pen from a point model to a triangular surface model; the voice channel collects user voice instructions by a microphone and converts the voice instructions into text information; and the sensor channel collects three-dimensional motion parameters and operation instructions of the intelligent pen by a posture sensor and a touch sensor; The processing of the visual channel further comprises: capturing the position of the tip of the intelligent pen in real time by a Realsense depth camera and projecting the position to a three-dimensional virtual space constructed by Unity to obtain the coordinates of a point P in the triangular surface model; obtaining the direction of a normal vector N of the plane by the posture sensor, and calculating a plane equation by combining the coordinates of the point P and the normal vector N; The processing of the voice channel comprises: after converting the voice instructions into Chinese text, performing word segmentation and removing modal particles, generating a sentence vector by using a word2vec Chinese word vector model, calculating the cosine similarity of the sentence vector and the sentence vectors in a voice intention library, and taking the results with a similarity greater than 80% as a voice sub-intention set; The hardware of the sensor channel comprises: a posture sensor MOU-6050 for obtaining acceleration, angular velocity and angle information of the intelligent pen; a touch sensor TS1 for executing a plane selection algorithm by long pressing, executing geometric body scaling by single clicking and executing geometric body rotation by double clicking; and touch sensors TS2 and TS3 for determining and canceling function buttons respectively. 3.The method of claim 1, wherein, In step S2, the matching degree of the cutting surface and the selected plane is calculated by using a bidirectional Hausdorff distance and a normal vector angle, and specifically: ； in, For the tangent plane and the first Hausdorff distance of each candidate plane The normal vector of the tangent plane and the first The angle between the normal vectors of the candidate planes and , For the first The probability of a plane being selected.

4. The human-machine collaborative method according to claim 3, wherein, Step S2 further comprises: before calculating the Hausdorff distance, filtering out the planes corresponding to the normal vectors with an angle greater than 45° with the normal vector direction of the plane of the intelligent pen when the user presses the touch sensor, and only performing Hausdorff distance calculation and probability sorting on the remaining planes.

5. The human-machine collaborative method according to claim 1, wherein, In step S3, the initial weight calculation process of calculating the channel expectation value by using information entropy and fusing by linear weighting is: ； ； wherein, is the initial weight for the visual channel, is the initial weight for the speech channel, and the weights are updated as follows when the user feeds back an understanding error: ； ； wherein the number of errors in the first plane, is a power exponent controlling the weight decrease speed, is the updated visual channel weight, is the updated speech channel weight.

6. The human-machine collaborative method according to claim 5, wherein, In step S3, the calculation formula of the information entropy is: ； wherein, is the expected value for a single channel intent, is the intent extracted for a certain channel, is the determined intent in the set of intents; the information entropy is normalized: ； wherein is the mean, of the desired is the standard deviation.

7. The human-machine collaborative method according to claim 1, wherein, In step S4, the interval threshold is specifically: , , , the specific rule of the interactive clarity evaluation is: If and , the evaluation is easy, the user denies and the plane with the second highest probability is selected automatically. If and or , evaluate as possible, prompting the user to input more refined information or re-enter; If , it is evaluated as difficult, the user is prompted to re-enter and a confirmation is requested.

8. A human-machine collaborative system oriented to a multi-dimensional intelligent interaction pen based on the human-machine collaborative method oriented to the multi-dimensional intelligent interaction pen according to claim 1, characterized in that, It comprises: An intelligent pen hardware module composed of a 3D printed pen body, a posture sensor MOU-6050, a touch sensor TS1, touch sensors TS2 and TS3, for collecting physical parameters of user operations; A multi-modal information acquisition module comprising a visual unit, a voice unit and a sensor unit, for synchronously collecting visual, voice and sensor information; Data processing module: including surface selection sub-module, intention fusion sub-module and dynamic weight updating sub-module; Interactive feedback module: according to the output result of data processing module, combined with evaluation rules, the operation prompt or navigation guide in the form of vision and voice is generated.

9. The human-machine collaborative system oriented to the multi-dimensional intelligent interaction pen according to claim 8, characterized in that, The intelligent pen hardware module further comprises a Bluetooth serial communication unit for real-time transmission of data collected by the posture sensor and the touch sensor to the computer end, realizing mapping of real motion trajectory to virtual space; the visual unit in the multi-modal information acquisition module maps the real-time tracking result of the intelligent pen in the real three-dimensional space to the virtual space through virtual-real fusion technology.

10. The human-machine collaborative system oriented to the multi-dimensional intelligent interaction pen according to claim 8, characterized in that, The data processing module further comprises a historical interaction information library for storing error information and corresponding plane selection records fed back by the user, providing a statistical basis for the number of errors for the dynamic weight updating sub-module; when the user expresses the intention of reselecting or abandoning the interaction through voice instructions, the historical interaction information library does not record the interaction data this time.

Images