Touch interaction method and device, intelligent interaction panel and storage medium

Abstract

The application provides a touch interaction method and device, an intelligent interaction panel and a storage medium. The method is applied to the technical field of interaction devices. The method comprises the following steps: acquiring a shooting picture of a camera; identifying the shooting picture and determining whether a writing action of a user on the intelligent interaction panel is identified; if the writing action is identified, determining the type of the writing action; wherein the type represents an action object generating the writing action; determining a target touch operation corresponding to the type of the writing action, and executing the target touch operation. The method can facilitate the user to perform a touch operation on the interface of the intelligent interaction panel, and improve the use experience of the user on the intelligent interaction panel.

Description

Technical Field

[0001] This application relates to the field of interactive device technology, and more specifically, to a touch interaction method, device, smart interactive flat panel, and storage medium in the field of interactive devices. Background Technology

[0002] With the rapid development of smart interactive whiteboards, the functions they can perform are becoming increasingly diverse. Currently, in order for users to perform touch operations on the smart interactive whiteboard interface, such as writing, users need to first click the "Annotate" button on the sidebar of the smart interactive whiteboard to enter annotation mode before they can start writing. This operation is sometimes inconvenient for users and is not conducive to the user's use and writing experience. Summary of the Invention

[0003] This application provides a touch interaction method, device, smart interactive whiteboard, and storage medium, which facilitates touch operation on the interface of the smart interactive whiteboard and improves the user experience.

[0004] In a first aspect, a touch interaction method is provided, comprising: acquiring a camera's captured image; recognizing the captured image and determining whether a user's writing action on a smart interactive tablet is recognized; if the writing action is recognized, determining the type of the writing action; wherein the type represents the action object that generates the writing action; determining a target touch operation corresponding to the type of the writing action, and executing the target touch operation.

[0005] The above technical solution allows the smart interactive flat panel to capture images from a camera and identify the user's writing action. If the writing action is detected, the type of writing action is determined, and a corresponding touch operation is executed. The entire process is seamless for the user; they only need to focus on writing and do not require additional operations (such as clicking buttons) to achieve the touch function. This greatly simplifies the user's operation process and facilitates touch operation on the smart interactive flat panel interface. Users do not need to remember complex operation steps; they can simply operate according to their daily writing habits, lowering the barrier to entry and improving the user experience of the smart interactive flat panel.

[0006] In conjunction with the first aspect, in some possible implementations, the above-mentioned identification of the captured image and determination of whether the user's writing action on the smart interactive flat panel is identified includes: preprocessing the captured image to obtain a preprocessed captured image; wherein the preprocessed captured image meets the input requirements of a pre-trained target detection model; inputting the preprocessed captured image into the target detection model to obtain a feature map output by the target detection model; wherein the feature map includes at least one prediction box, the feature information of the prediction box includes a target confidence score, the target confidence score being the confidence score that the writing action exists within the prediction box; and determining whether the user's writing action on the smart interactive flat panel is identified based on the target confidence score of the prediction box in the feature map.

[0007] The above technical solution, by preprocessing the captured image, ensures that the preprocessed image meets the input requirements of the object detection model, which helps improve the accuracy of subsequent model predictions. Object confidence reflects the probability of writing action within the predicted bounding box; therefore, based on the object confidence of the predicted bounding box, it is beneficial to accurately identify whether writing action exists in the current captured image.

[0008] In conjunction with the first aspect and the above implementation, in some possible implementations, the feature information of the predicted box further includes: a first confidence level and a second confidence level; the first confidence level is the confidence that the type of the writing action existing within the predicted box is a first type, and the first type indicates that the object of the writing action is a hand; the second confidence level is the confidence that the type of the writing action existing within the predicted box is a second type, and the second type indicates that the object of the writing action is an interactive accessory; determining the type of the writing action includes: when the feature map includes multiple predicted boxes, filtering the multiple predicted boxes in the feature map according to a preset filtering strategy to obtain a target predicted box in the feature map; and determining the type of the writing action within the target predicted box based on the first confidence level and the second confidence level of the target predicted box.

[0009] The above technical solution employs a preset filtering strategy to screen multiple predicted bounding boxes in the feature map, effectively removing low-confidence or irrelevant predicted bounding boxes, thereby concentrating computational resources on processing the regions most likely to contain actual writing actions. By providing a first confidence level (for the hand) and a second confidence level (for interactive accessories), different types of writing actions can be distinguished more accurately, improving the accuracy of the final determined writing action type.

[0010] In conjunction with the first aspect and the above implementation, in some possible implementations, the filtering of multiple predicted boxes in the feature map according to a preset filtering strategy to obtain the target predicted box in the feature map includes: filtering based on the target confidence of each of the multiple predicted boxes in the feature map, and taking the predicted boxes with a target confidence greater than a preset confidence threshold as the filtered candidate predicted boxes; calculating the Bayesian probability of the candidate predicted boxes based on the target confidence, the first confidence, and the second confidence of the candidate predicted boxes; filtering based on the Bayesian probability of the candidate predicted boxes, and taking the candidate predicted boxes with a Bayesian probability greater than a preset probability threshold as the filtered valid predicted boxes; and performing non-maximum suppression on the valid predicted boxes to filter out duplicate valid predicted boxes, thereby obtaining the target predicted box in the feature map.

[0011] The above technical solution first performs preliminary screening based on target confidence levels, retaining predicted boxes with a high probability of containing writing actions, thus improving the efficiency of subsequent processing steps. By setting a preset probability threshold, candidate predicted boxes with high Bayesian probabilities are further filtered as effective predicted boxes, ensuring the high reliability of the final result. After obtaining effective predicted boxes, non-maximum suppression is used to remove duplicate or overlapping predicted boxes, allowing the selection of the most suitable predicted box, i.e., the target predicted box, avoiding interference from multiple similar results. This implementation method, by combining confidence threshold filtering, Bayesian probability calculation, and non-maximum suppression, achieves high-precision recognition of writing action types, not only improving recognition accuracy but also effectively reducing redundant information and optimizing the performance of the smart interactive whiteboard.

[0012] In combination with the first aspect and the above implementation, in some possible implementations, the above-mentioned non-maximum suppression to filter duplicate valid prediction boxes based on the valid prediction boxes to obtain the target prediction boxes in the feature map includes: determining the target type corresponding to the valid prediction boxes based on the first confidence level and the second confidence level of the valid prediction boxes; wherein, the target type is the type of writing action corresponding to the highest confidence level among the first confidence level and the second confidence level of the valid prediction boxes; for each valid prediction box with the same target type, non-maximum suppression is performed on each valid prediction box to filter duplicate valid prediction boxes to obtain the target prediction boxes in the feature map.

[0013] The above technical solution selects the writing action type corresponding to the highest confidence level as the target type of the effective prediction box based on the first and second confidence levels of the effective prediction boxes. This ensures that each effective prediction box is assigned to the most likely writing action type, thereby improving classification accuracy. For each effective prediction box, non-maximum suppression is applied categorically, which helps obtain target prediction boxes of different types in the feature map, thus retaining the most representative detection and recognition results for different types.

[0014] In conjunction with the first aspect and the above implementation methods, in some possible implementation methods, the above-mentioned non-maximum suppression to filter duplicate effective prediction boxes includes: sorting the effective prediction boxes according to their Bayesian probabilities to obtain a sorting result; calculating the intersection-union ratio (IUR) between the effective prediction box with the highest Bayesian probability and a reference effective prediction box according to the sorting result; wherein the reference effective prediction box is an effective prediction box other than the effective prediction box with the highest Bayesian probability among the effective prediction boxes; if there is a reference effective prediction box with an IUR greater than a preset IUR threshold, it is determined that the reference effective prediction box is duplicated with the effective prediction box with the highest Bayesian probability, and the reference effective prediction box is filtered.

[0015] The above technical solution first sorts the valid predicted boxes according to their Bayesian probabilities, prioritizing those with high Bayesian probabilities to ensure that regions most likely to contain writing actions are considered first, thus improving the overall recognition accuracy. For the valid predicted box with the highest Bayesian probability, the intersection-union ratio (IUR) with all other reference valid predicted boxes is calculated sequentially. This accurately determines whether there is significant overlap between two valid predicted boxes, thereby deciding whether to filter out certain valid predicted boxes. Since the IUR is calculated with the valid predicted box with the highest Bayesian probability each time, and overlapping predicted boxes are filtered out step by step, the entire process becomes more efficient and reduces unnecessary computation. By combining Bayesian probability sorting and IUR calculation, the accuracy and efficiency of writing action recognition are significantly improved. It not only more accurately selects the target predicted box most likely to contain writing actions but also effectively reduces redundant information and optimizes the performance of the smart interactive whiteboard.

[0016] In combination with the first aspect and the above implementation, in some possible implementations, the calculation of the Bayesian probability of the candidate prediction box based on the target confidence level, the first confidence level, and the second confidence level of the candidate prediction box includes: selecting the highest confidence level among the first confidence level and the second confidence level of the candidate prediction box; and taking the product of the target confidence level and the highest confidence level as the Bayesian probability of the candidate prediction box.

[0017] The above technical solution, by selecting the higher value between the first and second confidence levels, can more accurately reflect the most likely type of writing action within the current candidate prediction box. This helps reduce misjudgments and improve classification accuracy. The target confidence level represents the overall probability of a writing action existing within the prediction box, while the highest confidence level further refines the confidence level for specific types. The product of the two can, to some extent, balance the influence of overall confidence and specific type confidence, providing more reliable evaluation results.

[0018] Combining the first aspect and the above implementation methods, in some possible implementation methods, the feature map output by the above object detection model includes multiple feature maps of different scales; when the above feature map includes multiple prediction boxes, filtering the multiple prediction boxes in the above feature map according to a preset filtering strategy to obtain the target prediction box in the above feature map includes: for each scale feature map in the multiple feature maps of different scales, when the above feature map includes multiple prediction boxes, filtering the multiple prediction boxes in the above feature map according to a preset filtering strategy to obtain the target prediction box for each of the multiple scale feature maps; determining the type of the writing action within the above target prediction box based on the first confidence and second confidence of the above target prediction box includes: filtering out non-repeating target prediction boxes from the target prediction boxes in the feature maps of the multiple scales; determining the type of the writing action within the non-repeating target prediction boxes based on the first confidence and second confidence of the above non-repeating target prediction boxes.

[0019] The above technical solution, because the object detection model outputs feature maps at multiple scales, can capture writing actions of varying sizes, effectively recognizing them regardless of the size of the area occupied by the writing action in the captured image. For each scale of feature map, the predicted bounding boxes are filtered according to a preset filtering strategy to obtain their respective target prediction boxes, ensuring that each scale of feature map provides the most likely prediction box containing the writing action, thereby improving classification accuracy. By integrating target prediction boxes from multiple scales, the existence and type of writing actions can be evaluated from different perspectives, reducing the risk of misjudgment that may arise from a single scale. By selecting non-repeating target prediction boxes from all scales of feature maps, the situation where the same writing action is detected multiple times at different scales is avoided, effectively removing redundant information and making the final result more concise and reliable.

[0020] Combining the first aspect and the above implementation methods, in some possible implementation methods, the above object detection model is trained in the following way: Obtaining a screen sample dataset; wherein the screen sample dataset includes several screen samples, each screen sample is labeled with a first label representing the presence or absence of a writing action; for screen samples labeled with the first label representing the presence or absence of a writing action, a second label is also labeled, the second label representing the type of writing action in the screen sample; preprocessing the screen sample dataset to obtain a preprocessed screen sample dataset; wherein several screen samples in the preprocessed screen sample dataset are adapted to the input of the object detection model to be trained; importing the preprocessed screen sample dataset into the object detection model to be trained for training, thereby obtaining a trained object detection model.

[0021] The above technical solution provides rich supervision information for the target detection model to be trained by labeling the image samples with a first label (representing the presence of writing action) and a second label (representing the type of writing action). Detailed labeling helps the target detection model learn the features of different writing actions, improving classification accuracy. Preprocessing the image sample dataset to meet the input requirements of the target detection model not only improves the efficiency of model training but also enhances the model's robustness to input data.

[0022] In conjunction with the first aspect and the above implementation methods, in some possible implementation methods, determining the target touch operation corresponding to the type of the writing action and executing the target touch operation includes:

[0023] If the type of the writing action is a first type, then the target touch operation corresponding to the type of the writing action is determined as a first operation; wherein, the first type indicates that the action object that produces the writing action is a hand; if the type of the writing action is a second type, then the target touch operation corresponding to the type of the writing action is determined as a second operation; wherein, the second type indicates that the action object that produces the writing action is an interactive accessory.

[0024] The aforementioned technical solution can distinguish whether the writing action is generated by the hand or an interactive accessory (such as a stylus), and execute different target touch operations accordingly, which helps improve the touch interaction experience and operational efficiency. It can not only accurately identify the type of the user's writing action, but also provide personalized operation support based on different types, offering users a more precise and smooth operating experience.

[0025] Combining the first aspect and the above implementation methods, in some possible implementation methods, one of the above first operation and the above second operation is an erase operation, and the other is a comment operation in the comment mode.

[0026] The above technical solution allows users to enter annotation mode and erase without clicking the annotation button. For example, when writing is detected as being done through an interactive accessory (such as a stylus), annotation mode is automatically entered, allowing users to annotate. This eliminates the need to manually click the annotation button, enabling more natural annotation. Similarly, when writing is detected as being done by hand, the erase operation is automatically executed, eliminating the need to manually click the erase button and allowing for easier content modification and adjustment. Automatically switching operating modes based on different writing action types reduces the time users spend switching between different functions, improving operational efficiency and user experience.

[0027] Secondly, a device for controlling an intelligent interactive flat panel is provided, the device comprising: an acquisition module for acquiring a camera image; an identification module for identifying the captured image and determining whether a user's writing action on the intelligent interactive flat panel is identified; a determination module for determining the type of the writing action if the writing action is identified; wherein the type represents the object of the action that generates the writing action; and an execution module for determining a target touch operation corresponding to the type of the writing action and executing the target touch operation.

[0028] Thirdly, a smart interactive flat panel is provided, comprising: a memory for storing executable program code; and a processor for calling and running the executable program code from the memory, so that the smart interactive flat panel performs the method described in the first aspect or any possible implementation thereof.

[0029] Fourthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0030] Fifthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the display interface of a smart interactive flat panel provided in an embodiment of this application;

[0032] Figure 2 This is a schematic diagram of the display interface of another smart interactive flat panel provided in an embodiment of this application;

[0033] Figure 3 This is a schematic diagram illustrating an application scenario of a touch interaction method provided in an embodiment of this application;

[0034] Figure 4 This is a schematic flowchart of a touch interaction method provided in an embodiment of this application;

[0035] Figure 5 This is a schematic diagram of a camera capture image provided in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of a YOLOv5s model architecture provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of a method for calculating the intersection-union ratio provided in an embodiment of this application;

[0038] Figure 8 This is a schematic diagram of multiple valid prediction boxes provided in an embodiment of this application;

[0039] Figure 9 This is a schematic flowchart of another touch interaction method provided in the embodiments of this application;

[0040] Figure 10 This is a schematic diagram of the structure of the touch interaction device provided in the embodiments of this application;

[0041] Figure 11 This is a schematic diagram of the structure of a smart interactive flat panel provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0043] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0044] With the rapid development of smart interactive whiteboards, the functions they can perform are becoming increasingly diverse. An interactive smart whiteboard is an integrated device that uses touch technology to control displayed content and achieve human-computer interaction. It integrates one or more functions such as a projector, electronic whiteboard, screen, speakers, television, and video conferencing terminal. Users can use interactive accessories or their own fingers to touch-operate the smart interactive whiteboard, enabling functions such as writing and erasing annotations.

[0045] See Figure 1 , Figure 1 This is a schematic diagram of the display interface of a smart interactive flat panel in related technologies. For example... Figure 1 As shown, the smart interactive whiteboard displays slide page 101 and sidebar 102. Figure 1 The display interface indicates that the smart interactive whiteboard has not yet entered annotation mode. The sidebar 102 includes an "Annotate" button 1021 for entering annotation mode. Currently, to enable users to write annotations on the smart interactive whiteboard interface, users need to first click the "Annotate" button 1021 in the sidebar 102 to enter annotation mode before they can begin writing. This operation is sometimes inconvenient for users, especially during PowerPoint presentations. When switching between slides, users need to click the "Annotate" button 1021 again to enter annotation mode, which is detrimental to the user experience. Furthermore, users may find it difficult to locate the "Annotate" button 1021 promptly, further hindering the user experience.

[0046] See Figure 2 , Figure 2 This is a schematic diagram of the display interface of another type of intelligent interactive flat panel in related technologies. For example... Figure 2 As shown, the smart interactive whiteboard displays a slide page 101, a sidebar 102, and a toolbar 201. Figure 2 The display interface indicates that the smart interactive whiteboard has entered annotation mode. Therefore, the toolbar 201 for annotation mode is displayed on the interface. The toolbar 201 includes: an exit button 2011 for exiting annotation mode, an erase button 2012 for erasing annotations, and a save button 2013 for saving annotations. Currently, in order for users to erase annotations on the smart interactive whiteboard interface, they need to click the erase button 2012 on the toolbar 201 before they can perform the erase operation. This operation is sometimes inconvenient for users and is not conducive to the user's experience in using and erasing annotations.

[0047] Therefore, how to facilitate touch operation on the interface of a smart interactive whiteboard and improve the user experience has become an urgent technical problem to be solved. To at least address this technical problem, embodiments of this application provide a touch interaction method applied to a smart interactive whiteboard. This method aims to automatically recognize the user's writing actions using a camera and visual recognition technology, and execute corresponding touch operations based on the recognition results. This eliminates the need for the user to click relevant buttons before the smart interactive whiteboard can perform touch operations. For example, the user does not need to click the "annotate" button; based on the recognized writing actions, they can enter annotation mode to perform writing operations, thereby improving the user experience of the smart interactive whiteboard.

[0048] The camera is used to capture images of the space where the smart interactive whiteboard is located. If the user writes in front of the whiteboard, the writing action will be captured in the camera's view; that is, the camera can capture the user's writing action in front of the whiteboard. In the specific implementation, the camera's view covers the space where the smart interactive whiteboard is located, ensuring that the user's whole body or at least upper body can be captured in the image.

[0049] In some embodiments, the camera can be a built-in camera of the smart interactive flat panel, meaning the built-in camera is a camera inherent to the smart interactive flat panel itself. This built-in camera supports a top-down view and can capture the user's writing actions in front of the smart interactive flat panel.

[0050] In other embodiments, the camera can be an external camera of the smart interactive tablet. The external camera is an additional camera that is connected to the smart interactive tablet, so that the smart interactive tablet can obtain the image captured by the external camera.

[0051] Figure 3 This is a schematic diagram illustrating an application scenario of a touch interaction method provided in an embodiment of this application.

[0052] For example, such as Figure 3 As shown, the application scenarios of the touch interaction method include: smart interactive flat panel 301 and camera 302, where camera 302 can be understood as an external camera of smart interactive flat panel 301.

[0053] A communication connection is established between the smart interactive flat panel 301 and the camera 302, so that the smart interactive flat panel 301 can obtain the shooting screen of the camera 302.

[0054] exist Figure 3 In this embodiment, camera 302 is positioned at the top center of the smart interactive flat panel 301. However, in specific implementations, camera 302 can also be positioned in other locations, such as the top left or top right of the smart interactive flat panel 301. This embodiment does not specifically limit the position of camera 302, as long as it can capture the user's writing actions on the smart interactive flat panel. Figure 3 The number of cameras is one. In specific implementations, multiple cameras can be set according to actual needs, such as setting cameras on the top left and top right positions of the smart interactive flat panel 301. However, this embodiment does not specifically limit the number of cameras.

[0055] Figure 4 This is a schematic flowchart of a touch interaction method provided in an embodiment of this application.

[0056] For example, such as Figure 4 As shown, the touch interaction method includes:

[0057] Step 401: Acquire the camera's captured image;

[0058] Step 402: Recognize the captured image and determine whether the user's writing action on the smart interactive whiteboard is recognized;

[0059] Step 403: If a writing action is detected, determine the type of the writing action; whereby the type represents the object of the action that produces the writing action;

[0060] Step 404: Determine the target touch operation corresponding to the type of writing action and execute the target touch operation.

[0061] exist Figure 4In the illustrated embodiment, the smart interactive flat panel can acquire the camera's captured image and recognize it to determine whether the user's writing action is detected. If the writing action is detected, the type of writing action is further determined, and a target touch operation corresponding to that type of writing action is executed. The entire process is "seamless" for the user; the user only needs to focus on writing and can achieve touch functionality without additional operations (such as clicking relevant buttons). This greatly simplifies the user's operation process and facilitates touch operations on the smart interactive flat panel's interface. Users do not need to remember complex operation steps; they can simply operate according to their daily writing habits, lowering the barrier to entry and improving the user experience of the smart interactive flat panel.

[0062] The following is about Figure 4 The specific implementation methods of each step in the illustrated embodiment are explained below:

[0063] In step 401, the smart interactive flat panel acquires the captured image from the camera, which is the image within the spatial area where the smart interactive flat panel is located. Specifically, the smart interactive flat panel can control the camera to start, and after the camera starts, it can take a picture and send the captured image to the smart interactive flat panel, so that the smart interactive flat panel can acquire the captured image from the camera.

[0064] Figure 5 This is a schematic diagram of a camera capture image provided in an embodiment of this application.

[0065] For example, such as Figure 5 As shown, the captured image 500 can be understood as... Figure 3 The image captured by camera 302 is shown in the image. Camera 302, capable of capturing a 500-degree field of view, is actually a fisheye camera. A fisheye camera is a special type of wide-angle lens camera that can capture a very wide field of view. Using a fisheye camera, the entire area in front of the smart interactive whiteboard can be covered, ensuring that the user can be captured regardless of their position while writing. If multiple users are operating simultaneously, the fisheye camera can better capture the movements of all users, improving the collaborative experience. Fisheye cameras provide a wide field of view at close range, making them effective even in confined spaces such as small meeting rooms or classrooms.

[0066] Image 500 shows a user writing with a stylus in front of a smart interactive tablet. Image 500 captures information such as how the user holds the stylus, their hand gestures while writing, and the specific position of the pen tip in contact with the smart interactive tablet. Although camera 302 primarily captures the user's movements, the smart interactive tablet's interface also appears in image 500, facilitating the identification of the pen tip's contact position with the tablet.

[0067] In step 402, the smart interactive whiteboard can use image recognition technology to identify the captured image and determine whether the user's writing action on the smart interactive whiteboard is recognized. The writing action may be performed by the user's hand or by the user holding an interactive accessory, such as a stylus.

[0068] For example, the writing action can be considered as the target to be detected, and target detection technology can be used to perform target detection in the captured image to obtain the target detection result. The target detection result includes: whether the writing action was recognized.

[0069] In some embodiments, step 402 is implemented as follows: S11 to S13:

[0070] S11: Preprocess the captured image to obtain the preprocessed captured image.

[0071] The preprocessed captured images meet the input requirements of the pre-trained object detection model. Before inputting the camera's captured images into the object detection model, a series of preprocessing steps are performed to ensure that they meet the model's input requirements. These preprocessing steps include, but are not limited to, format adjustment and resolution adjustment. Object detection models typically have fixed input requirements, such as a resolution of 640x640 pixels and an input format of RGB or grayscale. If the captured image resolution is 1920*1080 pixels and the format is NV12, then the captured image can be adjusted in both resolution and format to achieve a resolution of 640x640 pixels and an RGB or grayscale format.

[0072] In some embodiments, the resolution adjustment described above is achieved by scaling the captured image. During scaling, the image is scaled based on its shorter side. If the image size after scaling by the shorter side still does not meet the requirements of the object detection model (e.g., width or height not equal to 640), the missing portion can be filled with gray pixels. This ensures that the pre-processed image not only meets the input requirements of the object detection model but also avoids image distortion. For example, when the resolution of the captured image is 1920*1080 pixels, the image is scaled based on a shorter side of 1080 pixels.

[0073] S12: Input the preprocessed captured image into the target detection model to obtain the feature map output by the target detection model.

[0074] The feature map includes at least one prediction box. The feature information of the prediction box includes the target confidence score, which is the confidence score that there is a writing action in the prediction box. It represents the probability that the target detection model believes there is indeed a writing action in the prediction box.

[0075] The object detection model can be YOLO, a real-time object detection model known for its efficiency and accuracy. Considering that the object detection model will be deployed on the smart interactive whiteboard, this embodiment can use a lightweight object detection model to avoid consuming excessive processing resources on the smart interactive whiteboard.

[0076] For example, a lightweight object detection model can be YOLOv5s. YOLOv5s is a variant of the YOLO series of object detection algorithms. It is the smallest and lightest version in the YOLOv5 series, with fewer convolutional layers and parameters, making it suitable for running on devices with limited computing resources. Despite being a lightweight model, YOLOv5s still provides high detection speed, making it suitable for applications requiring real-time processing. Therefore, this embodiment uses YOLOv5s, which is beneficial for achieving real-time recognition of user handwriting actions.

[0077] Figure 6 This is a schematic diagram of a YOLOv5s model architecture provided in an embodiment of this application.

[0078] For example, such as Figure 6 As shown, the YOLOv5s model architecture includes: backbone, neck, and head.

[0079] The backbone typically refers to the feature extraction network used to extract image features from the input images. It extracts features step-by-step through multiple convolutional layers (Conv), residual blocks (C3), and SPFF (Spatial Pyramid Feature Fusion) layers, gradually reducing the size of the feature maps. These feature maps are extracted at different layers and then passed to the neck network. In this embodiment, the input image is the preprocessed captured image.

[0080] In Images[1,3,640,640], the first dimension "1" indicates a batch size of 1, the second dimension "3" represents the number of channels in the input image, which represents the three RGB channels, and the third and fourth dimensions "640,640" indicate that the resolution of the input image is 640*640.

[0081] In Conv(a1,b1,c1), the first dimension "a1" indicates that the convolutional layer will generate a1 feature maps, and the second and third dimensions "b1,c1" indicate that each feature map divides the input image into b1*c1 grids. For example, Figure 6 In Conv(32,320,320), the first dimension "32" indicates that the convolutional layer will generate 32 feature maps, and the second and third dimensions "320,320" indicate that each feature map divides the input image into a 320*320 grid.

[0082] In C3(a2,b2,c2), the first dimension "a2" indicates that the residual block will generate a2 feature maps, and the second and third dimensions "b2,c2" indicate that each feature map divides the input image into b2*c2 grids. For example, Figure 6 In C3(512,20,20), the first dimension "512" indicates that the residual block will generate 512 feature maps, and the second and third dimensions "20,20" indicate that each feature map divides the input image into 20*20 grids.

[0083] In SPFF(512,20,20), the first dimension "512" indicates that the SPFF will generate 512 feature maps, and the second and third dimensions "20,20" indicate that each feature map divides the input image into a 20*20 grid.

[0084] The neck, positioned between the backbone and the head, further integrates and refines the image features extracted by the backbone, improving the robustness of YOLOv5s. Even with minor perturbations and interferences added to the input image, a robust model can still accurately classify and recognize features. The neck receives feature maps from different levels of the backbone network. Upsampling and convolution operations are used to fuse feature maps of different scales. A concat operation is then used to stitch together feature maps from different levels to enhance feature representation capabilities.

[0085] The head, positioned after the neck, is used to predict the bounding box of the target object based on the image features output by the neck. The head receives multi-scale feature maps from the neck network and further processes these maps through multiple convolutional layers and residual blocks, ultimately outputting the target detection results, corresponding to small, medium, and large target detection results, respectively. In this embodiment, the target to be detected is the user's writing action.

[0086] In some embodiments, the object detection model can output multiple feature maps at different scales, such as... Figure 6 As shown, YOLOv5s outputs three feature maps at different scales. These three feature maps are obtained by downsampling multiple times using convolution kernels of different sizes, and correspond to target detection at different scales. They can detect large, medium, and small targets respectively. That is, they can detect large writing actions (large target detection result), medium writing actions (medium target detection result), and small writing actions (small target detection result) in the captured image.

[0087] exist Figure 6In the small object detection output [1,3,80,80,85], the first dimension "1" indicates a batch size of 1, the second dimension "3" indicates that each grid cell predicts 3 anchor boxes, the third and fourth dimensions "80,80" indicate the feature map size, that is, dividing the input image into an 80×80 grid, and the fifth dimension "85" indicates the attributes of each predicted box, i.e., the feature information of the predicted box. In the object detection output [1,3,40,40,85], the first dimension "1" indicates a batch size of 1, the second dimension "3" indicates that each grid cell predicts 3 anchor boxes, the third and fourth dimensions "40,40" indicate the feature map size, that is, dividing the input image into a 40×40 grid, and the fifth dimension "85" indicates the attributes of each predicted box, i.e., the feature information of the predicted box. The output of large object detection is [1,3,20,20,85]. The first dimension "1" indicates that the batch size is 1. The second dimension "3" indicates that each grid cell predicts 3 anchor boxes. The third and fourth dimensions "20,20" indicate the feature map size, that is, the input image is divided into a 20×20 grid. The fifth dimension "85" indicates the attributes of each predicted box, that is, the feature information of the predicted box.

[0088] Considering that in the actual implementation, there may be scenarios where multiple people are writing at the same time, it may be necessary to identify the writing actions of multiple users, which is a multi-object detection. Therefore, this embodiment uses three feature maps of different scales output by YOLOv5s, which is beneficial to realize multi-object detection in the captured image.

[0089] It should be noted that the above example only uses YOLOv5s as the object detection model. In actual implementation, other object detection models can also be selected. Furthermore, depending on the actual needs, the object detection model can be configured to output feature maps at multiple different scales. These multiple different scales are not limited to three different scales; they can also be four, five, or other scales.

[0090] S13: Based on the target confidence of the predicted box in the feature map, determine whether the user's writing action on the smart interactive whiteboard has been recognized.

[0091] The target confidence score represents the confidence level that a writing action exists within the predicted bounding box. A higher target confidence score indicates a greater probability of writing action within the predicted bounding box. Each feature map output by the object detection model may contain multiple predicted bounding boxes, each with its own target confidence score. Based on this, a pre-defined confidence threshold `object_thresh` can be defined. If a predicted bounding box with a target confidence score greater than `object_thresh` exists in the feature map, then the user's writing action on the smart interactive whiteboard is identified; otherwise, if no predicted bounding box with a target confidence score greater than `object_thresh` exists, then the user's writing action on the smart interactive whiteboard is not identified.

[0092] In some embodiments, the training method of the object detection model includes the following steps S21 to S23:

[0093] S21: Obtain the image sample dataset; wherein, the image sample dataset includes several image samples, each image sample is labeled with a first label indicating whether a writing action exists; for the image sample labeled with the first label indicating the existence of a writing action, a second label is also labeled, the second label indicating the type of writing action in the image sample.

[0094] The process of acquiring the image sample dataset mainly includes data collection and data annotation. For data collection, image samples containing or not containing writing actions are collected from various sources (such as cameras, video files, etc.). These sources should cover as many different scenes, lighting conditions, and angles as possible to increase the generalization ability of the object detection model. To make the trained object detection model perform better in recognizing writing actions on smart interactive whiteboards, during data collection, it is possible to... Figure 3 In the scenario shown, images captured by the camera on the smart interactive flat panel are collected as image samples. After data collection, data annotation is performed. For each image sample, a first label is assigned to indicate whether writing action exists. For example, binary labels 0 (no writing action) and 1 (writing action exists) can be used. For image samples labeled as having writing action, a second label is further assigned to distinguish different types of writing action (such as writing with a pen versus writing with a hand). In the specific implementation, for image samples known to have writing action, precise bounding box coordinates and corresponding second labels are generated for use in training the object detection model.

[0095] For example, suppose the collected image samples are as follows: Figure 5 The image shown is obtained through Figure 5 It can be seen that, Figure 5If a writing action exists in the image sample, and the writing action is performed by a user using a stylus, then the first label labeled on the image sample indicates the existence of a writing action, and the second label labeled on the image sample indicates that the type of writing action is pen writing.

[0096] S22: Preprocess the image sample dataset to obtain a preprocessed image sample dataset; wherein, some image samples in the preprocessed image sample dataset are adapted to the input of the target detection model to be trained.

[0097] For example, if the image format and resolution of the input to the object detection model to be trained are RGB format and 640x640 pixels, and the image format and resolution of the image samples in the image sample dataset are NV12 format and 1920x1080 pixels, then the image sample dataset is preprocessed by converting the format of the image samples in the image sample dataset from NV12 format to RGB format and scaling the size of the image samples to 640x640 to adapt to the input of the object detection model.

[0098] S23: Import the preprocessed image sample dataset into the target detection model to be trained for training, and obtain the trained target detection model.

[0099] Specifically, the preprocessed image sample dataset is imported into the target detection model to be trained. During the training process, the model parameters of the target detection model are adjusted, and the performance of the target detection model on the validation set is evaluated periodically. When the performance of the target detection model meets the preset performance requirements, the training is completed, and the trained target detection model is obtained.

[0100] After obtaining the trained object detection model, the trained object detection model is deployed on the smart interactive flat panel. After the smart interactive flat panel acquires the captured image from the camera, it can use the deployed object detection model to recognize the writing action in the captured image. If the writing action is recognized, the following step 403 is executed.

[0101] In step 403, if a writing action is identified, the type of writing action is further determined. This type of writing action represents the object that performs the writing action. The object may be a hand (such as a finger or palm), an interactive accessory, or a stylus. When the object is a hand, the type of writing action can be simply referred to as the handwriting type; when the object is a stylus, the type of writing action can be simply referred to as the pen writing type.

[0102] In some embodiments, the feature information of the prediction box further includes: a first confidence level and a second confidence level; the first confidence level is the confidence level that the type of writing action existing within the prediction box is a first type, where the first type indicates that the object of the writing action is a hand; the second confidence level is the confidence level that the type of writing action existing within the prediction box is a second type, where the second type indicates that the object of the writing action is an interactive accessory. That is, the first type refers to writing actions performed with a hand (such as fingers), and the second type refers to writing actions performed with an interactive accessory (such as a stylus).

[0103] In practical implementation, each scale's feature map may contain many prediction boxes. The feature information of each prediction box includes the center coordinates (x, y), width (w), and height (h), as well as the confidence score S0 (target confidence) indicating the presence of a writing action within the prediction box, and the confidence scores S1 and S2 for two writing types (e.g., handwriting and penmanship). Handwriting can be understood as the first type mentioned above, with S1 representing the confidence score of the first type. Penmanship is the second type, with S2 representing the confidence score of the second type. Based on this, the feature information of each prediction box can be represented by the following array: [x, y, w, h, S0, S2, S1].

[0104] For example, the array [0.513542,0.601389,0.1625,0.289815,1.0,0.95,0.05] represents the center coordinates of the prediction box, x,y, which are (0.513542,0.601389), the width w of the prediction box is 0.1625, the height h of the prediction box is 0.289815, 1.0 in the array means that the target confidence S0 is 1.0, 0.95 in the array means that the confidence of the handwritten type (i.e., the second confidence S2) is 0.95, and 0.05 in the array means that the confidence of the handwritten type (i.e., the first confidence S1) is 0.05. It should be noted that the center coordinates x, y, the width w, and the height h of the prediction box in the array [0.513542,0.601389,0.1625,0.289815,1.0,0.95,0.05] are the results after normalization.

[0105] For example, determining the type of writing action as described above includes the following steps S31 to S32:

[0106] S31: When the feature map includes multiple prediction boxes, filter the multiple prediction boxes in the feature map according to the preset filtering strategy to obtain the target prediction box in the feature map.

[0107] Understandably, when a feature map contains multiple predicted bounding boxes, there may be situations where some predicted bounding boxes enclose the same writing action, some enclose incomplete writing actions, or some have excessively low target confidence. Based on this, a preset filtering strategy can be used to filter the multiple predicted bounding boxes in the feature map, removing those with excessively low target confidence and duplicate predicted bounding boxes, thus obtaining the target predicted bounding box. The target predicted bounding box can be understood as the optimal predicted bounding box that can completely enclose the writing action; for example, the predicted bounding box that perfectly encloses the writing action can be considered the optimal predicted bounding box.

[0108] In some embodiments, the implementation of filtering multiple predicted boxes in the feature map according to a preset filtering strategy to obtain the target predicted box in the feature map may include the following steps S311 to S314:

[0109] S311: Filter based on the target confidence of each of the multiple predicted boxes in the feature map, and use the predicted boxes with a target confidence greater than the preset confidence threshold as the filtered candidate predicted boxes.

[0110] The preset confidence threshold `object_thresh` can be set according to actual needs, such as 0.4 or 0.5; this embodiment does not impose a specific limitation on this. It is understood that each predicted bounding box has its own target confidence level. Predicted bounding boxes with a target confidence level less than or equal to `object_thresh` are filtered out, and the remaining predicted bounding boxes with a target confidence level greater than `object_thresh` are used as the filtered candidate predicted bounding boxes. The target confidence level of each candidate predicted bounding box is greater than `object_thresh`.

[0111] S312: Calculate the Bayesian probability of the candidate prediction box based on the target confidence level, the first confidence level, and the second confidence level of the candidate prediction box.

[0112] Here, Bayesian probability represents the probability of two events occurring simultaneously: the event that a writing action exists in the candidate prediction box and the event that the highest confidence level between the first and second confidence levels of the candidate prediction box.

[0113] For example, the implementation of S312 above includes: selecting the highest confidence level from the first confidence level and the second confidence level of the candidate prediction box; and using the product of the target confidence level and the highest confidence level of the candidate prediction box as the Bayesian probability of the candidate prediction box.

[0114] Understandably, each candidate bounding box has its own target confidence level, first confidence level, and second confidence level. For each candidate bounding box, the target confidence level and the highest confidence level of its corresponding category are multiplied to obtain the Bayesian probability of the candidate bounding box. The highest confidence level of the category is the highest of the first and second confidence levels of the candidate bounding box. The target confidence level represents the overall probability of a writing action within the bounding box, while the highest confidence level further refines the confidence level for the specific type. The product of the two can, to some extent, balance the influence of the overall confidence level and the specific type confidence level, providing a more reliable evaluation result.

[0115] S313: Filter the candidate prediction boxes based on their Bayesian probabilities, and select the candidate prediction boxes whose Bayesian probabilities are greater than a preset probability threshold as the effective prediction boxes obtained by filtering.

[0116] The preset probability threshold can be set according to actual needs, aiming to filter out candidate prediction boxes with excessively low Bayesian probabilities. This embodiment does not specifically limit the value of the preset probability threshold. It is understood that each candidate prediction box has its own Bayesian probability. Candidate prediction boxes with Bayesian probabilities less than or equal to the preset probability threshold are filtered out, and the remaining candidate prediction boxes with Bayesian probabilities greater than the preset probability threshold are considered as the filtered valid prediction boxes. The Bayesian probability of each valid prediction box is greater than the preset probability threshold.

[0117] S314: Perform non-maximum suppression on the valid prediction boxes to filter out duplicate valid prediction boxes, and obtain the target prediction box in the feature map.

[0118] Non-maximum suppression (NMS) aims to address the issue of the same writing action being selected multiple times. When recognizing writing actions using an object detection model, the model may generate multiple predicted boxes associated with the same action. To select the optimal predicted box, i.e., the target prediction box, NMS can be used to filter out duplicate valid predicted boxes.

[0119] For example, the above-mentioned implementation of S314 includes the following S41 to S42:

[0120] S41: Determine the target type corresponding to the effective prediction box based on the first confidence level and the second confidence level of the effective prediction box.

[0121] The target type is the type of writing action corresponding to the highest confidence level between the first and second confidence levels of the valid predicted box. For example, if the first confidence level of the valid predicted box is the highest confidence level, then the target type is type one. If the second confidence level of the valid predicted box is the highest confidence level, then the target type is type two.

[0122] S42: For each valid predicted box with the same target type, non-maximum suppression is performed on each valid predicted box to filter out duplicate valid predicted boxes, and the target predicted box in the feature map is obtained.

[0123] Specifically, if each valid predicted box corresponds to a target type that exists in both the first and second categories, then non-maximum suppression (NMS) can be applied to all valid predicted boxes of the first category to filter out duplicate valid predicted boxes, resulting in target predicted boxes of the first category in the feature map. Similarly, NMS can be applied to all valid predicted boxes of the second category to filter out duplicate valid predicted boxes, resulting in target predicted boxes of the second category in the feature map. In other words, NMS is applied to both the first and second category valid predicted boxes to remove duplicate valid predicted boxes, leaving the optimal predicted box, which is the target predicted box. If each valid predicted box corresponds to only one target type, then NMS is applied to each valid predicted box to filter out duplicate valid predicted boxes, resulting in target predicted boxes of that specific type in the feature map.

[0124] For example, the implementation of non-maximum suppression to filter duplicate valid prediction boxes in S42 above includes the following steps S421 to S423:

[0125] S421: Sort the valid prediction boxes according to their Bayesian probabilities to obtain the sorting results.

[0126] Specifically, the effective prediction boxes can be sorted in descending order of their Bayesian probabilities to obtain the sorting results, which prepares for the final filtering of the prediction boxes using nonmaximum suppression.

[0127] S422: Based on the sorting results, calculate the crossover ratio (CROR) between the effective prediction box with the highest Bayesian probability and the reference effective prediction box.

[0128] The reference valid predicted boxes are the valid predicted boxes other than the one with the highest Bayesian probability. That is, the cross-union ratio (CURRO) is calculated between the valid predicted box with the highest Bayesian probability and each of the other valid predicted boxes following it. The CURRO measures the degree of overlap between two predicted boxes. Figure 7 As shown, the method for calculating the intersection-union ratio is to divide the intersection area of ​​two valid prediction boxes by their union area.

[0129] S423: If there is a reference valid prediction box with an intersection-union ratio greater than the preset intersection-union ratio threshold with the valid prediction box with the highest Bayesian probability, then it is determined that the reference valid prediction box is a duplicate of the valid prediction box with the highest Bayesian probability, and the reference valid prediction box is filtered out.

[0130] A higher Cross-Union Ratio (CUI) indicates greater overlap between the two valid prediction boxes. When the CUI exceeds a preset threshold, it means the two valid prediction boxes enclose the same region. In this case, the reference valid prediction box that overlaps with the valid prediction box with the highest Bayesian probability is filtered out, resulting in the final filtered target prediction box. When the CUI is less than or equal to the preset threshold, it means the reference valid prediction box and the valid prediction box with the highest Bayesian probability do not enclose the same writing action region, i.e., there is more than one writing action in the captured image. In this case, the reference valid prediction box is retained.

[0131] Figure 8 This is a schematic diagram of multiple valid prediction boxes provided in an embodiment of this application.

[0132] For example, such as Figure 8 As shown, the effective predicted box with the highest Bayesian probability is the optimal box. The reference effective predicted boxes include: effective predicted box 1 and effective predicted box 2. Since the intersection-union ratio (IU) between effective predicted box 1 and the optimal box is greater than a preset IU threshold, and the IU between effective predicted box 2 and the optimal box is also greater than a preset IU threshold, it indicates that effective predicted boxes 1 and 2 are both duplicates of the optimal box. Therefore, effective predicted boxes 1 and 2 can be filtered out, and the optimal box is obtained as the target predicted box.

[0133] It should be noted that, Figure 8 The example of a lemon is used for illustration only to avoid misunderstanding. In this embodiment, the target to be detected is actually the user's writing action.

[0134] S32: Determine the type of writing action within the target prediction box based on the first and second confidence levels of the target prediction box.

[0135] Specifically, if the first confidence level is the highest among the first and second confidence levels of the target prediction box, then the type of writing action within the target prediction box is determined to be type one. If the second confidence level is the highest among the first and second confidence levels of the target prediction box, then the type of writing action within the target prediction box is determined to be type two.

[0136] In some embodiments, the feature map output by the object detection model includes multiple feature maps of different scales. Based on this, when the feature map includes multiple prediction boxes, filtering the multiple prediction boxes in the feature map according to a preset filtering strategy to obtain the target prediction box in the feature map includes: for each scale of the feature map, when the feature map includes multiple prediction boxes, filtering the multiple prediction boxes in the feature map according to a preset filtering strategy to obtain the target prediction box for each of the multiple scales of the feature map; correspondingly, determining the type of writing action within the target prediction box based on the first confidence and second confidence of the target prediction box includes: filtering out non-repeating target prediction boxes from the target prediction boxes of the multiple scales of the feature map; and determining the type of writing action within the non-repeating target prediction boxes based on the first confidence and second confidence of the non-repeating target prediction boxes.

[0137] Specifically, for each scale of feature map, step S31 can be performed to obtain the optimal prediction box, i.e., the target prediction box, in the feature map of that scale. Considering that feature maps of multiple scales all come from the same shooting scene, the target prediction boxes in the feature maps of multiple scales may contain the same writing action. Therefore, non-repeating target prediction boxes can be filtered out from the target prediction boxes of the feature maps of multiple scales. For step S32, non-maximum suppression can be performed on all target prediction boxes of the feature maps of multiple scales to filter out duplicate target prediction boxes, resulting in non-repeating target prediction boxes. For each non-repeating target prediction box, the type of writing action within the target prediction box is determined based on the first confidence and second confidence of the target prediction box.

[0138] Assuming the feature map output by the object detection model includes three feature maps at different scales, the above S31 is performed on each of the three scales to obtain the object prediction boxes for each scale. Then, among all the object prediction boxes in the three scales, unique object prediction boxes are selected. Based on the relationship between the handwriting confidence (i.e., first confidence) and penmanship confidence (i.e., second confidence) of the unique object prediction boxes, the type of writing action within the unique object prediction boxes is determined. A higher handwriting confidence indicates a handwriting type (i.e., first type), and a higher penmanship confidence indicates a penmanship type (i.e., second type).

[0139] In step 404, a pre-set correspondence can be established between the type of writing action and the target touch operation. Based on this correspondence, the target touch operation corresponding to the type of writing action can be determined and executed. Different types can have the same or different target touch operations. In practice, this correspondence can be set before the smart interactive flat panel leaves the factory. Users can also modify this correspondence according to their operating habits to meet their personalized needs. Alternatively, the smart interactive flat panel may not have this correspondence set before leaving the factory; users can set it according to their operating habits when using the smart interactive flat panel.

[0140] In one possible implementation, step 404 is implemented as follows: if the type of writing action is a first type, then the target touch operation corresponding to the type of writing action is determined as a first operation; wherein, the first type indicates that the action object that generates the writing action is a hand; if the type of writing action is a second type, then the target touch operation corresponding to the type of writing action is determined as a second operation; wherein, the second type indicates that the action object that generates the writing action is an interactive accessory.

[0141] In this implementation, the correspondence between the type of writing action and the target touch operation is as follows: the first type corresponds to the first operation, and the second type corresponds to the second operation. The first operation and the second operation can be two different touch operations. That is, handwriting type corresponds to the first operation, and pen writing type corresponds to the second operation.

[0142] In one possible implementation, one of the first and second operations is an erase operation, and the other is an annotation operation in annotation mode. If the first operation is an erase operation, then the second operation is an annotation operation in annotation mode; if the first operation is an annotation operation in annotation mode, then the second operation is an erase operation. The annotation operation is used to generate handwriting in annotation mode corresponding to the touch path of the action object (such as a stylus or finger) on the smart interactive tablet. The erase operation is used to clear the handwriting in the touch area of ​​the action object in annotation mode.

[0143] In the specific implementation, if the user is accustomed to adding annotations with a stylus and erasing them with their hand, then the first operation can be set as the erase operation, and the second operation as the annotation operation. If the user is accustomed to erasing annotations with a stylus and adding annotations with their finger / palm, then the first operation can be set as the annotation operation, and the second operation as the erase operation.

[0144] For example, if the first operation is an erase operation and the second operation is an annotation operation, when the writing action is detected to be generated by an interactive accessory (such as a stylus), the second operation is executed to automatically enter annotation mode, responding to the user's annotation action. This eliminates the need to manually click the annotation button, allowing users to annotate more naturally and achieve the effect of writing without clicking the "annotate" button. When the writing action is detected to be generated by a hand, the first operation is executed to automatically perform an erase operation. This eliminates the need to manually click the erase button, allowing users to more conveniently modify and adjust the annotation content. Automatically switching operation modes according to different writing action types reduces the time users spend switching between different functions, improving operational efficiency and user experience.

[0145] For example, if multiple users' writing actions are identified in the captured image, the final number of target prediction boxes obtained after filtering will be multiple. Each target prediction box can capture the writing action of one user. In step 404 above, the target touch operation corresponding to the type of writing action in each target prediction box is determined, and the target touch operation corresponding to the writing action in each target prediction box is executed. For example, if target prediction box 1 captures the writing action of user 1, target prediction box 2 captures the writing action of user 2, user 1's writing action is of type 1, and user 2's writing action is of type 2, then for user 1's writing action, the first operation is executed in a portion of the smart interactive flat panel pointed to by user 1's writing action; for user 2's writing action, the second operation is executed in a portion of the smart interactive flat panel pointed to by user 2's writing action.

[0146] Figure 9 This is a flowchart illustrating another touch interaction method provided in an embodiment of this application.

[0147] For example, such as Figure 9 As shown, the touch interaction method includes:

[0148] Step 901: Obtain the image sample dataset.

[0149] Step 902: Preprocess the image sample dataset to obtain the preprocessed image sample dataset.

[0150] Step 903: Import the preprocessed image sample dataset into the target detection model to be trained for training, and obtain the trained target detection model.

[0151] Step 904: Deploy the trained object detection model on the smart interactive flat panel.

[0152] Among them, the smart interactive flat panel can be equipped with an NPU (Neural Processing Unit). An NPU is a processor specifically designed to accelerate machine learning and deep learning tasks. The trained object detection model can be deployed in the NPU. When performing neural network tasks, the NPU is usually more energy-efficient than general-purpose processors (such as CPUs and GPUs) and can provide higher performance with lower power consumption.

[0153] Step 905: Acquire the camera's captured image and output feature maps of three different scales of the captured image through the object detection model.

[0154] Specifically, the captured image can be preprocessed to obtain a preprocessed image, which can then be input into the object detection model to obtain three feature maps of different scales output by the object detection model.

[0155] Step 906: The feature maps at three different scales are processed using a post-processing algorithm to obtain the final recognition result.

[0156] The post-processing algorithms may include: (1) threshold filtering; (2) quicksort; and (3) nonmaximum suppression. The post-processing algorithms are described in detail below:

[0157] (1) Threshold Filtering: As mentioned above, there are many predicted boxes in the feature map at each scale. Therefore, a preset confidence threshold object_thresh is first defined to filter out some invalid boxes. That is, predicted boxes less than or equal to object_thresh are filtered out, leaving predicted boxes greater than object_thresh as candidate predicted boxes. Next, the Bayesian probability of each candidate predicted box is calculated. When the Bayesian probability is greater than the preset probability threshold, the candidate predicted box is considered a valid predicted box. The center point coordinates x, y, width w, and height h of the valid predicted box, as well as the Bayesian probability and the target type of the valid predicted box, are recorded for the next step of the filtering process.

[0158] (2) Quick sorting: Sort each effective prediction box from high to low according to the Bayesian probability to prepare for the final filtering of prediction boxes using non-maximum suppression.

[0159] (3) Non-maximum suppression: First, classify each effective prediction box according to the target type. For example, there are two types of target types, type 1 and type 2. Non-maximum suppression can be applied to the effective prediction boxes of type 1 to remove duplicate effective prediction boxes and leave the target prediction boxes of type 1. Non-maximum suppression can be applied to the effective prediction boxes of type 2 to remove duplicate effective prediction boxes and leave the target prediction boxes of type 2.

[0160] Step 907: Based on the recognition results, perform the target touch operation.

[0161] The recognition results include: the final filtered target prediction box; determining the type of writing action within the target prediction box based on its first and second confidence levels; and executing the corresponding target touch operation based on the type of writing action. For example, refer to... Figure 5 , Figure 5 The image shows the final filtered target prediction box 501, where the writing action type within the target prediction box is type two. Figure 5 The second type is the handwritten type.

[0162] In this embodiment, by acquiring the camera's captured image and using a target detection model to identify the captured image, it is determined whether the user is writing in the current captured image. If so, the type of writing action is further determined, i.e., whether the user is writing with a pen or by hand, and the corresponding target touch operation is executed, such as entering annotation mode and performing the operation of adding or erasing annotations. The whole process is imperceptible to the user, requires no excessive settings, has low requirements for the pen, and the user can write as soon as they pick up a pen, which is more in line with the user's daily experience and improves the user's experience of using the smart interactive whiteboard.

[0163] Figure 10 This is a schematic diagram of the structure of a touch interaction device provided in an embodiment of this application.

[0164] For example, such as Figure 10 As shown, the touch interaction device 1000 includes: an acquisition module 1001 for acquiring a camera image; an identification module 1002 for identifying the captured image and determining whether a user's writing action on the smart interactive tablet is identified; a determination module 1003 for determining the type of the writing action if the writing action is identified, wherein the type represents the object of the writing action; and an execution module 1004 for determining a target touch operation corresponding to the type of the writing action and executing the target touch operation.

[0165] In one possible implementation, the recognition module 1002 is specifically used for: preprocessing the captured image to obtain a preprocessed captured image; wherein the preprocessed captured image meets the input requirements of a pre-trained target detection model; inputting the preprocessed captured image into the target detection model to obtain a feature map output by the target detection model; wherein the feature map includes at least one prediction box, the feature information of the prediction box includes target confidence, the target confidence being the confidence that the writing action exists within the prediction box; and determining whether the user's writing action on the smart interactive tablet is recognized based on the target confidence of the prediction box in the feature map.

[0166] In one possible implementation, the feature information of the predicted bounding box further includes: a first confidence level and a second confidence level; the first confidence level is the confidence level that the type of the writing action existing in the predicted bounding box is a first type, and the first type indicates that the action object that produces the writing action is a hand; the second confidence level is the confidence level that the type of the writing action existing in the predicted bounding box is a second type, and the second type indicates that the action object that produces the writing action is an interactive accessory; the determining module 1003 includes: a filtering unit, used to filter the multiple predicted bounding boxes in the feature map according to a preset filtering strategy when the feature map includes multiple predicted bounding boxes, to obtain a target predicted bounding box in the feature map; and a determining unit, used to determine the type of the writing action in the target predicted bounding box based on the first confidence level and the second confidence level of the target predicted bounding box.

[0167] In one possible implementation, the filtering unit includes: a first filtering unit, configured to filter based on the target confidence of each of the multiple predicted boxes in the feature map, and to select predicted boxes with a target confidence greater than a preset confidence threshold as candidate predicted boxes obtained through filtering; a calculation unit, configured to calculate the Bayesian probability of the candidate predicted boxes based on the target confidence, the first confidence, and the second confidence of the candidate predicted boxes; a second filtering unit, configured to filter based on the Bayesian probability of the candidate predicted boxes, and to select candidate predicted boxes with a Bayesian probability greater than a preset probability threshold as valid predicted boxes obtained through filtering; and a third filtering unit, configured to perform non-maximum suppression on the valid predicted boxes to filter out duplicate valid predicted boxes, thereby obtaining the target predicted boxes in the feature map.

[0168] In one possible implementation, the third filtering unit is specifically used to: determine the target type corresponding to the effective prediction box based on the first confidence level and the second confidence level of the effective prediction box; wherein the target type is the type of writing action corresponding to the highest confidence level among the first and second confidence levels of the effective prediction box; for each effective prediction box with the same target type, non-maximum suppression is performed on each effective prediction box to filter out duplicate effective prediction boxes, thereby obtaining the target prediction box in the feature map.

[0169] In one possible implementation, the third filtering unit is specifically configured to: sort the effective prediction boxes according to their Bayesian probabilities to obtain a sorting result; calculate the intersection-union ratio (IUR) between the effective prediction box with the highest Bayesian probability and a reference effective prediction box according to the sorting result; wherein the reference effective prediction box is an effective prediction box other than the effective prediction box with the highest Bayesian probability among the effective prediction boxes; if there is a reference effective prediction box with an IUR greater than a preset IUR threshold, then it is determined that the reference effective prediction box is a duplicate of the effective prediction box with the highest Bayesian probability, and the reference effective prediction box is filtered.

[0170] In one possible implementation, the computing unit is specifically used to: select the highest confidence level among the first confidence level and the second confidence level of the candidate prediction box; and take the product of the target confidence level and the highest confidence level of the candidate prediction box as the Bayesian probability of the candidate prediction box.

[0171] In one possible implementation, the feature map output by the above object detection model includes multiple feature maps of different scales; the filtering unit is specifically used to: for each scale of the feature map of the multiple different scales, if the feature map includes multiple prediction boxes, filter the multiple prediction boxes in the feature map according to a preset filtering strategy to obtain the target prediction boxes of each feature map of the multiple scales; the determining unit is specifically used to: filter out non-repeating target prediction boxes from the target prediction boxes of the feature maps of the multiple scales; and determine the type of the writing action within the non-repeating target prediction boxes according to the first confidence and second confidence of the non-repeating target prediction boxes.

[0172] In one possible implementation, the touch interaction device further includes: a training module, configured to: acquire a screen sample dataset; wherein the screen sample dataset includes several screen samples, each screen sample being labeled with a first label representing the presence or absence of a writing action; for screen samples labeled with the first label representing the presence or absence of a writing action, a second label is also labeled, the second label representing the type of writing action in the screen sample; preprocess the screen sample dataset to obtain a preprocessed screen sample dataset; wherein several screen samples in the preprocessed screen sample dataset are adapted to the input of the target detection model to be trained; and import the preprocessed screen sample dataset into the target detection model to be trained for training to obtain a trained target detection model.

[0173] In one possible implementation, the execution module is specifically used to: if the type of the writing action is a first type, then determine the target touch operation corresponding to the type of the writing action as a first operation; wherein the first type indicates that the action object that generates the writing action is a hand; if the type of the writing action is a second type, then determine the target touch operation corresponding to the type of the writing action as a second operation; wherein the second type indicates that the action object that generates the writing action is an interactive accessory.

[0174] In one possible implementation, one of the first operation and the second operation is an erase operation, and the other is a comment operation in comment mode.

[0175] Figure 11 This is a schematic diagram of the structure of a smart interactive flat panel provided in an embodiment of this application.

[0176] For example, such as Figure 11 As shown, the smart interactive flat panel 1100 includes a memory 1101 and a processor 1102. The memory 1101 stores executable program code 11011, and the processor 1102 is used to call and execute the executable program code 11011 to perform a touch interaction method.

[0177] Furthermore, embodiments of this application also protect an apparatus that may include a memory and a processor, wherein the memory stores executable program code, and the processor is used to call and execute the executable program code to perform a touch interaction method provided in embodiments of this application.

[0178] This embodiment can divide the device into functional modules based on the above method example. For example, each module can correspond to a separate function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0179] When the functional modules are divided according to their respective functions, the device may also include an acquisition module, an identification module, a determination module, and an execution module. It should be noted that all relevant content regarding the steps involved in the above method embodiments can be referenced to the functional descriptions of the corresponding functional modules, and will not be repeated here.

[0180] It should be understood that the device provided in this embodiment is used to execute the above-described touch interaction method, and therefore can achieve the same effect as the above-described implementation method.

[0181] When using integrated units, the device may include a processing module and a storage module. Specifically, when the device is applied to a smart interactive whiteboard, the processing module can be used to control and manage the actions of the smart interactive whiteboard. The storage module can be used to support the execution of relevant program code by the smart interactive whiteboard.

[0182] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits shown in conjunction with the disclosure of this application. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0183] In addition, the device provided in the embodiments of this application may specifically be a chip, component or module. The chip may include a connected processor and a memory. The memory is used to store instructions. When the processor calls and executes the instructions, the chip can execute a touch interaction method provided in the above embodiments.

[0184] This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement a touch interaction method provided in the above embodiment.

[0185] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement a touch interaction method provided in the above embodiment.

[0186] In this embodiment, the device, computer-readable storage medium, computer program product, or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0187] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0188] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0189] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A touch interaction method, characterized in that, Applied to smart interactive flat panels; the method includes: To capture images from the camera; The captured image is identified, and it is determined whether the user's writing action on the smart interactive flat panel is recognized; If the writing action is detected, the type of the writing action is determined; wherein, the type represents the action object that produces the writing action; Determine the target touch operation corresponding to the type of the writing action, and execute the target touch operation.

2. The touch interaction method according to claim 1, characterized in that, The step of recognizing the captured image and determining whether a user's writing action on the smart interactive flat panel is recognized includes: The captured image is preprocessed to obtain a preprocessed captured image; wherein the preprocessed captured image meets the input requirements of a pre-trained target detection model; The preprocessed captured image is input into the target detection model to obtain a feature map output by the target detection model; wherein, the feature map includes at least one prediction box, and the feature information of the prediction box includes target confidence, the target confidence being the confidence that the writing action exists within the prediction box; Based on the target confidence score of the prediction box in the feature map, it is determined whether the user's writing action on the smart interactive whiteboard is recognized.

3. The touch interaction method according to claim 2, characterized in that, The feature information of the prediction box further includes: a first confidence level and a second confidence level; the first confidence level is the confidence level that the type of the writing action existing in the prediction box is a first type, and the first type indicates that the action object that produces the writing action is a hand; the second confidence level is the confidence level that the type of the writing action existing in the prediction box is a second type, and the second type indicates that the action object that produces the writing action is an interactive accessory. Determining the type of the writing action includes: When the feature map includes multiple predicted boxes, the multiple predicted boxes in the feature map are filtered according to a preset filtering strategy to obtain the target predicted box in the feature map. The type of writing action within the target prediction box is determined based on the first confidence level and the second confidence level of the target prediction box.

4. The touch interaction method according to claim 3, characterized in that, The step of filtering multiple predicted bounding boxes in the feature map according to a preset filtering strategy to obtain the target predicted bounding box in the feature map includes: Based on the target confidence scores of the multiple predicted boxes in the feature map, the predicted boxes with target confidence scores greater than a preset confidence threshold are selected as candidate predicted boxes. The Bayesian probability of the candidate prediction box is calculated based on the target confidence level, the first confidence level, and the second confidence level of the candidate prediction box. The candidate prediction boxes are filtered based on their Bayesian probabilities, and those with Bayesian probabilities greater than a preset probability threshold are selected as the effective prediction boxes obtained through filtering. Non-maximum suppression is performed on the effective prediction boxes to filter out duplicate effective prediction boxes, thereby obtaining the target prediction box in the feature map.

5. The touch interaction method according to claim 4, characterized in that, The step of performing non-maximum suppression based on the effective predicted boxes to filter out duplicate effective predicted boxes, thereby obtaining the target predicted box in the feature map, includes: Based on the first confidence level and the second confidence level of the effective prediction box, the target type corresponding to the effective prediction box is determined; wherein, the target type is the type of writing action corresponding to the highest confidence level among the first and second confidence levels of the effective prediction box; For each valid predicted bounding box with the same target type, non-maximum suppression is applied to each valid predicted bounding box to filter out duplicate valid predicted bounding boxes, thereby obtaining the target predicted bounding box in the feature map.

6. The touch interaction method according to claim 5, characterized in that, The step of performing non-maximum suppression on each valid prediction box to filter out duplicate valid prediction boxes includes: Based on the Bayesian probability of each effective prediction box, the effective prediction boxes are sorted to obtain the sorting result of the effective prediction boxes. Based on the sorting results, the intersection-union ratio (IUU) between the effective prediction box with the highest Bayesian probability and the reference effective prediction box is calculated sequentially; wherein, the reference effective prediction box is the effective prediction box other than the effective prediction box with the highest Bayesian probability among the effective prediction boxes. If there exists a reference valid prediction box whose intersection-union ratio (CUN) with the valid prediction box with the highest Bayesian probability is greater than a preset CUN threshold, then the reference valid prediction box is determined to be a duplicate of the valid prediction box with the highest Bayesian probability, and the reference valid prediction box is filtered out.

7. The touch interaction method according to claim 4, characterized in that, The step of calculating the Bayesian probability of the candidate prediction box based on the target confidence score, the first confidence score, and the second confidence score of the candidate prediction box includes: The highest confidence level is selected from the first confidence level and the second confidence level of the candidate prediction box; The product of the target confidence level and the highest confidence level of the candidate prediction box is used as the Bayesian probability of the candidate prediction box.

8. The touch interaction method according to claim 3, characterized in that, The feature map output by the target detection model includes multiple feature maps of different scales; When the feature map includes multiple predicted bounding boxes, filtering the multiple predicted bounding boxes in the feature map according to a preset filtering strategy to obtain the target predicted bounding box in the feature map includes: For each feature map at different scales, if the feature map includes multiple prediction boxes, the multiple prediction boxes in the feature map are filtered according to a preset filtering strategy to obtain the target prediction boxes of the feature maps at each scale. Determining the type of the writing action within the target prediction box based on the first and second confidence scores of the target prediction box includes: Among the target prediction boxes in the feature maps of the multiple scales, non-overlapping target prediction boxes are selected. The type of writing action within the non-overlapping target prediction boxes is determined based on the first and second confidence levels of the non-overlapping target prediction boxes.

9. The touch interaction method according to claim 2, characterized in that, The target detection model is trained in the following manner: Obtain a screen sample dataset; wherein the screen sample dataset includes several screen samples, each screen sample is labeled with a first label indicating whether a writing action exists; for screen samples labeled with the first label indicating the existence of a writing action, a second label is also labeled, the second label indicating the type of writing action in the screen sample; The image sample dataset is preprocessed to obtain a preprocessed image sample dataset; wherein, some image samples in the preprocessed image sample dataset are adapted to the input of the target detection model to be trained. The preprocessed image sample dataset is imported into the target detection model to be trained, and a trained target detection model is obtained.

10. The touch interaction method according to any one of claims 1 to 9, characterized in that, The step of determining the target touch operation corresponding to the type of the writing action and executing the target touch operation includes: If the type of the writing action is a first type, then the target touch operation corresponding to the type of the writing action is determined to be the first operation; wherein, the first type indicates that the action object that generates the writing action is the hand; If the type of the writing action is the second type, then the target touch operation corresponding to the type of the writing action is determined to be the second operation; wherein, the second type indicates that the action object that generates the writing action is an interactive accessory.

11. The touch interaction method according to claim 10, characterized in that, Of the first operation and the second operation, one is an erase operation and the other is a comment operation in comment mode.

12. A touch-screen interactive device, characterized in that, The device includes: The acquisition module is used to acquire images captured by the camera; The recognition module is used to recognize the captured image and determine whether the user's writing action on the smart interactive flat panel is recognized. A determining module is configured to determine the type of the writing action if the writing action is detected; wherein the type represents the action object that generates the writing action; An execution module is used to determine the target touch operation corresponding to the type of the writing action and to execute the target touch operation.

13. A smart interactive flat panel, characterized in that, The smart interactive flat panel includes: Memory, used to store executable program code; A processor is configured to call and run the executable program code from the memory, causing the smart interactive flat panel to perform the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 11.

Images