Method and apparatus for displaying touch trajectory, device, and medium

Abstract

The present application discloses a method and apparatus for displaying a touch trajectory, a device, and a medium. The method comprises: acquiring an image of a target region photographed by a camera, the target region comprising at least part of a touch region; on the basis of the image of the target region, determining position information and color information of a stylus located in the target region, the color information being used for representing an outer surface color of the stylus; acquiring position information of a touch point generated by the stylus touching the touch region; and, on the basis of the position information of the touch point and the position information of the stylus, determining the stylus that generated the touch point, and, on the basis of color information of the stylus that generated the touch point, assigning a corresponding color to a touch trajectory formed by the touch point. The method and apparatus for displaying a touch trajectory, the device, and the medium provided by the present application are convenient for a user to use.

Description

Method, device, equipment and medium for displaying touch track TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of touch device, in particular to a method, device, equipment and medium for displaying touch track. BACKGROUND

[0002] With the continuous improvement of scientific level, the interactive panel with touch is widely used in various scenes such as conference scene and teaching scene due to its convenience and powerful information processing and display function.

[0003] A user can write or draw on the interactive panel by using a touch pen. In an application scenario of the interactive panel, at least two users touch the interactive panel by using different touch pens. In order to distinguish the touch tracks of different touch pens, different colors can be assigned to the touch tracks of different touch pens. In order to realize the above-mentioned multi-pen and multi-color function, the touch pen can send color information to the interactive panel, and the interactive panel can receive the color information and make the touch track of the touch pen have the corresponding color.

[0004] However, the above-mentioned method has the problem of being easily interfered and affecting user use.

[0005] SUMMARY

[0006] Embodiments of the present application aim to provide a method, device, equipment and medium for displaying touch track, which can solve the problem of affecting user use.

[0007] In order to achieve the above-mentioned purpose, one aspect of embodiments of the present application provides a method for displaying touch track, which is used for a touch device with a touch panel, and the method comprises: acquiring an image of a target region shot by a camera, the target region comprising at least part of a touch region; determining position information and color information of a touch pen located in the target region according to the image of the target region, the color information being used for representing the color of the outer surface of the touch pen; acquiring position information of a touch point generated due to the touch pen touching the touch region; determining the touch pen generating the touch point according to the position information of the touch point and the position information of the touch pen, and assigning a corresponding color to a touch track formed by the touch point according to the color information of the touch pen generating the touch point.

[0008] The display method of the touch track provided in the embodiments of the present application obtains the position and color of the stylus that is touching the touch panel according to the image of the target region, and determines which stylus produces the touch point according to the position information of the touch point and the position information of the stylus, and assigns the color of the stylus that produces the touch point to the touch track formed by the touch point, so that the touch device can obtain the color of the stylus by itself, avoiding the communication between the stylus and the touch device, and facilitating the use of the user.

[0009] Another aspect of the embodiments of the present application further provides a display device of a touch track, which is used for a touch device with a touch panel having a touch region, and the touch device is integrated with a camera or externally connected with the camera, and the device comprises: an acquisition module, configured to acquire an image of a target region captured by the camera, the target region comprising at least part of the touch region; a first processing module, configured to determine the position information and color information of a stylus located in the target region according to the image of the target region; a second processing module, configured to acquire the position information of a touch point produced by the stylus touching the touch panel in the touch region; and a third processing module, configured to determine the stylus that produces the touch point according to the position information of the touch point and the position information of the stylus, and assign the touch track formed by the touch point with a corresponding color according to the color information of the stylus that produces the touch point.

[0010] Still another aspect of the embodiments of the present application further provides a touch device, comprising a touch panel, a memory and at least one processor; wherein the memory is configured to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the method as described above.

[0011] Still another aspect of the embodiments of the present application further provides a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to implement the method as described above. BRIEF DESCRIPTION OF DRAWINGS

[0012] FIG. 1 is a schematic diagram of a scenario to which the display method provided in the embodiments of the present application is applied.

[0013] FIG. 2 is a schematic diagram provided in the embodiments of the present application;

[0014] FIG. 3 is a flowchart of the display method of the touch track provided in the embodiments of the present application;

[0015] FIG. 4 is a schematic diagram of a recognition model provided in the embodiments of the present application;

[0016] FIG. 5 is a schematic diagram of a feature grid and its feature frame provided in the embodiments of the present application;

[0017] FIG. 6 is a schematic diagram of a touch point, a historical touch trajectory passing through the touch point, and a new touch trajectory passing through the touch point according to an embodiment of the present application;

[0018] FIG. 7 is a schematic diagram of a touch trajectory display device according to an embodiment of the present application;

[0019] FIG. 8 is a structural diagram of a touch device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application with reference to the accompanying drawings. It should be noted that the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0021] The following description refers to the accompanying drawings. Unless otherwise indicated, same or similar elements in different drawings are denoted by same or similar reference numerals. The implementations described in the following exemplary embodiments are not meant to represent all implementations consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0022] FIG. 1 is a schematic diagram of a scenario to which a display method according to an embodiment of the present application is applied. Referring to FIG. 1, the application scenario includes a touch object and a touch device 100. The touch device 100 is a device capable of realizing a touch function, for example, a touch device with a touch panel such as a smart interactive panel, a mobile phone, a pad, a computer, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, a personal digital assistant (PDA), a vehicle-mounted device, a wearable device, and the like. In FIG. 1, the touch device 100 is taken as an example of a smart interactive panel.

[0023] The touch object can touch the touch device 100 and provide input to the touch device 100, and the touch device 100 performs an operation in response to the input based on the input of the touch object. The touch object described herein can be any object capable of performing a touch operation on the touch device 100, such as a stylus 200, a human hand 300, and the like.

[0024] In the scene of teaching or meeting, there are often multiple users simultaneously touch the smart interactive panel through different styluses to write. In the related art, the infrared electronic pen in writing is increased with the information of emitting color ID, the infrared touch frame of the smart interactive panel needs to increase the function of receiving and decoding the color ID information, and then sends the color ID information to the whiteboard writing software to set the corresponding color, so as to realize the effect of one pen with multiple colors and different pens with different colors. However, when multiple pens write at the same time, the infrared electronic pen can only send ID information without position information, so multiple pens cannot write at the same time in this scene, because the smart interactive panel cannot confirm the corresponding relationship between the multiple writing traces and the pens according to the received multiple ID information, so the effect of displaying different colors of writing traces when multiple pens write at the same time cannot be realized. In addition, the above-mentioned scheme requires real-time communication between the stylus and the smart interactive panel during the use of the stylus. However, if the interference of the use environment is serious, the stylus and the smart interactive panel cannot communicate in real time, which will further cause the problem of incomplete touch track or serious delay.

[0025] To solve the above technical problems, the inventors of the present application consider a way of obtaining the color of the stylus by the interactive panel itself to avoid the related problems caused by the communication between the stylus and the interactive panel to obtain the color. Color is a visual perception, therefore, in the embodiments of the present application, the outer surface of the stylus is designed to have a color; the interactive panel can obtain the image of the stylus writing on the touch area of the smart interactive panel through the self-provided camera or the externally connected camera, and analyze the image, so as to obtain the color of the outer surface of the stylus.

[0026] If there is only one stylus currently touching the interactive panel, the interactive panel can obtain the color of the stylus based on the image of the stylus writing, and display the writing track with the same color as the stylus as the stylus slides. However, if multiple styluses touch the interactive panel at the same time, and the colors of the multiple styluses are different, the interactive panel can identify multiple colors based on the image of the styluses writing, and the multiple styluses can generate multiple touch tracks. At this time, how to match the corresponding colors for the multiple touch tracks is a problem to be solved. The present inventors have observed that in the scenario of multiple styluses writing at the same time, the positions of the multiple styluses touching the interactive panel relative to the interactive panel are different, and the positions of the touch points generated by the styluses are also different. Therefore, the embodiments of the present application can analyze the positions of the styluses of different colors in the image, and correspond the positions of the styluses of different colors to the positions of the multiple touch points obtained by the intelligent interactive panel, so as to know which position of the touch point is generated by the stylus of a certain color, and assign the corresponding color to the touch track of the position. For example, it is analyzed from FIG. 1 that the green stylus is located at the lower left, and the red stylus is located at the right middle. Then, the first touch point located at the lower left is generated by the green stylus, and the touch track formed by the first touch point is assigned the color green; the second touch point located at the right middle is generated by the red stylus, and the touch track formed by the second touch point is assigned the color red.

[0027] The embodiments of the present application provide a display method of a touch track, which can be applied to a touch device. The touch device can have a touch panel. FIG. 3 is a flowchart of the display method of the touch track provided by the embodiments of the present application. Referring to FIG. 3, the display method of the touch track provided by the embodiments of the present application can include the following steps:

[0028] S101, obtaining an image of a target region captured by a camera.

[0029] Specifically, the touch panel can have a touch region for a touch object to touch, and the touch device can generate a touch point when the touch object touches the touch region. The target region can include at least part of the touch region, and the target region can also include a region within a preset range from the touch region. The preset range can be selected to cover the overall appearance of the stylus currently located in the target region, or a specified position (such as the tip of the stylus, the middle of the stylus, the tail of the stylus, the position where a hand holds the stylus, etc.). For example, the stylus is generally 17 cm, and the preset range can be 19 cm. The target region can include the touch region and the region within 19 cm from the touch region. At this time, the preset range can cover the overall appearance of the stylus.

[0030] In order to obtain the image of the target region, the touch device can be integrated with a camera or externally connected with a camera, and a lens of the camera can be directed to the touch panel to take a picture. The camera can be arranged at the middle of the top end of the touch device, and the camera can be a fisheye camera (wide-angle camera) or a camera assembled by multiple lenses with different orientations, so that the picture taken can cover the whole touch panel; or the camera can be arranged at the top end of the side of the touch device, and the lens of the camera can be directed to the bottom end of the touch device to take a picture; or the camera can be arranged at the middle of the side of the touch device, the bottom end of the side, the middle of the bottom end, etc. The above are only examples of several possible positions of the camera, and the embodiments of the present application do not make specific limitations on the position of the camera and the number of cameras, as long as the camera can take pictures of the touch region and the region within the preset range from the touch region.

[0031] Optionally, when the user operates the stylus to write and slide on the touch panel of the touch device, that is, during the user uses the stylus to touch the touch panel of the touch device, the position of the stylus changes at any time. Therefore, the image of the target region can be obtained every preset time. The preset time can be selected as 2s, 1s, 0.5s, 0.1s, etc.

[0032] Optionally, in order to avoid the change of the ambient brightness causing the change of the brightness of the picture taken by the camera, thereby affecting the color of the image of the target region, the parameters of the camera can be set to a white balance mode to avoid the influence of the ambient brightness on the color part of the image of the target region.

[0033] S102, according to the image of the target region, determining the position information and color information of the stylus located in the target region.

[0034] Specifically, the color information can represent the color of the stylus. There are many methods to identify the stylus and the color of the stylus from the image. The present inventors consider that in the conference scene and the teaching scene, a large-size touch panel will be used, which will make the touch region larger. In the image of the target region, the stylus accounts for a small proportion, which belongs to small target recognition. Moreover, when multiple styluses touch the touch region at the same time, the positions and colors of the multiple styluses need to be identified at the same time, which is multi-target data recognition. Based on the above two points, the embodiments of the present application select the YOLO series model for identification.

[0035] Specifically, the image of the target area is input into the recognition model, which then outputs the position and color information of the stylus touching the touch panel within the touch area. The recognition model is trained based on the YOLO series of models, including YOLOv1, YOLOv2, YOLOv3, YOLOv4, YOLOv5, YOLOX, YOLOv6, YOLOv7, and YOLOv8. The core idea of ​​the YOLO series is to transform object detection into a regression problem, using the entire image as input to the network. Through a neural network, the location of the bounding box and its category are obtained.

[0036] The following describes how the recognition model obtains the position and color information of the stylus that is touching the touch panel.

[0037] S1021. Obtain feature information of multiple feature boxes using the image of the target region.

[0038] Specifically, referring to Figure 4, the recognition model may include a backbone network, a neck network, and a detection network (head). The backbone network extracts features from the image information of the touch area for use by subsequent networks. The neck network adds extra layers to the backbone network to improve feature representation and robustness, ensuring accurate classification even under minor perturbations and interference. The detection network, as the model's output, utilizes the features extracted from the backbone and neck networks to detect the target. Scale feature maps can be obtained using the backbone, neck, and detection networks.

[0039] It should be noted that, due to the principle of perspective, a stylus of the same size will occupy a larger area in the image of the target region captured by the camera the closer it is to the camera; conversely, the further away the stylus is from the camera, the smaller the area it occupies in the image of the target region captured by the camera. To adapt to this change, feature maps of various scales can be obtained using backbone networks, neck networks, and detection networks. These scale feature maps can be obtained by downsampling multiple times using convolutional kernels of different sizes, and these scale feature maps can correspond to targets of different scales.

[0040] For example, in Figure 4, a first-scale feature map, a second-scale feature map, and a third-scale feature map can be obtained using a backbone network, a neck network, and a detection network. The first-scale feature map is 80×80-dimensional to facilitate the detection of smaller targets far from the camera; the second-scale feature map is 40×40-dimensional to facilitate the detection of medium-sized targets at a suitable position with the camera; and the third-scale feature map is 20×20-dimensional to facilitate the detection of larger targets close to the camera.

[0041] The scale feature map can include a plurality of feature grids, at least two feature boxes of different sizes can be assigned to each feature grid, and feature information of each feature box is obtained. For example, the first scale feature map in FIG. 4 has 6400 feature grids, the second scale feature map has 1600 feature grids, and the third scale feature map has 400 feature grids. For example, the feature grid a shown in FIG. 5 has a first feature box 1, a second feature box 2, and a third feature box 3. It should be noted that FIG. 5 is only used as an example to illustrate that one feature grid has three feature boxes, and the number of feature boxes of one feature grid is not limited in the embodiments of the present application.

[0042] In addition, the feature information of the feature box can include position information, target confidence information, and color confidence information. The position information can represent the coordinates (x, y) of the center point of the feature box. The target confidence information can represent the proportion of the stylus in the feature box, and the color confidence can represent the proportion of each color of the stylus in the feature box. For example, the feature information of a certain feature box is (0.513542, 0.601389, 0.1625, 0.289815, 1.0, 0.95, 0.05, 0.00, 0.00). The feature information of the feature box can represent that the center coordinates of the feature box are (0.513542, 0.601389), the width w of the feature box is 0.1625, and the height h of the feature box is 0.289815. The target confidence is 1, the confidence of red is 0.95, the confidence of yellow is 0.05, the confidence of blue is 0.00, and the confidence of green is 0.00. It should be noted that the colors involved in the color confidence information can be two, three, etc., and the colors can be red, yellow, green, blue, black, etc. The number and types of colors involved in the color confidence information are not limited in the embodiments of the present application.

[0043] Optionally, in order to facilitate the drawing of the feature box, the feature information of the feature box can further include size information. The size information can represent the length in the x direction and the length in the y direction of the feature box.

[0044] Optionally, in order to facilitate the identification of the convergence of the model, the image can be normalized before step S1021.

[0045] S1022, according to the color confidence information, classifying the plurality of feature boxes by color.

[0046] Optionally, in order to improve the efficiency of step S1022 and step S1023, the invalid feature frame can be filtered out by target confidence first. Specifically, according to the target confidence information of the plurality of feature frames, the feature frame with target confidence higher than the first preset threshold is screened out, and the feature frame with target confidence lower than the first preset threshold is filtered out. For example, the first preset threshold is 0.75, the target confidence information of the first feature frame is 0.88, and the target confidence information of the second feature frame is 0.70. Then, the first feature frame will perform step S1022, and the second feature frame will be filtered out. In summary, the feature frame of the stylus is filtered out by the target confidence and the first preset threshold first, and then the color classification is performed by the color confidence and the adaptive frame is screened out, so as to reduce the data amount in the color classification in step S1022 and the data amount in the screening in step S1023.

[0047] Optionally, the feature frame with target confidence higher than the first preset threshold can be sorted according to the size of the target confidence information, so as to perform the plurality of feature frames in step S1022 and the plurality of feature frames in step S1023 in a certain order, and also facilitate the quick finding of the first feature frame, the second feature frame, etc. in step S1023.

[0048] Further, step S1022 can include: determining the main color of each feature frame with target confidence higher than the first preset threshold according to the color confidence information of each feature frame with target confidence higher than the first preset threshold. The main color is the color corresponding to the highest color confidence in the color confidence information of the feature frame. The feature frames with the same main color in the feature frames with target confidence higher than the first preset threshold are classified into a class. That is to say, the feature frames belonging to the same class have the same color corresponding to the highest color confidence; in other words, the feature frames belonging to the same class can frame at least part of the stylus with the same surface color, and the surface color of the stylus is the main color of the class. In other words, the position of this kind of feature frame is the position of the stylus with the surface color of the main color of the class or the vicinity thereof.

[0049] For example, the target confidence of the feature box E is higher than the first preset threshold, the color confidence of the feature box E is (0.95, 0.05, 0.00, 0.00), wherein the confidence of red is 0.95, the confidence of yellow is 0.05, the confidence of blue is 0.00, and the confidence of green is 0.00. The confidence of red is the largest, so the main color of the feature box E is red, and the feature box E can frame at least part of the red stylus; the color confidence of the feature box F is (0.00, 0.08, 0.80, 0.12), wherein the confidence of red is 0.00, the confidence of yellow is 0.08, the confidence of blue is 0.80, and the confidence of green is 0.12. The confidence of blue is the largest, so the main color of the feature box F is blue, and the feature box F can frame at least part of the blue stylus; the color confidence of the feature box G is (0.00, 0.08, 0.92, 0.00), wherein the confidence of red is 0.00, the confidence of yellow is 0.08, the confidence of blue is 0.92, and the confidence of green is 0.00. The confidence of blue is the largest, so the main color of the feature box G is blue, and the feature box G can frame at least part of the blue stylus. The main color of the feature box F and the main color of the feature box G are both blue, so the feature box F and the feature box G belong to the same class, and the feature boxes of the same class can frame at least part of the blue stylus. In other words, the blue stylus is at or near the feature box F and the feature box G.

[0050] Referring to FIG. 5, it is worth noting that the feature box E, the feature box F, and the feature box G can all frame part of the stylus. Among them, the feature box E can frame most or even all of the stylus, so that the center point coordinate of the feature box E is closest to the center point coordinate of the stylus. The position information of the feature box E can be assigned as the position information of the stylus framed by it.

[0051] Therefore, step S102 can further include: S1023, selecting an adaptive box from the feature boxes of the same class; and S1024, taking the position information of the adaptive box as the position information of one of the styluses currently touching the touch panel, and taking the color of the classification to which the adaptive box belongs as the color information of one of the styluses currently touching the touch panel.

[0052] Specifically, the adaptive box is the feature box that frames the largest same-color stylus among the feature boxes of the same class, that is, the area of the adaptive box that frames the same-color stylus is the largest among the feature boxes of the same class, so that the center point coordinate of the adaptive box is closest to the center point coordinate of the stylus, thereby improving the accuracy of image recognition. The same-color stylus refers to a stylus whose surface color is the same as the main color of the adaptive box.

[0053] For example, if the main color of the adaptive frame is red, the adaptive frame is the feature frame that encloses the stylus with the red appearance in the largest size. If the position coordinate of the adaptive frame is (x, y), the position coordinate of one of the styluses that is currently touching the touch panel is (x, y), and the color of the stylus is red.

[0054] Step S1023 can include S10231, finding the first feature frame from the same type of feature frames.

[0055] Specifically, the first feature frame (e.g., feature frame E in FIG. 5) is the feature frame that encloses the stylus with the same color in the largest size from the same type of feature frames. The stylus with the same color is the stylus with the color that is the same as the main color of the first feature frame.

[0056] The greater the target confidence and the greater the confidence of the main color, the greater the part of the feature frame that can enclose the stylus with the same color. In the same type of feature frames, the maximum target confidence and the maximum confidence of the main color can exist in the same feature frame, or the maximum target confidence and the maximum confidence of the main color can exist in different feature frames. To find the first feature frame, the target confidence and the confidence of the main color can be considered comprehensively to obtain the feature frame that can enclose the stylus with the color that is the same as the main color of the type.

[0057] For example, the product of the target confidence and the confidence of the main color of all the feature frames in the same type of feature frames can be calculated, and the feature frame with the maximum product is the first feature frame. For example, the red type of feature frames includes the first feature frame and the second feature frame. The target confidence of feature frame E is 1, the confidence of red is 0.8, and the product of feature frame E is 0.8. The target confidence of feature frame F is 0.9, the confidence of red is 0.95, and the product of feature frame F is 0.855. Since the product of feature frame F is 0.855, which is higher than the product of feature frame E, which is 0.8, feature frame F is the first feature frame.

[0058] For another example, the target confidence and the confidence of the main color can be assigned the same or different weights, the weighted target confidence and the weighted confidence of the main color can be added, and the feature frame with the maximum intersection is feature frame E. For example, the weight of the target confidence is 0.4, and the weight of the confidence of the main color is 0.6. The red type of feature frames includes feature frame E and feature frame F. The target confidence of feature frame E is 1, the confidence of red is 0.8, and the sum of feature frame E is 1 x 0.4 + 0.8 x 0.6 = 0.88. The target confidence of feature frame F is 0.9, the confidence of red is 0.95, and the sum of feature frame F is 0.9 x 0.4 + 0.95 x 0.6 = 0.93. Since the sum of feature frame F is 0.93, which is higher than the sum of feature frame E, which is 0.88, feature frame F is the first feature frame.

[0059] It is worth noting that if there are multiple feature boxes with the same maximum confidence in the same class of feature boxes, one of the feature boxes can be selected as the first feature box.

[0060] In addition, the outer surface colors of the styluses that simultaneously touch the touch panel of the touch device can be the same or different. That is, among the multiple styluses that simultaneously touch the touch panel of the touch device, there can be a case where the outer surface colors of at least two styluses are the same. In order to distinguish the positions of the two styluses, step S1023 provided by the embodiments of the present application can further include the following steps:

[0061] S10232, calculating a first intersection-over-union between the first feature box and the remaining feature boxes in the same class of feature boxes.

[0062] The first intersection-over-union is used to represent the ratio between the intersection area and the union area between the first feature box and each of the remaining feature boxes.

[0063] For example, as shown in FIG. 5, feature box E, feature box F, and feature box G belong to the same class of feature boxes. Among them, feature box E is the first feature box, and feature box F and feature box G belong to the remaining feature boxes. The intersection area of feature box E and feature box F is the intersection area of the two, and the union area of feature box E and feature box F is the sum of the area of feature box E and the area of feature box F, minus the intersection area of feature box E and feature box F. The first intersection-over-union between feature box E and feature box F is the ratio of the intersection area and the union area. Similarly, the first intersection-over-union between feature box E and feature box G can be obtained.

[0064] S10233, when all the first intersection-over-unions in the same class of feature boxes are higher than a second preset threshold, the first feature box is the adaptive box.

[0065] Specifically, in the same class of feature boxes, the number of first intersection-over-unions is the number of feature boxes in the same class of feature boxes minus one. If the intersection-over-union of two feature boxes is higher than the second preset threshold, it can be indicated that the two feature boxes are in the same region. Therefore, when all the first intersection-over-unions in the same class of feature boxes are higher than the second preset threshold, it can be indicated that all the feature boxes in the same class of feature boxes are concentrated in the same region, and there is only one same-color stylus (a stylus with an outer surface color that is the main color of the first feature box) in the touch region. The position information of the first feature box is the position information of the same-color stylus. The feature boxes in the same class of feature boxes except the first feature box can be filtered out.

[0066] S10234、in the same class of feature frame, when there is at least one small ratio feature frame, find a second feature frame from the at least one small ratio feature frame, and calculate a second intersection-over-union ratio between the second feature frame and the rest of the small ratio feature frames in the same class of feature frame. When all the second intersection-over-union ratios in the same class of feature frame are higher than a second preset threshold, the first feature frame and the second feature frame can be the adaptive frames.

[0067] Specifically, the small ratio feature frame is a feature frame in the same class of feature frame whose first intersection-over-union ratio is not higher than the second preset threshold. When there is at least one small ratio feature frame in the same class of feature frame, it indicates that there are at least two parts in the same class of feature frame that are not in the same area. Then, there are multiple same-color styluses (styluses with an outer color that is the main color of the first feature frame and the second feature frame) in the touch area, and the multiple same-color styluses can be dispersed in different positions of the touch area, and the position information of one of the styluses is the position information of the first feature frame.

[0068] The second feature frame is a feature frame in the at least one small ratio feature frame that encloses the most same-color styluses. Referring to the manner of finding the first feature frame in step S10231, those skilled in the art can easily deduce how to find the second feature frame from the at least one small ratio feature frame (such as maximum product or maximum addition, etc.), and thus will not be described here.

[0069] By finding the second feature frame and calculating the second intersection-over-union ratio, it is determined whether all the small ratio feature frames in the same class of feature frame are located in the same area. When all the second intersection-over-union ratios in the same class of feature frame are higher than the second preset threshold, it indicates that all the small ratio feature frames are located in the same area. Then, there are two same-color styluses in the touch area, the position information of one of the two styluses is the position information of the first feature frame, and the position information of the other stylus is the position information of the second feature frame. The small ratio feature frames in the same class of feature frame except the second feature frame can be filtered out.

[0070] It is worth noting that when there is a second intersection-over-union ratio in the same class of feature frame that is not higher than the second preset threshold, it indicates that all the small ratio feature frames are not located in the same area, and there are three or more same-color styluses in the touch area. Among them, a third feature frame is found, and a third intersection-over-union ratio is calculated to determine whether there is a fourth same-color stylus. A fourth feature frame is found, and a fourth intersection-over-union ratio is calculated to determine whether there is a fifth same-color stylus. In this way, it goes down until all the intersection-over-union ratios in the same class of feature frame are higher than the second preset threshold, or only one feature frame is left.

[0071] The method provided by the embodiment of the present application can further include: S103, obtaining position information of a touch point generated due to the stylus touching the touch area.

[0072] Specifically, when the stylus touches the touch control area, the touch control panel can sense the change of infrared light, the change of current, the change of voltage or the change of magnetic flux (corresponding to the specific setting type of infrared display screen, capacitive display screen, resistive display screen or electromagnetic display screen), and then the position information of the touch point is obtained.

[0073] It should be noted that when the stylus touches the touch control panel of the touch control device, not only the position information of the touch point can be obtained, but also the time information of the touch point can be obtained. During the user touch, the position information and the time information of multiple touch points can be obtained, and according to the position information and the time information, the touch trajectory formed by the touch points can be rendered and displayed on the display screen of the touch control device.

[0074] S104, according to the position information of the touch point and the position information of the stylus, the stylus generating the touch point is determined; and according to the color information of the stylus generating the touch point, the touch trajectory formed by the touch point is given a corresponding color.

[0075] Specifically, the position information of the stylus and the position information of the touch point can be fused to determine which stylus generates the touch point, and the color of the stylus is determined, and the color of the trajectory formed by the touch point is the color of the stylus.

[0076] It should be noted that when the position information and the color information of the stylus located in the target area are determined to be one in step S102, and the position information of one touch point is obtained in step S103, the stylus can be easily determined, and then the color of the touch trajectory is given. For example, in step S102, the position information (X1, Y1) of the red stylus is determined, and the color information is red. In step S103, the position information (X2, Y2) of one touch point is obtained. In step S104, it is determined that the touch trajectory formed by the touch point is red.

[0077] In addition, when the position information and the color information determined in step S102 are both multiple, and the position information of multiple touch points is obtained in step S103, in step 104, according to the position information of the touch point and the position information of the stylus, the stylus generating the touch point can include the following:

[0078] S1041, according to the position information of the multiple styluses, the multiple styluses are sorted according to a preset position rule; S1042, according to the position information of the multiple touch points, the multiple touch points are sorted according to a preset position rule; S1043, the stylus with the same sorting position as the touch point is determined as the stylus generating the touch point.

[0079] For example, the position information and color information of three styluses are determined in step S102: Q1 (0.05, 0.06, red), Q2 (0.1, 0.2, blue), and Q3 (0.3, 0.3, green). The position information of the three touch points W1 (7, 8), W2 (12, 22), and W3 (32, 32) is obtained in step S103. Then, in step S1041, the position information of the three styluses can be sorted according to the horizontal coordinates from small to large, and the following sorting is obtained: Q1, Q2, and Q3. In step S1042, the position information of the three touch points can be sorted according to the horizontal coordinates from small to large, and the following sorting is obtained: W1, W2, and W3. In step S1043, it is determined that the touch point W1 is generated by the stylus Q1, the touch point W2 is generated by the stylus Q2, and the touch point W3 is generated by the stylus Q3. In this way, the color of the touch track formed by the touch point W1 is red, the color of the touch track formed by the touch point W2 is blue, and the color of the touch track formed by the touch point W3 is green.

[0080] It should be noted that the above-mentioned preset position rule is to sort according to the size of the X-axis coordinate. Of course, the preset position rule can also be to sort according to the size of the Y-axis coordinate.

[0081] Optionally, the touch device provided by the embodiment of the present application can be installed with at least one operating system. The operating system can be one or more of Windows system, Android system, IOS system, and Linux system. The format of the image captured by the camera under the control of the above-mentioned operating system is NV12, and the picture size is generally 1920x1080, etc. In order to adapt to the recognition model, the image of the target region can be preprocessed first, and the preprocessing can include converting the format of the image of the target region to RGB and the size to 640x640.

[0082] It should be noted that when the camera is located above the touch device and the two styluses that are currently touching the touch panel of the touch device are arranged above and below, the stylus located below is likely to be blocked by the stylus located above, so that the image of the target region captured by the camera does not show the existence of the stylus located below, so that the position of the stylus generating a certain touch point cannot be determined. In order to avoid interruption of the program, when the position of the stylus generating a certain touch point cannot be determined, referring to FIG. 6, the color information of the historical touch track L1 in which the touch point b is located is obtained, and the color of the new touch track L2 is determined according to the color information of the historical touch track L1.

[0083] With reference to FIG. 7, the embodiment of the present application further provides a display device of a touch track, which can be used in a touch device with a touch panel. The touch panel can include a touch area, and the touch device can be integrated with a camera or connected with the camera. The display device of the touch track can include an acquisition module 410, a first processing module 420, a second processing module 430, and a third processing module 440. Specifically, the acquisition module 410 can acquire an image of a target area captured by the camera, wherein the target area includes at least part of the touch area. The first processing module 420 can determine position information and color information of a stylus located in the target area according to the image of the target area. The color information is used to represent the color of the stylus. The second processing module 430 can acquire position information of a touch point generated by the stylus touching the touch area. The third processing module 440 can determine the stylus generating the touch point according to the position information of the touch point and the position information of the stylus, and assign a corresponding color to a touch track formed by the touch point according to the color information of the stylus generating the touch point.

[0084] Optionally, when the position information and the color information of the stylus located in the target area are both multiple according to the image of the target area, and the position information of multiple touch points is acquired correspondingly, the third processing module 440 can determine the stylus generating the touch point according to the position information of the touch point and the position information of the stylus, which can include:

[0085] The third processing module 440 can sort the multiple styluses according to a preset position rule according to the position information of the multiple styluses. The third processing module 440 can also sort the multiple touch points according to the preset position rule according to the position information of the multiple touch points. The third processing module 440 can also determine that the stylus with the same sorting position as the touch point is the stylus generating the touch point.

[0086] Further, the position information of the stylus can include X-axis coordinate and Y-axis coordinate of the stylus, and the position information of the touch point can include X-axis coordinate and Y-axis coordinate of the touch point. The preset position rule is the size of the X-axis coordinate; or the preset position rule is the size of the Y-axis coordinate.

[0087] Further, when the third processing module 440 cannot determine the stylus generating the touch point, the third processing module 440 can acquire color information of a historical touch track where the touch point is located, and assign a new touch track color according to the color information of the historical touch track.

[0088] Optionally, the first processing module 420 can determine the position information and color information of the stylus in the target region according to the image of the target region, which can include:

[0089] The first processing module 420 can input the image of the target region into a recognition model and output the position information and color information of the stylus touching the touch panel in the touch region by using the recognition model. The recognition model is trained based on a YOLO series model.

[0090] Further, the first processing module 420 can input the image of the target region into a recognition model and output the position information and color information of the stylus touching the touch panel in the touch region by using the recognition model, which can include:

[0091] The first processing module 420 can obtain feature information of a plurality of feature boxes by using the image of the target region. The feature information of each feature box can include position information, target confidence information, and color confidence information. The first processing module 420 can further classify the plurality of feature boxes by color according to the color confidence information of the plurality of feature boxes. The first processing module 420 can further select an adaptive box from the same class of feature boxes. The adaptive box is a feature box that best encloses a stylus of the same color in the same class of feature boxes. The first processing module 420 can further take the position information of the adaptive box as the position information of one of the styluses touching the touch panel and take the color of the classification to which the adaptive box belongs as the color information of one of the styluses touching the touch panel.

[0092] Further, before the first processing module 420 classifies the plurality of feature boxes by color according to the color confidence information, the first processing module 420 can further include: the first processing module 420 can select a feature box with a target confidence higher than a first preset threshold according to the target confidence information of the plurality of feature boxes.

[0093] Further, before the first processing module 420 classifies the plurality of feature boxes by color according to the color confidence information, the first processing module 420 can further include: the first processing module 420 can sort the feature boxes with the target confidence higher than the first preset threshold according to the size of the target confidence.

[0094] Further, the first processing module 420 can classify the multiple feature boxes by color according to the color confidence information, which can include: the first processing module 420 can determine a main color of each feature box with a target confidence higher than a first preset threshold according to the color confidence information of each feature box with a target confidence higher than a first preset threshold. The main color is a color corresponding to the highest color confidence in the color confidence information of the feature box. The first processing module 420 can further classify the feature boxes with the same main color in the feature boxes with a target confidence higher than the first preset threshold into a same category.

[0095] Further, the first processing module 420 can select an adaptive box from the same category of feature boxes, which can include: the first processing module 420 can find a first feature box from the same category of feature boxes. The first feature box is a feature box that encloses the most same-color stylus in the same category of feature boxes. The first processing module 420 can further calculate a first intersection-over-union between the first feature box and each remaining feature box in the same category of feature boxes. The first processing module 420 can further determine that the first feature box is the adaptive box when all the first intersection-overs of the same category of feature boxes are higher than a second preset threshold.

[0096] Further, the first processing module 420 can select an adaptive box from the same category of feature boxes, which can include: the first processing module 420 can find a first feature box from the same category of feature boxes. The first feature box is a feature box that encloses the most same-color stylus in the same category of feature boxes. The first processing module 420 can further calculate a first intersection-over-union between the first feature box and each remaining feature box in the same category of feature boxes. The first processing module 420 can further determine that the first feature box is the adaptive box when all the first intersection-overs of the same category of feature boxes are higher than a second preset threshold.

[0097] Further, the first processing module 420 can find a first feature box from the same category of feature boxes, which can include: the first processing module 420 can calculate a product of a target confidence and a confidence of the main color of all feature boxes in the same category of feature boxes. The first processing module 420 can further determine the feature box with the largest product in the same category of feature boxes as the first feature box.

[0098] Further, the first processing module 420 can obtain feature information of multiple feature boxes according to the image of the target region, which can include: the first processing module 420 can obtain a scale feature map according to the image of the target region. The scale feature map includes multiple feature grids. The first processing module 420 can further assign multiple feature boxes to each feature grid and obtain feature information of each feature box.

[0099] Optionally, the first processing module 420 can further include, before obtaining the feature information of the multiple feature frames from the image of the target region, normalizing the image of the target region.

[0100] Referring to FIG. 8, the embodiments of the present application further provide a touch device, which can include a touch panel 530, a memory 520, and at least one processor 510. The memory 520 can store one or more computer instructions, which can be executed by the processor 510 to implement the display method of the touch trajectory as mentioned above.

[0101] The processor 510 is the control center of the computer device, and can connect various parts of the computer device through various interfaces and lines, and implement image processing by running or executing the instructions stored in the memory 520 and calling the data stored in the memory 520. In some embodiments, the processor 510 and the memory 520 can be implemented on the same chip, and in some embodiments, they can also be implemented on separate chips respectively.

[0102] The memory 520 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read only memory (PROM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. The memory 520 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by the computer device, but is not limited to this. The memory 520 in the embodiments of the present application can also be a circuit or other any device capable of realizing the storage function, used to store program instructions and / or data.

[0103] Optionally, the at least one processor 510 can include a NPU (Neural network Processing Unit) and a CPU (Central Processing Unit). The touch device can be installed with a recognition application (such as a whiteboard application, a system interface recognition application, etc.), which can call the NPU and the CPU to perform the following steps:

[0104] The CPU obtains an image of the target area captured by the camera, the target area including a touch area and an area within a preset range from the touch area; the NPU determines position information and color information of the stylus located in the target area according to the image of the target area; the CPU obtains position information of a touch point generated due to the stylus touching the touch panel 530 in the touch area, and determines the stylus generating the touch point according to the position information of the touch point and the position information of the stylus, and assigns a corresponding color to a touch track formed by the touch point according to the color information of the stylus generating the touch point.

[0105] The embodiments also provide a computer storage medium, which can store a plurality of instructions, the instructions being adapted to be loaded and executed by a processor to implement the above-mentioned touch track display method. The specific implementation process can refer to the specific description of the embodiment shown in FIG. 3, and will not be repeated here. The device in which the storage medium is located can be a smart interactive tablet.

[0106] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the above-mentioned touch track display method.

[0107] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0108] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0109] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0111] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0112] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, for storing files and other data to be used by one or more of the applications and / or other computer programs that are running on the computing device. The memory can also include non-volatile memory, such as read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other non-volatile memory.

[0113] The computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

[0114] Examples of storage media include, but are not limited to, a phase-change memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), another type of random access memory (RAM), a read-only memory (ROM), an electrically programmable read-only memory (EPROM), a flash memory, or other memory technology, a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD) or other optical storage, a magnetic cassette, a magnetic tape, a magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In context with the present specification, computer readable medium does not encompass transitory media, such as modulated data signals and carrier waves.

[0115] It is also to be noted that the term "comprising" or "including" or any other variation thereof is intended to cover a non-exclusive inclusion, such that processes, methods, articles, or apparatuses that comprise a list of elements are not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such processes, methods, articles, or apparatuses.

[0116] The embodiments of the present application as described above are merely used to illustrate the present application and should not be used to limit the present application. Any modifications, equivalent replacements, improvements and the like made by those skilled in the art based on the principles of the present application shall fall within the scope of claims of the present application.

Claims

A method for displaying touch traces, used in a touch device with a touch panel, the touch panel including a touch area, the touch device integrating a camera or having an external camera, characterized in that... The method includes: Acquire an image of a target area captured by the camera, the target area including at least a portion of the touch area; Based on the image of the target area, determine the position information and color information of the stylus located in the target area, wherein the color information is used to characterize the appearance color of the stylus; Obtain the position information of the touch point generated when the stylus touches the touch area; Based on the position information of the touch point and the position information of the stylus, the stylus that generated the touch point is determined, and based on the color information of the stylus that generated the touch point, the touch trajectory formed by the touch point is assigned a corresponding color. The method according to claim 1, characterized in that, When, based on the image of the target area, it is determined that there are multiple positions and colors of the stylus located in the target area, and correspondingly, the position information of multiple touch points is obtained, determining the stylus that generated the touch point based on the position information of the touch point and the position information of the stylus includes: Based on the position information of the multiple styluses, the multiple styluses are sorted according to a preset position rule; Based on the position information of the multiple touch points, the multiple touch points are sorted according to the preset position rules; The stylus that has the same sorting position as the touch point is identified as the stylus that generates the touch point. The method according to claim 2, characterized in that, The position information of the stylus includes the X-axis coordinate and Y-axis coordinate of the stylus, and the position information of the touch point includes the X-axis coordinate and Y-axis coordinate of the touch point; The preset positional pattern is based on the order of X-axis coordinates; or, the preset positional pattern is based on the order of Y-axis coordinates. The method according to claim 3, characterized in that, When the stylus that generated the touch point cannot be determined, the color information of the historical touch trajectory where the touch point is located is obtained, and a new touch trajectory color is assigned according to the color information of the historical touch trajectory. The method according to any one of claims 1-4, characterized in that, The step of determining the position and color information of the stylus located in the target area based on the image of the target area includes: The image of the target area is input into the recognition model, and the recognition model is used to output the position and color information of the stylus that is touching the touch panel in the touch area; the recognition model is trained based on the YOLO series models. The method according to claim 5, characterized in that, The step of inputting the image of the target area into the recognition model and using the recognition model to output the position and color information of the stylus touching the touch panel in the touch area includes: The feature information of multiple feature boxes is obtained using the image of the target region; the feature information of each feature box includes location information, target confidence information, and color confidence information; Based on the color confidence information of the multiple feature boxes, the multiple feature boxes are classified according to color; Filter out the matching boxes from the same type of feature boxes; wherein, the matching box is the feature box in the same type of feature box that can enclose the stylus of the same color; The position information of the adapter frame is used as the position information of one of the styluses that is touching the touch panel, and the color of the category to which the adapter frame belongs is used as the color information of one of the styluses that is touching the touch panel. The method according to claim 6, characterized in that, Before classifying the multiple feature boxes by color based on the color confidence information, the method further includes: Based on the target confidence information of the multiple feature boxes, feature boxes with a target confidence level higher than a first preset threshold are selected. The method according to claim 7, characterized in that, Before classifying the multiple feature boxes by color based on the color confidence information, the method further includes: The feature boxes with a target confidence level higher than a first preset threshold are sorted according to the magnitude of the target confidence level. The method according to claim 7, characterized in that, The step of classifying multiple feature boxes by color based on the color confidence information includes: Based on the color confidence information of each feature box whose target confidence level is higher than a first preset threshold, the main color of each feature box whose target confidence level is higher than the first preset threshold is determined; the main color is the color corresponding to the highest color confidence level in the color confidence information of the feature box. Feature boxes with the same main color are grouped into one category among feature boxes whose target confidence level is higher than the first preset threshold. The method according to claim 9, characterized in that, The step of filtering fitting boxes from the same type of feature boxes includes: Find the first feature box from the same type of feature boxes; the first feature box is the feature box that encloses the stylus of the same color among the same type of feature boxes. Calculate the first intersection-union ratio (IU) between the first feature box and each of the remaining feature boxes in the same category; When all first intersection-union ratios in the same type of feature box are higher than the second preset threshold, the first feature box is the fitting box. The method according to claim 10, characterized in that, The step of filtering fitting boxes from the same type of feature boxes also includes: When at least one small-ratio feature box exists within the same type of feature box, a second feature box is found from the at least one small-ratio feature box, and a second intersection-union ratio (IUU) is calculated between the second feature box and each of the remaining small-ratio feature boxes within the same type of feature box; wherein, the small-ratio feature box is the feature box within the same type of feature box whose first IUU ratio is not higher than the second preset threshold; the second feature box is the feature box within the at least one small-ratio feature box that can contain the same color stylus. When all second intersection-union ratios in the same type of feature box are higher than the second preset threshold, both the first feature box and the second feature box are the fitting boxes. The method according to claim 10, characterized in that, Finding the first feature box from the same type of feature boxes includes: Calculate the product of the target confidence of all feature boxes in the same category and the confidence of the main color; The feature box with the largest product among the same type of feature boxes is selected as the first feature box. The method according to any one of claims 6-12, characterized in that, The feature information obtained from the image of the target region includes: A scale feature map is obtained from the image of the target region; the scale feature map includes multiple feature grids. Multiple feature boxes are assigned to each feature grid, and feature information for each feature box is obtained. The method according to any one of claims 1-4, characterized in that, Before obtaining feature information of multiple feature boxes using the image of the target region, the method further includes: The image of the target region is normalized. A touch trajectory display device for a touch device having a touch panel, the touch panel having a touch area, the touch device having an integrated camera or an external camera, characterized in that... The device includes: The acquisition module is used to acquire an image of a target area captured by the camera, wherein the target area includes at least a portion of the touch area; The first processing module determines the position and color information of the stylus located in the target area based on the image of the target area. The second processing module is used to obtain the position information of the touch point generated by the stylus touching the touch panel in the touch area; The third processing module is used to determine the stylus that generated the touch point based on the position information of the touch point and the position information of the stylus, and to assign a corresponding color to the touch trajectory formed by the touch point based on the color information of the stylus that generated the touch point. A touch device, characterized in that, The device includes a touch panel, a memory, and at least one processor; wherein the memory is used to store one or more computer instructions, which are executed by the processor to implement the method as described in any one of claims 1-14. The touch device according to claim 16 is characterized in that, The at least one processor includes an NPU and a CPU; the touch device is equipped with a recognition application, which invokes the NPU and the CPU to perform the following steps: The CPU acquires an image of the target area captured by the camera. The target area includes a touch area and an area within a preset range from the touch area. The NPU determines the position and color information of a stylus located in the target area based on the image of the target area. The CPU acquires the position information of a touch point generated by the stylus touching the touch panel in the touch area, and determines the stylus that generated the touch point based on the position information of the touch point and the position information of the stylus. Based on the color information of the stylus that generated the touch point, the CPU assigns a corresponding color to the touch trajectory formed by the touch point. A computer-readable storage medium having computer instructions stored thereon, characterized in that, When executed by a processor, the computer instructions implement the method as described in any one of claims 1-14.

Images