Touch detection method, device, equipment and computer readable storage medium

Abstract

The present application discloses a touch detection method, device, equipment and computer-readable storage medium, the method comprising: obtaining a first image and a second image captured on a touch detection plane; performing target detection on the first image and the second image respectively, obtaining a first image position of a target part of a target object in the first image and a second image position in the second image respectively; performing a joint analysis on the first image position and the second image position to obtain a touch detection result; wherein the touch detection result includes whether the target part touches the touch detection plane. The above scheme can improve the accuracy of the touch detection result and has a low cost.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a touch detection method, device, equipment and computer-readable storage medium. Background Art

[0002] At present, most fingertip touch detection technologies or products require matching pressure sensors. The layout range and density of the pressure sensors determine the touch effect. If large-area touch detection is required, large-sized pressure sensors need to be configured, which greatly limits the usage scenarios.

[0003] In addition, vision-based touch solutions in general scenarios require the configuration of special function devices such as depth cameras and structured light cameras, which increases the difficulty and cost of use. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a touch detection method, device, equipment and computer-readable storage medium, which can improve the accuracy of touch detection results and have low cost.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a touch detection method, including: acquiring a first image and a second image captured of a touch detection plane; performing target detection on the first image and the second image respectively to obtain a target part of the target object at a first image position in the first image and a second image position in the second image respectively; performing a joint analysis on the first image position and the second image position to obtain a touch detection result; wherein the touch detection result includes whether the target part touches the touch detection plane.

[0006] Among them, for the first image position, a first area image is extracted from the first image, and based on the second image position, a second area image is extracted from the second image; based on the first pixel point belonging to the target object in the first area image, a first optimized position of the target part in the first image is obtained, and based on the second pixel point belonging to the target object in the second area image, a second optimized position of the target part in the second image is obtained; the first optimized position and the second optimized position are jointly analyzed to obtain a touch detection result.

[0007] Wherein, the target part is located at the end of the target object, and the step of obtaining the first optimized position or the second optimized position includes: performing target segmentation on the regional image to obtain a mask image of the target object; wherein the pixel values ​​of the pixels belonging to the target object in the mask image are all first values; based on the mask image, the center line and the edge of the target object are obtained; based on the intersection of the center line and the edge, the optimized position of the target object is obtained; wherein, when the regional image is the first regional image, the optimized position is the first optimized position, and when the regional image is the second regional image, the optimized position is the second optimized position.

[0008] The first region image and the second region image have the same size, and the relative position of the first image and the relative position of the second image and the second region image are the same.

[0009] The first optimized position and the second optimized position are jointly analyzed to obtain a touch detection result, including: extracting a first reference image from the first image based on the first optimized position, and extracting a second reference image from the second image based on the second optimized position; wherein the first reference image and the second reference image have the same size; dividing the first reference image and the second reference image into a preset number of first sub-images and a preset number of second sub-images along the same direction, respectively, and forming a group of sub-image pairs with the first sub-image and the second sub-image arranged in the same order; respectively obtaining the coordinate difference of the matching point pairs in each group of sub-image pairs; wherein the matching point pairs include a first feature point from the first sub-image and a second feature point from the second sub-image; and obtaining the touch detection result based on the coordinate difference of each matching point pair.

[0010] Among them, based on the coordinate difference of each matching point pair, the touch detection result is obtained, including: counting the coordinate difference of each matching point pair to obtain a final difference; when the final difference is lower than a preset threshold, determining that the touch detection result includes that the target part touches the touch detection plane; when the final difference is not lower than the preset threshold, determining that the touch detection result includes that the target part does not touch the touch detection plane.

[0011] Wherein, respectively obtaining the coordinate difference of the matching point pair in each group of sub-image pairs includes: obtaining the first pixel coordinates and the second pixel coordinates of the first feature point and the second feature point in the matching point pair, respectively in the first reference image and the second reference image; obtaining the coordinate difference of the matching point pair based on the absolute value of the difference between the first pixel coordinate and the second pixel coordinate.

[0012] Among them, neither the first reference image nor the second reference image contains the target object; the relative position of the first optimized position and the first reference image, and the relative position of the second optimized position and the second reference image are the same; and / or the first reference image and the second reference image are equally divided along the first direction, and the coordinate difference is obtained based on the coordinate difference of the first feature point and the second feature point in the second direction, respectively, and the first direction is perpendicular to the second direction.

[0013] The first image is captured by the first camera, and the second image is captured by the second camera; a line connecting the optical centers of the first camera and the second camera is parallel to the touch detection plane, and / or the vertical distances from the optical centers of the first camera and the second camera to the touch detection plane are the same.

[0014] In order to solve the above technical problems, the second aspect of the present application provides a touch detection device, including: an image acquisition module, used to acquire a first image and a second image captured of a touch detection plane; a target detection module, used to perform target detection on the first image and the second image, respectively, to obtain a target part of the target object at a first image position in the first image and a second image position in the second image, respectively; a joint analysis module, used to perform a joint analysis on the first image position and the second image position to obtain a touch detection result; wherein the touch detection result includes whether the target part touches the touch detection plane.

[0015] In order to solve the above technical problems, the third aspect of the present application provides a touch detection device, including a memory and a processor coupled to each other, the memory stores program instructions, and the processor is used to execute the program instructions to implement the touch detection method in the above first aspect.

[0016] The touch detection device further includes a first camera and a second camera, both of which are coupled to the processor, the first camera is used to photograph the touch detection plane to obtain a first image, and the second camera is used to photograph the touch detection plane to obtain a second image.

[0017] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, wherein the program instructions are used to implement the touch detection method in the above first aspect.

[0018] The above scheme can determine whether the target part of the target object is touching the touch detection plane or in a hovering state by jointly analyzing the target part of the target object at the first image position in the first image and the second image position in the second image. Compared with the traditional technical solution of touch detection based on pressure sensors, the implementation of this application does not require the configuration of pressure sensors, has a wider range of applications and lower costs; and compared with the technical solution of determining whether the fingertip touches the touch detection plane by comparing the depth information difference between the fingertip and the touch detection plane, this application does not require the configuration of a depth camera with highly integrated hardware and software packaging, which is simple, reliable, and more versatile and flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of an embodiment of a touch detection method provided by the present application;

[0020] Figure 2 is a schematic diagram of the position of a binocular camera according to an embodiment of the present application;

[0021] Figure 3 It is a structural schematic diagram of an embodiment of a fingertip detection model provided by the present application;

[0022] Figure 4 yes Figure 1 The flowchart of step S13 is shown as an embodiment;

[0023] Figure 5 is a schematic diagram of optimizing the first image position and the second image position provided by the present application;

[0024] Figure 6 is a schematic diagram of a process for obtaining a first optimized position or a second optimized position provided by the present application;

[0025] Figure 7 is a schematic diagram of an embodiment of a target object skeleton extraction algorithm provided by the present application;

[0026] Figure 8 is a schematic diagram of an embodiment of extracting feature matching points provided by the present application;

[0027] Fig. 9 yes Figure 4 The flowchart of step S133 is shown as an embodiment;

[0028] Fig.10 It is a flowchart of an embodiment of obtaining coordinate differences of matching point pairs in each group of sub-image pairs provided by the present application;

[0029] Fig.11 It is a flowchart of an embodiment of obtaining a touch detection result based on the coordinate difference of each matching point pair provided by the present application;

[0030] Fig.12 It is a schematic diagram of a framework of an embodiment of a touch detection device provided by the present application;

[0031] Fig.13 It is a schematic diagram of the framework of an embodiment of a touch detection device provided by the present application;

[0032] Fig.14 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0033] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0034] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0035] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0036] See also Figure 1 , Figure 1 is a flow chart of an embodiment of the touch detection method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0037] Step S11: Acquire a first image and a second image captured on the touch detection plane.

[0038] The method of this embodiment is used to detect whether the target part of the target object touches the touch detection plane. The touch detection plane described in this article can be but is not limited to the display plane of the device (for example, the display screen of a learning machine), newspapers, journals, magazines, textbooks, etc.

[0039] In one embodiment, the first image and the second image captured by the touch detection plane may be obtained by capturing the touch detection plane by a binocular camera. For example, the first image captured by the binocular camera is an image captured by a left camera in the binocular camera, and the second image is an image captured by a right camera in the binocular camera. Of course, the first image may also be an image captured by a right camera in the binocular camera, and the second image may also be an image captured by a left camera in the binocular camera. In other embodiments, two or more cameras may be used to capture the touch detection plane to obtain the first image and the second image.

[0040] In one embodiment, the first image and the second image captured on the touch detection plane are obtained, which can be obtained from local storage or cloud storage. It can be understood that in other embodiments, the first image and the second image can also be obtained by capturing the current touch detection plane with a binocular camera or two cameras or multiple cameras.

[0041] See also Figure 2 , Figure 2: is a schematic diagram of the position of an embodiment of a binocular camera provided in the present application. In order to enable subsequent touch detection based on the first image and the second image captured by the binocular camera on the touch detection plane, in one embodiment, the line connecting the optical centers of the two cameras in the binocular camera is parallel to the touch detection plane, that is, the left camera C L and right camera C R The line D between the optical centers of the two cameras is parallel to the touch detection plane, wherein the parameters of the two cameras are set the same, and the distance between the optical centers of the two cameras is not limited, and can be specifically set according to actual use needs. In other embodiments, the vertical distances from the optical centers of the two cameras in the binocular camera to the touch detection plane are the same, that is, the left camera C L The vertical distance H from the optical center to the touch detection plane 1 With right camera C R The vertical distance H from the optical center to the touch detection plane 2 The same, wherein the vertical distance from the optical center of the camera to the touch detection plane is not limited and can be specifically set according to actual use needs. In other embodiments, the line connecting the optical centers of the two cameras in the binocular camera is parallel to the touch detection plane and the vertical distances from the optical centers of the two cameras in the binocular camera to the touch detection plane are the same.

[0042] In other embodiments, if the first image and the second image are obtained by photographing the touch detection plane using a dual camera, i.e., a first camera and a second camera, the first image is photographed by the first camera, and the second image is photographed by the second camera. Then, the line connecting the optical centers of the first camera and the second camera is parallel to the touch detection plane. Alternatively, in other embodiments, the vertical distances from the optical centers of the first camera and the second camera to the touch detection plane are the same. Alternatively, in other embodiments, the line connecting the optical centers of the first camera and the second camera is parallel to the touch detection plane and the vertical distances from the optical centers of the first camera and the second camera to the touch detection plane are the same.

[0043] Step S12: performing target detection on the first image and the second image respectively, and obtaining the first image position of the target part of the target object in the first image and the second image position in the second image respectively.

[0044] In one embodiment, the target object includes but is not limited to a finger, an ordinary pen (eg, a pencil, a fountain pen, a fountain pen, etc.), a stylus pen (eg, a reading pen provided with a reading machine), etc.

[0045] In one embodiment, the target part of the target object may be located at the end of the target object. For example, when the target object is a finger, the target part of the target object may be the fingertip; when the target object is an ordinary pen, the target part of the target object may be the tip of the pen; when the target object is a stylus pen, the target part of the target object may be the tip of the stylus pen, wherein a marker (e.g., a red or green dot, etc.) for easy identification and detection may be set at the tip of the stylus pen, so as to facilitate the subsequent determination of the position of the stylus pen tip. It can be understood that in other embodiments, the target part of the target object may also be the tail of the target object, etc., which is not limited here and can be specifically set according to actual use needs.

[0046] In this embodiment, target detection is performed on the first image and the second image respectively, and the target part of the target object is obtained at the first image position of the first image and the second image position of the second image respectively. The algorithm for performing target detection on the first image and the second image is not limited and can be specifically set according to actual use needs.

[0047] In one embodiment, taking the target part of the target object as the fingertip as an example, the deep neural network can be trained by collecting image data labeled with the fingertip points, so that the trained deep neural network can detect the fingertip.

[0048] In order to help improve the accuracy of the deep neural network in detecting fingertips, in other embodiments, in addition to the fingertip positions, the image data used for training the deep neural network may also be marked with the hand center position, the hand area height, i.e., the height and width of the rectangular area surrounding the hand, or the hand center position after optimization and fine-tuning. In a specific embodiment, Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of an embodiment of the fingertip detection model provided by the present application. The image data used for deep neural network training is annotated with the fingertip position, hand center position, hand area height and hand center position after optimization and fine-tuning. Specifically, a deep residual network (ResNet) is used for feature extraction, and then a feature pyramid network (FPN) structure is used for sampling and the features are input into four branches to complete the corresponding detection tasks, thereby realizing the training of the deep neural network model. It should be noted that, if Figure 3 The four branches shown can all be composed of a convolutional layer (Conv) and a normalization layer (BN). Of course, the network parameters of different branches are also different, which will not be repeated here.

[0049] Step S13: jointly analyzing the first image position and the second image position to obtain a touch detection result.

[0050] In this embodiment, the target part of the target object is jointly analyzed at the first image position in the first image and the second image position in the second image, respectively, to obtain a touch detection result. The touch detection result includes whether the target part touches the touch detection plane. In other words, by jointly analyzing the first image position and the second image position, it can be determined whether the target part of the target object touches the touch detection plane or is in a hovering state. Compared with the traditional technical solution of touch detection based on pressure sensors, the implementation of the present application does not require the configuration of pressure sensors, has a wider range of applications and lower costs; compared with the technical solution of determining whether the fingertip touches the touch detection plane by comparing the depth information difference between the fingertip and the touch detection plane, the present application does not require the configuration of a depth camera with highly integrated hardware and software packaging, which is simple, reliable, and more versatile and flexible.

[0051] Due to slight differences between the camera and the current environment, the first image position and the second image position obtained according to the above steps may not be synchronized. Therefore, in order to improve the accuracy of the touch detection result, in one embodiment, the target part of the target object can be optimized at the first image position in the first image and the second image position in the second image, respectively. For example, the first image position and the second image position are synchronized to ensure that the first image position of the target part of the target object in the first image and the second image position in the second image are accurate and consistent, thereby performing a joint analysis based on the optimized first image position and the optimized second image position to obtain the touch detection result.

[0052] In one embodiment, the joint analysis of the first image position and the second image position may be to intercept the preset direction areas of the first image position and the second image position respectively and perform matching feature point extraction, and determine the touch detection result based on the error of the horizontal position of the matching point. Among them, the matching feature point extraction algorithm includes but is not limited to the scale-invariant feature transform (SIFT), the accelerated robust features (SURF), the FAST (Features From Accelerated Segment Test) algorithm, etc. Among them, the preset direction area is not specifically limited, and can be specifically set according to actual use needs. For example, the preset direction area is a preset size area above the first image position and the second image position.

[0053] Since the extracted matching feature points may be concentrated in areas with drastic color changes, the diversity of the sampling points may be insufficient, which in turn affects the accuracy of the subsequent touch detection results. In order to improve the accuracy of the touch detection results, in one embodiment, the captured preset direction area can be divided into a preset number of sub-image areas and form sub-image pairs, and matching feature points are extracted for each sub-image pair. Finally, the touch detection result is obtained based on the coordinate difference of each matching feature point pair.

[0054] In the above implementation, by jointly analyzing the target part of the target object at the first image position in the first image and the second image position in the second image, it is possible to determine whether the target part of the target object is touching the touch detection plane or in a hovering state. Compared with the traditional technical solution of touch detection based on pressure sensors, the implementation of this application does not require the configuration of pressure sensors, has a wider range of applications, and has lower costs; and compared with the technical solution of determining whether the fingertips touch the touch detection plane by comparing the depth information difference between the fingertips and the touch detection plane, this application does not require the configuration of a depth camera with highly integrated hardware and software packaging, which is simple, reliable, and more versatile and flexible.

[0055] See also Figure 4 , Figure 4 yes Figure 1 It should be noted that if there is substantially the same result, this embodiment does not use Figure 4 The process sequence shown is limited. Figure 4 As shown, in this embodiment, the target part of the target object is optimized at the first image position in the first image and the second image position in the second image, respectively, and a joint analysis is performed based on the optimized first image position and the second image position to obtain a touch detection result, specifically including:

[0056] Step S131: extracting a first region image from the first image based on the first image position, and extracting a second region image from the second image based on the second image position.

[0057] Due to slight differences between the camera and the current environment, the first image position and the second image position obtained according to the above steps may be out of sync, which may lead to inaccurate subsequent touch detection results. Therefore, in order to improve the accuracy of the touch detection results, in this embodiment, the target part of the target object can be optimized at the first image position in the first image and the second image position in the second image. Specifically, first, based on the first image position, the first area image is extracted from the first image, and based on the second image position, the second area image is extracted from the second image.

[0058] In one embodiment, the first region image and the second region image have the same size, and the relative position of the first image position to the first region image and the relative position of the second image position to the second region image are the same. The specific sizes of the first region image and the second region image are not limited and can be set according to actual use needs. For example, the center of the first region image can be the first image position, the center of the second region image can be the second image position, and the first region image and the second region image have the same size, such as the first region image and the second region image are rectangular regions with a height of H and a width of W.

[0059] For example, if Figure 5 As shown, Figure 5 is a schematic diagram of optimizing the first image position and the second image position provided by the present application. The first image position obtained by performing target detection on the first image is (x' L ,y' L ), the second image position obtained by performing target detection on the second image is (x' R ,y' R ), since the first image position (x' L ,y' L ) and the second image position (x' R ,y' R ) may lead to inaccurate subsequent touch detection results, so the first image position (x' L ,y' L ) and the second image position (x' R ,y' R ) is optimized. Specifically, the first image position (x' L ,y' L ) is the center point, and the first region image with a height of H and a width of W is extracted from the first image, that is, Figure 5 and the second image position (x' R ,y' R ) is taken as the center point, and a second region image with a height of H and a width of W is extracted from the second image, that is, Figure 5 "Right Eye Intercept Rectangle" in the navigation pane.

[0060] Step S132: Based on the first pixel point belonging to the target object in the first area image, obtain the first optimized position of the target part in the first image, and based on the second pixel point belonging to the target object in the second area image, obtain the second optimized position of the target part in the second image.

[0061] In this embodiment, the first optimized position of the target part in the first image is obtained based on the first pixel point belonging to the target object in the first area image, and the second optimized position of the target part in the second image is obtained based on the second pixel point belonging to the target object in the second area image.

[0062] Among them, Figure 6 As shown, Figure 6 : is a schematic diagram of a process for obtaining a first optimized position or a second optimized position provided in the present application. The obtaining of the first optimized position or the second optimized position specifically includes the following sub-steps:

[0063] Step S61: performing target segmentation on the region image to obtain a mask image of the target object.

[0064] In this embodiment, the target part is located at the end of the target object, and the regional image is segmented to obtain a mask image of the target object. That is to say, the regional image is segmented so that the target object and the background area can be separated, thereby facilitating the subsequent determination of the optimal position of the target object. Among them, the algorithm for performing target segmentation is not limited, and can be specifically set according to actual use needs. For example, Roberts algorithm, Sobel algorithm, Canny edge detection algorithm, Marr-Hildreth edge detection algorithm, etc., or it can also be implemented through target segmentation networks such as U-Net, MaskRCNN, etc., which are not limited here.

[0065] The pixel values ​​of the pixels belonging to the target object in the mask image are all the first value, and the pixel values ​​of other pixels in the mask image are other values ​​except the first value, so that the target object and the background area are separated to obtain the mask image of the target object. The first value is not specifically limited and can be specifically set according to actual use needs. For example, the first value is 255.

[0066] For example, if Figure 5 As shown, taking the target object as a finger as an example, the "left eye interception rectangle" and the "right eye interception rectangle" are segmented to obtain a mask image of the finger, that is, a binary image. The pixel value of the pixel point of the finger in the mask image is 255, and the pixel value of the pixel point of the background area in the mask image is 0, thereby separating the finger and the background area.

[0067] Step S62: Based on the mask image, obtain the center line and edge of the target object.

[0068] In this embodiment, the center line and edge of the target object can be obtained according to the mask image of the target object.

[0069] In one embodiment, the mask image of the target object may be processed using an image erosion method to filter out invalid pixels at the edge of the target object, thereby obtaining the edge of the target object, that is, the remaining pixels belonging to the target object in the mask image are close to the real skeleton of the target object; further, the center line of the target object is generated based on the obtained edge of the target object. Of course, in other embodiments, other methods may be used to process the mask image of the target object to obtain the edge and center line of the target object.

[0070] Specifically, Figure 7 As shown, Figure 7 This is a schematic diagram of an embodiment of the target object skeleton extraction algorithm provided by the present application. "3 connected pixels" means that the x pixel has 8 adjacent pixels, among which the three pixels a / b / c form a connected domain. According to the skeleton extraction algorithm, x and its three adjacent pixels a / b / c should be eroded and deleted, and the 8-pixel neighborhood of the target object edge contour is detected in turn until no new pixel points of the target object edge contour are eroded. Figure 7 The algorithm shown processes the edge contour points of the target object, and the specific process includes the following: extracting the edge contour points of the target object and recording them; successively detecting whether the 8-pixel neighborhood of these edge contour points contains only 3 connected pixels, if it contains only 3 connected pixels, then deleting this edge contour point and deleting the corresponding pixel point in the target object; successively detecting whether the 8-pixel neighborhood of the remaining edge contour points in the previous step contains only 3 or 4 connected pixels, if it contains only 3 or 4 connected pixels, then deleting this edge contour point and deleting the corresponding pixel point in the target object; successively detecting whether the 8-pixel neighborhood of the remaining edge contour points in the previous step contains only 3 or 4 or 5 connected pixels, if it contains only 3 or 4 or 5 connected pixels, then deleting this edge contour point and deleting the corresponding pixel point in the target object. Delete the contour point and delete the corresponding pixel point in the target object; check whether the 8-pixel neighborhood of the remaining edge contour point in the previous step contains only 3 or 4 or 5 or 6 connected pixels. If it contains only 3 or 4 or 5 or 6 connected pixels, delete this edge contour point and delete the corresponding pixel point in the target object; check whether the 8-pixel neighborhood of the remaining edge contour point in the previous step contains only 3 or 4 or 5 or 6 or 7 connected pixels. If it contains only 3 or 4 or 5 or 6 or 7 connected pixels, delete this edge contour point and delete the corresponding pixel point in the target object until the 8-pixel area of ​​the edge contour point of the target object has 8 connected pixels. The remaining edge contour points of the target object finally obtained are the skeleton points (x 1 ,y 1 ),(x 2 ,y 2 ),...,(x n ,y n ).

[0071] Furthermore, in one embodiment, the skeleton points of the target object are processed using a least squares fitting algorithm to obtain the skeleton line of the target object, that is, the center line of the target object. The specific formula is as follows:

[0072]

[0073] Wherein, k is the slope of the center line of the target object; b is the intercept of the center line of the target object.

[0074] like Figure 5 As shown, the skeleton points of the target object are processed using the least squares fitting algorithm, and the center line corresponding to the finger in the "left eye intercept rectangle" is obtained as y=k L x+b L The center line of the finger in the "right eye intercept rectangle" is y=k R x+b R It is understandable that in other implementations, other algorithms may be used to process the skeleton points of the target object to obtain the center line of the target object, which is not specifically limited here.

[0075] Step S63: based on the intersection of the center line and the edge, obtain the optimized position of the target object.

[0076] In this embodiment, the optimized position of the target object is obtained according to the intersection of the center line and the edge of the target object. Specifically, a point-by-point comparison is performed along the center line direction of the target object. Specifically, the pixel values ​​of the center line and the edge can be compared. The intersection of the pixel value of the center line and the pixel value of the edge is the intersection, and the intersection is the optimized position of the target object.

[0077] For example, if Figure 5 As shown in the figure, the target object is a finger, and the target part of the target object is the fingertip point. The center line of the finger, i.e., the skeleton line, is compared point by point to find the intersection of the skeleton line of the finger and the edge of the finger, i.e., the intersection of 0 and 255 in the binary image. Specifically, Figure 5 As shown in (x L ,y L ) and (x R ,y R ), so (x L ,y L ) and (x R ,y R ) is the optimized position of the finger tip. Compared with the first image position and the second image position obtained by target detection, Figure 5 As shown in (x' L ,y' L ) and (x' R ,y' R ), the first optimal position and the second optimal position are Figure 5 As shown in (x L ,y L ) and (x R ,y R ) is more accurate and consistent, so that the subsequent touch detection result obtained by joint analysis based on the first optimized position and the second optimized position is more accurate.

[0078] Step S133: jointly analyzing the first optimized position and the second optimized position to obtain a touch detection result.

[0079] In this embodiment, the first optimized position of the target part of the target object in the first image and the second optimized position in the second image are jointly analyzed to obtain the touch detection result. That is, by jointly analyzing the first optimized position and the second optimized position, it can be determined whether the target part of the target object is touching the touch detection plane or in a hovering state.

[0080] In one embodiment, the joint analysis of the first optimized position and the second optimized position can be performed by respectively intercepting the preset direction areas of the first optimized position and the second optimized position and performing matching feature point extraction, and determining the touch detection result based on the error of the horizontal position of the matching point. Among them, the matching feature point extraction algorithm includes but is not limited to the scale-invariant feature transform (SIFT), the accelerated robust features (SURF), the FAST (Features From Accelerated Segment Test) algorithm, etc. Among them, the preset direction area is not specifically limited, and can be specifically set according to actual use needs. For example, the preset direction area is a preset size area above the first optimized position and the second optimized position.

[0081] See also Figure 8 , Figure 8 Schematic diagram of an embodiment of extracting feature matching points provided by the present application. Figure 8 As shown in area a in the figure, since the extracted matching feature points may be concentrated in the area with drastic color changes, the diversity of the sampling points is not rich enough, which in turn affects the accuracy of the subsequent touch detection results. Therefore, in order to improve the accuracy of the touch detection results, in one embodiment, as shown in FIG. Fig. 9 As shown, Fig. 9 yes Figure 4 The flowchart of step S133 of an embodiment is shown as follows. In this embodiment, the captured preset direction area can be equally divided into a preset number of sub-image areas and form sub-image pairs, and matching feature points are extracted for each sub-image pair, specifically including the following sub-steps:

[0082] Step S1331: extracting a first reference image from the first image based on the first optimized position, and extracting a second reference image from the second image based on the second optimized position.

[0083] In this embodiment, a first reference image is extracted from the first image according to the first optimized position, and a second reference image is extracted from the second image according to the second optimized position. The first reference image and the second reference image have the same size so that a touch detection result can be obtained by comparison based on the same standard.

[0084] In one embodiment, the first reference image is a preset size area above the first optimized position, and the second reference image is a preset size area above the second optimized position, that is, the first reference image and the second reference image do not contain the target object. In order to ensure that the touch detection result can be obtained by comparison based on the same standard in the future, the relative position of the first optimized position and the first reference image, and the relative position of the second optimized position and the second reference image are the same.

[0085] Alternatively, if Figure 8 As shown, in one embodiment, the first reference image and the second reference image are rectangular areas with a height of h and a width of w above the corresponding optimization position, and the first reference image is Figure 8 The second reference image is the "left eye rectangular area" shown in Figure 8 The "right eye rectangular area" shown. The height h and width w are not limited and can be set according to actual use needs. It can be understood that in other embodiments, the first reference image and the second reference image can also be areas of other shapes above the corresponding optimization position, which are not specifically limited here.

[0086] Step S1332: Divide the first reference image and the second reference image equally into a preset number of first sub-images and a preset number of second sub-images along the same direction, and group the first sub-images and the second sub-images at the same sequence into a group of sub-image pairs.

[0087] In this implementation, the first reference image and the second reference image are equally divided into a preset number of first sub-images and a preset number of second sub-images along the same direction, and the first sub-image and the second sub-image arranged at the same sequence are combined into a group of sub-image pairs. The direction for equally dividing the first reference image and the second reference image and the preset value for equally dividing are not limited and can be specifically set according to actual use needs.

[0088] In one embodiment, the first reference image and the second reference image are equally divided along a first direction, wherein the first direction is not specifically limited. For example, the first direction may be along the height direction of the first reference image and the second reference image, or along the width direction of the first reference image and the second reference image, etc.

[0089] For example, if Figure 8 As shown, the first direction for equally dividing the first reference image and the second reference image is the height h direction of the first reference image and the second reference image. Specifically, the first reference image, i.e., the "left-eye rectangular area", is equally divided into four first sub-images L1, L2, L3 and L4, and the second reference image, i.e., the "right-eye rectangular area", is equally divided into four second sub-images R1, R2, R3 and R4. Among them, L1 and R1 form a sub-image pair, L2 and R2 form a sub-image pair, L3 and R3 form a sub-image pair, and L4 and R4 form a sub-image pair.

[0090] Step S1333: respectively obtain the coordinate differences of the matching point pairs in each group of sub-image pairs.

[0091] In this embodiment, the coordinate difference of the matching point pairs in each group of sub-image pairs is obtained respectively, so that the touch detection result can be obtained based on the coordinate difference of each matching point pair later. The matching point pair includes a first feature point from the first sub-image and a second feature point from the second sub-image.

[0092] In one embodiment, the coordinate difference is obtained based on the coordinate difference of the first feature point and the second feature point in the second direction, and the first direction is perpendicular to the second direction. Optionally, when the first direction is the height direction of the first reference image and the second reference image, the second direction is the width direction of the first reference image and the second reference image.

[0093] Among them, Fig.10 As shown, Fig.10 1 is a flow chart of an embodiment of obtaining the coordinate difference of the matching point pair in each group of sub-image pairs provided by the present application. Obtaining the coordinate difference of the matching point pair in each group of sub-image pairs specifically includes the following sub-steps:

[0094] Step S101: obtaining the first pixel coordinates and the second pixel coordinates of the first feature point and the second feature point in the matching point pair in the first reference image and the second reference image respectively.

[0095] In this embodiment, the first pixel point coordinates of the first feature point in the matching point pair in the first reference image are obtained, and the second pixel point coordinates of the second feature point in the matching point pair in the second reference image are obtained.

[0096] For example, if Figure 8As shown, there are 3 pairs of matching points in each group of sub-image pairs, and the coordinates of the first pixel point of the first feature point in the matching point pair in the first reference image are (x Lij ,y Lij ), the coordinates of the second pixel point of the second feature point in the matching point pair in the second reference image are (x Rij ,y Rij ).

[0097] Step S102: obtaining a coordinate difference of a matching point pair based on the absolute value of the difference between the first pixel coordinate and the second pixel coordinate.

[0098] In this implementation, the coordinate difference of the matching point pair is obtained according to the absolute value of the difference between the first pixel coordinate and the second pixel coordinate.

[0099] In one embodiment, the absolute value of the difference between the first pixel horizontal coordinate and the second pixel horizontal coordinate can be used as the coordinate difference of the matching point pair. It can be understood that in other embodiments, the absolute value of the difference between the first pixel vertical coordinate and the second pixel vertical coordinate can also be used as the coordinate difference of the matching point pair, or the absolute value of the difference between the horizontal coordinates and the absolute value of the difference between the vertical coordinates can be weighted to obtain the coordinate difference.

[0100] For example, the absolute value of the difference between the first pixel horizontal coordinate and the second pixel horizontal coordinate is taken as the coordinate difference of the matching point pair. The horizontal coordinate of the first pixel point of the first feature point in the matching point pair in the first reference image is x Lij , the horizontal coordinate of the second pixel point of the second feature point in the matching point pair in the second reference image is x Rij , so the absolute value of the difference between the first pixel coordinate and the second pixel coordinate is |x Lij -x Rij |, that is, the coordinate difference of the matching point pair is |x Lij -x Rij |.

[0101] Step S1334: obtaining a touch detection result based on the coordinate difference of each matching point pair.

[0102] In this embodiment, the touch detection result is obtained based on the coordinate difference of each matching point pair. In one embodiment, the touch detection result can be obtained by calculating the average error of the coordinate difference of all matching point pairs and judging the average error. It can be understood that in other embodiments, the touch detection result can also be obtained by performing other processing on the coordinate difference of all matching point pairs.

[0103] Among them, Fig.11 As shown, Fig.11This is a flow chart of an embodiment of obtaining a touch detection result based on the coordinate difference of each matching point pair provided by the present application. Obtaining the touch detection result based on the coordinate difference of each matching point pair specifically includes the following sub-steps:

[0104] Step S111: Count the coordinate differences of each matching point pair to obtain the final difference.

[0105] In this embodiment, the coordinate differences of each matching point pair are counted to obtain the final difference. Specifically, the final difference is obtained by calculating the average error of the coordinate differences of each matching point pair. The formula for calculating the average error of the coordinate differences of all matching point pairs is:

[0106]

[0107] Where e is the average error; |x Lij -x Rij | is the coordinate difference of the matching point pair; x Lij is the horizontal coordinate of the jth matching point in the i-th sub-image in the first reference image; Rij is the horizontal coordinate of the jth matching point in the i-th sub-image in the second reference image.

[0108] Step S112: Determine whether the final difference is lower than a preset threshold.

[0109] In this embodiment, the size relationship between the final difference and the preset threshold is determined to determine whether the target part of the target object touches the touch detection plane. The size of the preset threshold is not limited and can be specifically set according to actual use needs.

[0110] When the final difference is lower than the preset threshold, step S113 is executed; when the final difference is not lower than the preset threshold, step S114 is executed.

[0111] Step S113: when the final difference is lower than the preset threshold, determining that the touch detection result includes that the target part touches the touch detection plane.

[0112] In this embodiment, when the final difference is lower than the preset threshold, there is no height difference between the target part of the target object in the first image and the second image, which indicates that the target part of the target object touches the touch detection plane.

[0113] Step S114: when the final difference is not lower than the preset threshold, determining the touch detection result includes that the target part does not touch the touch detection plane.

[0114] In this embodiment, when the final difference is not lower than the preset threshold, there is a height difference between the target part of the target object in the first image and the second image, which indicates that the target part of the target object does not touch the touch detection plane and the target part of the target object is in a hovering state.

[0115] See also Fig.12 , Fig.12 It is a schematic diagram of the framework of an embodiment of a touch detection device provided by the present application. The touch detection device 120 includes: an image acquisition module 121, a target detection module 122 and a joint analysis module 123. The image acquisition module 121 is used to acquire the first image and the second image captured on the touch detection plane; the target detection module 122 is used to perform target detection on the first image and the second image respectively, and obtain the target part of the target object at the first image position in the first image and the second image position in the second image respectively; the joint analysis module 123 is used to perform a joint analysis on the first image position and the second image position to obtain a touch detection result; wherein the touch detection result includes whether the target part touches the touch detection plane.

[0116] Among them, the joint analysis module 123 is used to perform a joint analysis on the first image position and the second image position to obtain a touch detection result, specifically including: based on the first image position, extracting a first area image from the first image, and based on the second image position, extracting a second area image from the second image; based on the first pixel point belonging to the target object in the first area image, obtaining a first optimized position of the target part in the first image, and based on the second pixel point belonging to the target object in the second area image, obtaining a second optimized position of the target part in the second image; performing a joint analysis on the first optimized position and the second optimized position to obtain a touch detection result.

[0117] Wherein, the above-mentioned target part is located at the end of the target object, and the step of obtaining the above-mentioned first optimized position or the second optimized position includes: performing target segmentation on the regional image to obtain a mask image of the target object; wherein the pixel values ​​of the pixel points belonging to the target object in the mask image are all first values; based on the mask image, the center line and edge of the target object are obtained; based on the intersection of the center line and the edge, the optimized position of the target object is obtained; wherein, when the regional image is the first regional image, the optimized position is the first optimized position, and when the regional image is the second regional image, the optimized position is the second optimized position.

[0118] The first region image and the second region image have the same size, and the relative position of the first image and the relative position of the second image and the relative position of the second region image are the same.

[0119] Among them, the joint analysis module 123 is used to perform a joint analysis on the first optimized position and the second optimized position to obtain a touch detection result, specifically including: based on the first optimized position, extracting a first reference image from the first image, and based on the second optimized position, extracting a second reference image from the second image; wherein the first reference image and the second reference image have the same size; respectively dividing the first reference image and the second reference image into a preset number of first sub-images and a preset number of second sub-images along the same direction, and forming a group of sub-image pairs with the first sub-image and the second sub-image arranged in the same order; respectively obtaining the coordinate difference of the matching point pairs in each group of sub-image pairs; wherein the matching point pairs include a first feature point from the first sub-image and a second feature point from the second sub-image; based on the coordinate difference of each matching point pair, obtaining the touch detection result.

[0120] Among them, the joint analysis module 123 is used to obtain the touch detection result based on the coordinate difference of each matching point pair, specifically including: counting the coordinate difference of each matching point pair to obtain the final difference; when the final difference is lower than the preset threshold, determining that the touch detection result includes that the target part touches the touch detection plane; when the final difference is not lower than the preset threshold, determining that the touch detection result includes that the target part does not touch the touch detection plane.

[0121] Among them, the joint analysis module 123 is used to obtain the coordinate difference of the matching point pairs in each group of sub-image pairs, specifically including: obtaining the first pixel coordinates and the second pixel coordinates of the first feature point and the second feature point in the matching point pair, respectively in the first reference image and the second reference image; based on the absolute value of the difference between the first pixel coordinate and the second pixel coordinate, obtaining the coordinate difference of the matching point pair.

[0122] Among them, neither the first reference image nor the second reference image contains the target object; the relative position of the first optimized position and the first reference image, and the relative position of the second optimized position and the second reference image are the same; and / or, the first reference image and the second reference image are equally divided along the first direction, and the coordinate difference is obtained based on the coordinate difference of the first feature point and the second feature point in the second direction, respectively, and the first direction is perpendicular to the second direction.

[0123] The first image is captured by a first camera, and the second image is captured by a second camera; a line connecting the optical centers of the first camera and the second camera is parallel to the touch detection plane, and / or the vertical distances from the optical centers of the first camera and the second camera to the touch detection plane are the same.

[0124] See also Fig.13 , Fig.131 is a schematic diagram of a framework of an embodiment of a touch detection device provided by the present application. The touch detection device 130 includes a memory 131 and a processor 132 coupled to each other, the memory 131 stores program instructions, and the processor 132 is used to execute the program instructions to implement the steps in any of the above touch detection method embodiments. Specifically, the touch detection device 130 may include but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited here.

[0125] Specifically, the processor 132 is used to control itself and the memory 131 to implement the steps in any of the above-mentioned touch detection method embodiments. The processor 132 can also be called a CPU (Central Processing Unit). The processor 132 may be an integrated circuit chip with signal processing capabilities. The processor 132 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 132 can be implemented by an integrated circuit chip.

[0126] Among them, the above-mentioned touch detection device also includes a first camera and a second camera, both of which are coupled to the processor, the first camera is used to photograph the touch detection plane to obtain a first image, and the second camera is used to photograph the touch detection plane to obtain a second image.

[0127] See also Fig.14 , Fig.14 The computer-readable storage medium 140 stores program instructions 141 that can be executed by a processor, and the program instructions 141 are used to implement the steps in any of the above touch detection method embodiments.

[0128] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation thereof can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0129] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0130] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0131] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0132] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0133] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0134] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A touch detection method, characterized in that: include: Acquire a first image and a second image captured on the touch detection plane; Performing target detection on the first image and the second image respectively, to obtain a first image position of a target part of a target object in the first image and a second image position in the second image respectively; Performing a joint analysis on the first image position and the second image position to obtain a touch detection result; wherein the touch detection result includes whether the target part touches the touch detection plane; The step of jointly analyzing the first image position and the second image position to obtain a touch detection result includes: Based on the first image position and a first pixel point belonging to the target object, obtaining a first optimized position of the target part in the first image; and Based on the second image position and a second pixel point belonging to the target object, obtaining a second optimized position of the target part in the second image; The first optimized position and the second optimized position are jointly analyzed to obtain the touch detection result.

2. The method according to claim 1, characterized in that The obtaining, based on the first image position and a first pixel point belonging to the target object, a first optimized position of the target part in the first image includes: extracting a first region image from the first image based on the first image position; Based on a first pixel point belonging to the target object in the first region image, obtaining a first optimized position of the target part in the first image; The obtaining, based on the second image position and a second pixel point belonging to the target object, a second optimized position of the target part in the second image comprises: extracting a second region image from the second image based on the second image position; Based on a second pixel point belonging to the target object in the second region image, a second optimized position of the target part in the second image is obtained.

3. The method according to claim 1, characterized in that The target part is located at the end of the target object, and the step of acquiring the first optimized position or the second optimized position includes: Performing target segmentation on the regional image to obtain a mask image of the target object; wherein the pixel values ​​of the pixel points belonging to the target object in the mask image are all first values; Based on the mask image, acquiring a center line and an edge of the target object; Based on the intersection of the center line and the edge, obtaining an optimized position of the target object; Wherein, when the region image is a first region image, the optimized position is the first optimized position; and when the region image is a second region image, the optimized position is the second optimized position.

4. The method according to claim 2, characterized in that: The first region image and the second region image have the same size, and a relative position of the first image to the first region image and a relative position of the second image to the second region image are both the same.

5. The method according to claim 1, characterized in that The jointly analyzing the first optimized position and the second optimized position to obtain the touch detection result includes: Extracting a first reference image from the first image based on the first optimized position, and extracting a second reference image from the second image based on the second optimized position; wherein the first reference image and the second reference image have the same size; The first reference image and the second reference image are equally divided into a preset number of first sub-images and second sub-images along the same direction, and the first sub-images and the second sub-images arranged at the same sequence are grouped into a group of sub-image pairs; Respectively obtaining coordinate differences of matching point pairs in each group of sub-image pairs; wherein the matching point pairs include a first feature point from the first sub-image and a second feature point from the second sub-image; The touch detection result is obtained based on the coordinate difference of each of the matching point pairs.

6. The method according to claim 5, characterized in that The step of obtaining the touch detection result based on the coordinate difference of each of the matching point pairs includes: Counting the coordinate differences of each pair of matching points to obtain a final difference; In a case where the final difference is lower than a preset threshold, determining that the touch detection result includes that the target part touches the touch detection plane; When the final difference is not lower than the preset threshold, it is determined that the touch detection result includes that the target part does not touch the touch detection plane.

7. The method according to claim 5, characterized in that The step of respectively obtaining the coordinate differences of the matching point pairs in each group of sub-image pairs comprises: Obtaining first pixel coordinates and second pixel coordinates of the first feature point and the second feature point in the matching point pair in the first reference image and the second reference image respectively; Based on the absolute value of the difference between the first pixel coordinates and the second pixel coordinates, a coordinate difference of the matching point pair is obtained.

8. The method according to claim 5, characterized in that Neither the first reference image nor the second reference image contains the target object; The relative position between the first optimized position and the first reference image, and the relative position between the second optimized position and the second reference image are the same; And / or, the first reference image and the second reference image are equally divided along a first direction, the coordinate difference is obtained based on the coordinate difference between the first feature point and the second feature point in a second direction, and the first direction is perpendicular to the second direction.

9. The method according to claim 1, characterized in that: The first image is captured by a first camera, and the second image is captured by a second camera; The line connecting the optical centers of the first camera and the second camera is parallel to the touch detection plane, and / or the vertical distances from the optical centers of the first camera and the second camera to the touch detection plane are the same.

10. A touch detection device, characterized in that: include: An image acquisition module, used to acquire a first image and a second image captured of the touch detection plane; A target detection module, used to perform target detection on the first image and the second image respectively, to obtain a first image position of a target part of a target object in the first image and a second image position in the second image respectively; A joint analysis module, configured to perform a joint analysis on the first image position and the second image position to obtain a touch detection result; wherein the touch detection result includes whether the target part touches the touch detection plane; The step of jointly analyzing the first image position and the second image position to obtain a touch detection result includes: Based on the first image position and a first pixel point belonging to the target object, obtaining a first optimized position of the target part in the first image; and Based on the second image position and a second pixel point belonging to the target object, obtaining a second optimized position of the target part in the second image; The first optimized position and the second optimized position are jointly analyzed to obtain the touch detection result.

11. A touch detection device, characterized in that: The invention comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the touch detection method according to any one of claims 1 to 9.

12. The device according to claim 11, characterized in that The touch detection device also includes a first camera and a second camera, both of which are coupled to the processor, the first camera is used to photograph the touch detection plane to obtain a first image, and the second camera is used to photograph the touch detection plane to obtain a second image.

13. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the touch detection method according to any one of claims 1 to 9.

Images