Method, device and computer-readable storage medium for matching a target based on pointing information of an indicator

Abstract

The embodiments of the present invention provide a method, device and computer-readable storage medium for matching a target based on the pointing information of an indicator, the method comprising: obtaining an intention vector of the indicator, wherein the intention vector reflects the position and indication direction of the indicator; screening out multiple candidate targets in the direction of the intention vector; and selecting a target that matches the intention vector from the multiple candidate targets based on the distance between the intention vector and the multiple candidate targets. The present invention can eliminate targets that do not match the direction of the intention vector, thereby removing targets that may interfere with the final result, thereby improving the accuracy of the match and ultimately improving the user experience.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of computer vision and human-computer interaction. More specifically, embodiments of the present invention relate to a method for matching a target based on pointing information of an indicator, a device for matching a target based on pointing information of an indicator, and a computer-readable storage medium. Background Art

[0002] This section is intended to provide background or context for embodiments of the invention set forth in the claims. The descriptions herein may include concepts that may be explored, but not necessarily concepts that have been previously thought of or explored. Therefore, unless noted herein, the content described in this section is not prior art with respect to the specification and claims of the present invention, and is not admitted to be prior art by inclusion in this section.

[0003] Gestures, or hand postures, refer to the specific movements and positions of people when using their arms. Gestures can be used for expression. They are one of the earliest and still widely used communication tools used by humans. In the field of human-computer interaction and computer vision, gesture interaction technology can be used to control devices through gestures. Gesture interaction can be applied in interactive scenarios such as automobiles, home appliances, mobile phones, AR / VR, games, marketing, education, and government affairs. Generally speaking, gesture interaction includes: capturing images or videos through an optical system to collect gesture information; based on image processing algorithms, determining the intention of the user's gesture based on the collected gesture information.

[0004] The intention of the user's gesture may be a target that the user desires to obtain. For example, in an application scenario of intelligent learning, the indicated text paragraph may be determined by a gesture, or the indicated topic may be determined by a gesture.

[0005] There are many technical solutions in the prior art for determining user intention based on user pointing information (e.g., user pointing with a finger), but these technical solutions provide poor user experience. For example, some technical solutions match the target area that the user wants to obtain to the wrong area, that is, the user's true intention cannot be identified, resulting in errors in subsequent processes (e.g., errors in collecting, querying, or reading content). Summary of the invention

[0006] In the prior art, gesture interaction has the above-mentioned drawbacks. Therefore, a method for matching a target based on the pointing information of an indicator is very much needed to solve at least one of the above-mentioned drawbacks. At the same time, the present invention also provides a device for matching a target based on the pointing information of an indicator and a computer-readable storage medium.

[0007] In this context, embodiments of the present invention are intended to provide a method for matching a target based on pointing information of a pointer and related products thereof.

[0008] In a first aspect of an embodiment of the present invention, a method for matching a target based on pointing information of an indicator is provided, comprising: obtaining an intention vector of the indicator, wherein the intention vector reflects the position and indication direction of the indicator; screening out a plurality of candidate targets in the direction of the intention vector; and selecting a target that matches the intention vector from the plurality of candidate targets based on the distance between the intention vector and the plurality of candidate targets.

[0009] In one embodiment, screening out multiple candidate targets in the direction of the intention vector includes: calculating the angle between the target and the intention vector, wherein the angle between the target and the intention vector is: the minimum absolute value of the angle between a vector from an indicator to any point in the area where the target is located and the intention vector; and selecting a target whose angle with the intention vector is less than a predetermined angle as a candidate target.

[0010] In another embodiment, screening out multiple candidate targets in the direction of the intention vector includes: taking targets that intersect with the intention vector as candidate targets, wherein the intersection of the target and the intention vector means that: the intention vector or an extension line of the intention vector intersects with the area where the target is located.

[0011] In yet another embodiment, the target area is a target polygonal area.

[0012] In yet another embodiment, the minimum value of the absolute value of the angle between the vector from the pointer to any point in the target area and the intention vector includes: the minimum value of the absolute value of the angle between the vector from the pointer to each edge of the target polygonal area and the intention vector; wherein, for any edge of the target polygonal area, its first vertex corresponds to a first angle, and its second vertex corresponds to a second angle, wherein the first angle is: the angle between the vector from the pointer to the first vertex and the intention vector, and the second angle is: the angle between the vector from the pointer to the second vertex and the intention vector; in response to When the product of the first angle and the second angle is less than or equal to zero, the angle between the vector from the pointer to the edge of the target polygonal area and the intention vector is zero; in response to the product of the first angle and the second angle being greater than zero, the angle between the vector from the pointer to the edge of the target polygonal area and the intention vector is the minimum of the first angle and the second angle; and based on the angle between the vector from the pointer to an edge of the target polygonal area and the intention vector, the minimum value of the absolute values ​​of the angles between the vectors from the pointer to each edge of the target polygonal area and the intention vector is obtained.

[0013] In yet another embodiment, the predetermined angle is less than or equal to 20°.

[0014] In one embodiment, selecting based on the distance between the intention vector and the multiple candidate targets includes: calculating the distance between the intention vector and the multiple candidate targets to obtain the minimum distance therebetween; in response to the minimum distance being less than a predetermined distance, selecting the candidate target corresponding to the minimum distance as the target matching the intention vector.

[0015] In another embodiment, calculating the distance between the intention vector and multiple candidate targets includes: calculating the distance between the intention vector and all edges of a candidate target, and selecting the minimum distance therebetween as the distance between the intention vector and the candidate target.

[0016] In another embodiment, calculating the distance between the intention vector and multiple candidate targets includes: determining one or more target edges in a candidate target, wherein the target edge is an edge of the candidate target that faces the intention vector; calculating the distance between the intention vector and the target edge of the candidate target, and selecting the minimum distance therebetween as the distance between the intention vector and the candidate target.

[0017] In another embodiment, determining one or more target edges in a candidate target includes: traversing each edge of a candidate target; in response to not intersecting with other edges when extending from a predetermined point on an edge in the opposite direction of the intention vector, the edge is the target edge.

[0018] In yet another embodiment, the predetermined point is a midpoint.

[0019] In one embodiment, calculating the distance between the intention vector and the target edge of the candidate target includes: in response to the intention vector being parallel to the target edge, the distance between the intention vector and the target edge of the candidate target is: the minimum value of the distance between the indicator and the target edge; in response to the extension line of the intention vector intersecting the target edge at a first intersection, the distance between the intention vector and the target edge of the candidate target is: the distance between the indicator and the first intersection; in response to the extension line of the intention vector intersecting the extension line of the target edge at a second intersection, determining the distance between the intention vector and the target edge of the candidate target according to the position of the endpoint corresponding to the extension line of the target edge.

[0020] In another embodiment, the distance between the intention vector and the target edge of the candidate target is determined based on the position of the endpoint corresponding to the extension line of the target edge, including: taking the distance between the indicator and the endpoint corresponding to the extension line of the target edge as the distance between the intention vector and the target edge of the candidate target; or taking the sum of the distance between the indicator and the second intersection and the distance between the second intersection and the endpoint corresponding to the extension line of the target edge as the distance between the intention vector and the target edge of the candidate target.

[0021] In yet another embodiment, calculating the distance between the intention vector and a plurality of candidate targets includes: determining the center of mass of each candidate target, and calculating the distance between the indicator and the center of mass as the distance between the intention vector and the corresponding candidate target.

[0022] In yet another embodiment, the angle and the distance between the target and the intention vector are calculated in parallel.

[0023] In yet another embodiment, the intention vector comprises: a vector from a center of the pointer to a tip of the pointer.

[0024] In one embodiment, the indicator includes a fingertip, a pen tip or a pointer; or, the target includes a topic or a paragraph.

[0025] In a second aspect of an embodiment of the present invention, a device for matching a target based on pointing information of an indicator is provided, comprising: a processor configured to execute program instructions; and a memory configured to store the program instructions, wherein when the program instructions are loaded and executed by the processor, the processor executes a method for matching a target based on pointing information of an indicator as described in any one of the first aspect of the embodiment of the present invention.

[0026] In a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which program instructions are stored. When the program instructions are loaded and executed by a processor, the processor executes the method for matching a target based on pointing information of an indicator as described in any one of the first aspect of the embodiment of the present invention.

[0027] The beneficial effects of the present invention include: since the intention vector is adopted, and the intention vector not only contains position information but also direction information, the targets that do not match the direction of the intention vector can be directly eliminated, thereby removing the targets that may interfere with the final result, thereby improving the matching accuracy and ultimately improving the user experience.

[0028] Furthermore, the polygonal outline of the target area is used for matching, so that the correct target can be matched in most scenarios where the user's pointing intention is clear (including complex scenarios such as tilt or distortion). Furthermore, when calculating the distance between the target and the intention vector, the direction of the intention vector can still be used to select the target edge of the target polygon, thereby further improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:

[0030] Figure 1 A block diagram schematically illustrates an exemplary computing system 100 suitable for implementing embodiments of the present invention;

[0031] Figure 2 The application scenario of a method for matching a target based on pointing information of an indicator in the prior art is schematically shown;

[0032] Figure 3 The application scenario of another method for matching a target based on pointing information of an indicator in the prior art is schematically shown;

[0033] Figure 4 A flowchart of a method 400 for matching a target based on pointing information of a pointer according to an embodiment of the present invention is schematically shown;

[0034] Figure 5 The flowchart of step S402 according to the embodiment of the present invention is schematically shown;

[0035] Figure 6 Schematic diagrams schematically showing several situations of step S402 according to an embodiment of the present invention;

[0036] Figure 7 The flowchart of step S403 according to the embodiment of the present invention is schematically shown;

[0037] Figure 8 A schematic diagram schematically shows a target edge selection method according to an embodiment of the present invention;

[0038] Fig. 9 Schematic diagrams of three situations of calculating the distance between the intention vector and the target edge of the candidate target according to an embodiment of the present invention are exemplarily shown;

[0039] Fig.10 One of the typical example diagrams according to the embodiment of the present invention is exemplarily shown;

[0040] Fig.11 A second typical example diagram according to an embodiment of the present invention is exemplarily shown; and

[0041] Fig.12 A schematic block diagram of a device 1200 for matching a target based on pointing information of a pointer according to an embodiment of the present invention is schematically shown;

[0042] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0043] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0044] Figure 1 1 is a block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present invention. Figure 1 As shown, the computing system 100 may include: a central processing unit (CPU) 101, a random access memory (RAM) 102, a read-only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial external device 112, a parallel external device 113, and a display 114. Among these devices, the CPU 101, the RAM 102, the ROM 103, the hard disk controller 105, the keyboard controller 106, the serial controller 107, the parallel controller 108, and the display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial external device 112 is coupled to the serial interface controller 107, the parallel external device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that Figure 1 The structural block diagram is only for the purpose of illustration, rather than limiting the scope of the present invention. In some cases, some equipment may be added or reduced according to specific circumstances.

[0045] It is known to those skilled in the art that the embodiments of the present invention may be implemented as a system, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit", "module", "unit" or "system". In addition, in some embodiments, the present invention may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0046] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive examples) of computer-readable storage media may include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0047] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0048] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0049] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0050] The following will describe the implementation of the present invention with reference to the flowchart of the method of the embodiment of the present invention and the block diagram of the device (or system). It should be understood that each square frame of the flowchart and / or block diagram and the combination of square frames in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, thereby producing a machine, and these computer program instructions are executed by a computer or other programmable data processing device to produce a device that implements the function / operation specified in the square frame in the flowchart and / or block diagram.

[0051] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0052] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable device to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0053] According to an embodiment of the present invention, a method for matching a target based on pointing information of an indicator and a related product thereof are provided. In addition, any number of elements in the drawings is for illustration and not for limitation, and any naming is only for distinction and does not have any limiting meaning.

[0054] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention. SUMMARY OF THE INVENTION

[0056] The inventors have found that in order to identify the user's intention, the prior art uses the position coordinates of the indicator (such as the position coordinates of the finger, which can also be called the pointing coordinates). For example, whether the user's intention matches the target is determined based on whether the pointing coordinates fall within the target area. In some application scenarios, this method can determine the target corresponding to the user's true intention; however, in other application scenarios, this method cannot obtain the target corresponding to the user's true intention. In other words, the target in the area where the user's pointing coordinates are located is not necessarily the user's true intention.

[0057] Figure 2 The application scenario of a method for matching a target based on pointing information of a pointer in the prior art is schematically shown.

[0058] like Figure 2 As shown, F is a pointer, which can be a fingertip, a pen tip, or a pointer, etc. The following description takes a finger as an example. The indicated target includes a title or paragraph in a book, and the target area is a target polygon area, for example Figure 2 In FIG. 1 , the region where the targets 201 , 202 , 203 , 204 and 205 are located is a polygonal region (convex polygonal region).

[0059] like Figure 2 In the application scenario shown, the finger F obviously points to the target 203 , and the pointing coordinates are in the area where the target 204 is located. Therefore, according to the technical solution of the prior art, the matched target is likely to be the target 204 , rather than the target 203 .

[0060] Figure 3 Another application scenario of a method for matching a target based on pointing information of a pointer in the prior art is schematically shown.

[0061] like Figure 3 In the application scenario shown, the technical solution used is to use the horizontal rectangular box of the target area for matching. The reason for using the horizontal rectangular box matching is that the horizontal rectangular box is easier to obtain than the fine polygon outline (polygon outline is used to surround the target area), and the matching strategy is also simpler. Figure 3As shown, target 301 includes a horizontal rectangular frame 3011, and target 302 includes a horizontal rectangular frame 3021. The placement of the book is tilted or distorted (the dotted line indicates the central axis of the book), and the indicator F obviously indicates target 302. However, due to the influence of the horizontal rectangular frame 3011 of target 301, it is easy to judge the user's intention as target 301. If a more refined polygonal outline is used to represent the target area, it is possible to match the correct target 302.

[0062] based on Figure 2 and Figure 3 In order to solve the problem reflected in the application scenarios, the inventor proposed a solution, which makes a judgment based on the direction and coordinates of the pointing vector, and uses the polygonal contour box of the target area for matching, so that the correct target can be matched in most scenarios where the user's pointing intention is clear (including complex scenarios such as tilt or distortion).

[0063] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.

[0064] Exemplary Methods

[0065] Reference below Figure 4 A method for matching a target based on the pointing information of an indicator according to an exemplary embodiment of the present invention is described. It should be noted that the embodiments of the present invention can be applied to any applicable scenario. For example, intelligent learning, assisted learning, and intelligent reading. As long as it is an application scenario that can use gesture interaction, the method introduced in the present invention can be used.

[0066] Figure 4 A flowchart of a method 400 for matching a target based on pointing information of an indicator according to an embodiment of the present invention is schematically shown, including steps S401, S402 and S403. In this embodiment, the example of pointing to a title / paragraph in a book by a finger in the field of intelligent learning is used for explanation.

[0067] In step S401, the intention vector of the indicator is obtained. As an example, a capture device (such as a fixed camera or a mobile camera) can be used to capture an indicator (such as a finger) and a target carrier (such as a book) to obtain an image containing the finger and the book. Depending on the different capture devices or sensors used, the acquired image can be an RGB image or an RGB-D image.

[0068] The pointing information mainly includes an intention vector, which reflects the position and pointing direction of the finger. The intention vector can be expressed as a vector from the center of the pointer to the tip of the pointer. For example, the intention vector of the finger can be a fingertip vector, which can be expressed as a vector F from the center of the finger to the fingertip: F ​​= f c f t , where f c is the center position, f t is the fingertip position. Each target can be surrounded by a convex polygon, which can be expressed as: V = {v1,v2,…,v n}, where v1, v2…v n Represents all vertices of a convex polygon.

[0069] In the following description, the fingertip vector is used to represent the intention vector; the target area can also be called the target convex polygon area, the target polygon area or the target polygon. It should be noted that the target polygon described below refers specifically to a convex polygon, and does not include a concave polygon.

[0070] In step S402, multiple candidate targets in the direction of the intention vector are screened. Among them, the targets can be divided into two categories, the first category is the targets in the direction of the intention vector, and the second category is the targets not in the direction of the intention vector. Through this step, the second category of targets can be directly eliminated, and the first category of targets can be used as candidate targets for further processing in subsequent steps.

[0071] In step S403, a target matching the intention vector is selected from the multiple candidate targets according to the distance between the intention vector and the multiple candidate targets.

[0072] According to the above-mentioned embodiment of the present invention, since an intention vector is adopted, and the intention vector contains not only position information but also direction information, the target that does not match the direction of the intention vector can be directly eliminated in step S402, thereby removing the target that may interfere with the final result, thereby improving the matching accuracy and ultimately improving the user experience.

[0073] The details of the embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0074] Figure 5 The flowchart of step S402 according to an embodiment of the present invention is schematically shown. The purpose of step S402 is to preliminarily determine the user's true intention, for example, based on the position of the fingertip vector and the target, select the target in the direction of the fingertip vector as a candidate target. Figure 5 As shown, step S402 includes step S501 to step S502.

[0075] In step S501, the angle between the target and the intention vector is calculated, where the angle between the target and the intention vector is the minimum absolute value of the angle between the vector from the indicator to any point in the target area and the intention vector (wherein, according to a predetermined positive direction, the angle between the vector from the indicator to any point in the target area and the intention vector can be either positive or negative).

[0076] The intention vector can be expressed as the fingertip vector F = f c f t ; The target polygonal area is represented by P; a point on the finger can be selected as the indicator. For example, in this embodiment, the coordinates of the center of the finger are used as the coordinates of the indicator.

[0077] The angle between the target polygonal area and the intention vector is defined as the minimum absolute value of the angle between the vector from the finger center to any point in the target polygonal area and the fingertip vector, ranging from [0, π], expressed as:

[0078]

[0079] Since the angle between the vector from the finger center to any point in the target polygonal area and the fingertip vector (the angle can be either positive or negative) must take the maximum and minimum values ​​at the vertex of the target polygon. In addition, according to the density of points, the angle value on any edge of the target polygonal area is continuous, and the maximum and minimum values ​​are respectively taken at the two endpoints of any edge. Therefore, it is not actually necessary to calculate the angle corresponding to all points in the target polygonal area. It is only necessary to calculate the angle corresponding to each vertex (that is, the endpoints of all edges) of the target polygonal area. The calculation method can be expressed as:

[0080] The value range is [-π,π]. Where sign(·) is an indicator function (when (·) is greater than 0, it takes the value 1; when (·) is less than 0, it takes the value -1; when (·) is equal to 0, it takes the value 0).

[0081] Based on the above formula, the angles corresponding to all vertices of the target polygonal area can be obtained. According to the angles corresponding to the two vertices of each edge, the angle between each edge of the target polygonal area and the fingertip vector F can be obtained. It should be noted that in this embodiment, the angle between a certain edge of the target polygonal area and the fingertip vector F is defined as a positive value.

[0082] As an example, for any edge of the target polygonal area, its first vertex corresponds to a first angle, and its second vertex corresponds to a second angle, wherein the first angle is: the angle between the vector from the indicator (such as a fingertip) to the first vertex and the intention vector, and the second angle is: the angle between the vector from the indicator (such as a fingertip) to the second vertex and the intention vector. According to the density of points, the angle value on any edge of the target polygonal area is continuous. Therefore, there are three cases for the angle between the edge and the fingertip vector F: in the first case, the directions of the first angle and the second angle are the same (both are positive values). In this case, the angle between the edge and the fingertip vector F is the minimum of the first angle and the second angle; in the second case, the directions of the first angle and the second angle are the same (both are negative values). In this case, the angle between the edge and the fingertip vector F is the minimum of the absolute values ​​of the first angle and the second angle; in the third case, the directions of the first angle and the second angle are opposite. At this time, there must be a point in the edge, and the angle corresponding to the point is zero. In this case, the angle between the edge and the fingertip vector F is zero.

[0083] In one embodiment, the method for calculating the angle between each edge of the target polygonal area and the fingertip vector F includes: for an edge, calculating the product of its first angle and second angle. If the product of its first angle and second angle is greater than zero, it means that the directions of the first angle and the second angle are the same, and this corresponds to the first case and the second case above, that is, the edge is on the same side of the fingertip vector, then the angle between the vector from the finger center to the edge and the fingertip vector is the minimum value of the absolute values ​​of the first angle and the second angle.

[0084] For an edge, if the product of its first angle and the second angle is less than or equal to zero, it means that the first angle and the second angle are in opposite directions (or the first angle and / or the second angle are zero), which corresponds to the third case mentioned above, that is, there must be a point in the edge, and the angle corresponding to the point is zero. Therefore, it can be determined that the angle between the vector from the finger center to the edge and the fingertip vector is zero.

[0085] In summary, the angle between the vector from the finger center to a certain edge and the fingertip vector F can be expressed as:

[0086]

[0087] Figure 6 The following is a schematic diagram schematically showing several situations of step S402 according to an embodiment of the present invention.

[0088] In (a), the angle between the vector from the finger center to a certain edge and the fingertip vector F is zero, so the minimum value of the angle between the vectors from the finger center to each edge of the target polygonal area P and the fingertip vector F must be zero, and then the angle Angle between the target polygonal area P and the intention vector is Angle = 0°.

[0089] In (b) and (c), if all the edges of the target polygonal area belong to the first case or the second case, it means that the entire target polygonal area P is on the same side of the fingertip vector F. Then, the minimum value of the angle between the vectors from the finger center to each edge of the target polygonal area and the fingertip vector F can be calculated, and the minimum value is the angle between the target polygonal area P and the intention vector F.

[0090] like Figure 6 As shown, the angle between the target polygon area P and the intention vector F can be expressed as:

[0091] Where i represents the ordinal number of the target polygon vertex.

[0092] In step S502, the target whose angle with the intention vector is less than a predetermined angle is taken as a candidate target. After calculating the angle between the fingertip vector F and each target polygonal area, the target can be screened according to a predetermined angle θ, that is, the target whose angle is less than or equal to the predetermined angle θ is a candidate target; the target whose angle is greater than the threshold θ is a non-candidate target.

[0093] The predetermined angle may be determined according to factors such as the actual habits of the user. For example, in this embodiment, the predetermined angle is 20°. In other embodiments, a smaller angle, such as 10° or 15°, or a larger angle, such as 25°, may be used. Step S502 may be expressed as:

[0094] in,

[0095] P candidate Indicates a candidate target.

[0096] As an example, Figure 6 As shown in (a), the angle between the target polygonal area P and the fingertip vector F is 0°, so the corresponding target belongs to the candidate target. In (b), the angle between the target polygonal area P and the fingertip vector F is 15°, which is less than the predetermined angle of 20°, so the corresponding target belongs to the candidate target. In (c), the angle between the target polygonal area P and the fingertip vector F is 40°, which is greater than the predetermined angle of 20°, so the corresponding target does not belong to the candidate target.

[0097] An implementation of step S402 is described above. In another implementation, step S402 may also be implemented in other ways, such as determining whether the intention vector intersects with the target area, or determining whether the extension line of the intention vector intersects with the target area. If so, the target is used as a candidate target. The technical means for determining whether a straight line intersects with an area, or determining whether straight lines intersect with each other, are conventional technical means, and therefore will not be elaborated on in detail here.

[0098] Figure 7 The flowchart of step S403 according to an embodiment of the present invention is schematically shown. The purpose of step S403 is to finally determine the user's true intention, that is, to further filter out the target that matches the intention vector from the candidate targets. For example, by calculating the distance from the fingertip vector to the target polygon area, the candidate target corresponding to the minimum distance can be selected and used as the final target. Figure 7 As shown, step S403 includes steps S701 to S703.

[0099] In step S701, one or more target edges in a candidate target are determined, wherein the target edge is an edge of the candidate target that is oriented toward the intention vector; the distance between the intention vector and the target edge of the candidate target is calculated, and the minimum distance is selected as the distance between the intention vector and the candidate target.

[0100] As an example, Figure 8 The schematic diagram of target edge selection according to an embodiment of the present invention is schematically shown. There is a candidate target polygon P and a fingertip vector F. The above-mentioned "direction" can be figuratively understood as a light source at infinity of the candidate target polygon P, which generates parallel light in the same direction as the fingertip vector F, irradiating the candidate target polygon P, and the edge illuminated by the parallel light is the target edge, such as Figure 8 Shown by the thick solid line.

[0101] For example, the target edge selection method may be: traverse each edge of a candidate target polygon; in response to not intersecting with other edges when extending from a predetermined point on an edge in the opposite direction of the intention vector, the edge is the target edge. The predetermined point may be a midpoint, a 1 / 3 point, or a 2 / 3 point.

[0102] Specifically, first traverse each edge of the candidate target polygon and determine whether the ray ray(m e ,F) intersects with each edge of the candidate target convex polygon, where the ray ray(m e ,F) is from the midpoint m of each edge e e Starting from the fingertip vector F, the direction is opposite to the fingertip vector F. If the ray ray(m e,F) does not intersect with any other edge, then the edge e is the target edge; if the ray ray(m e ,F) intersects with any edge, then the edge e is not the target edge. Figure 8 As shown in the figure, the ray corresponding to the thick solid line (such as ray1) obviously will not intersect with other edges of the candidate target polygon P, so the edge corresponding to the thick solid line is the target edge; while the ray corresponding to the thin solid line (such as ray2) will at least intersect with the thick solid line, so the edge corresponding to the thin solid line is obviously not the target edge. Based on the above process, the target edge set E can be obtained:

[0103]

[0104] In step S702, the distance between the intention vector and multiple candidate targets is calculated to obtain the minimum distance. According to step S701, the distance between the intention vector F and a candidate target P can be expressed as the minimum value of the distance between the intention vector F and all target edges e of the candidate target:

[0105] Where e is the target edge belonging to the target edge set E.

[0106] Fig. 9 The schematic diagrams of three cases of calculating the distance between the intention vector and the target edge of the candidate target according to the embodiment of the present invention are exemplarily shown, including three cases (x), (y) and (z).

[0107] (x) case, the fingertip vector F is parallel to the target edge e, so the distance between the intention vector and the target edge of the candidate target is: the minimum value of the distance between the fingertip vector and the target edge. Fig. 9 As shown, the minimum distance between the fingertip vector F and the target edge e is the line segment f t P e 1 Or the f of a line segment t P e 2 Length, f t is the fingertip coordinate, P e 1 is the first end of the target edge e, P e 2 is the second end of the target edge e.

[0108] (y) case, the extension line of the fingertip vector F intersects the target edge at the first intersection point p, then the distance between the fingertip vector F and the target edge e of the candidate target is: the distance between the fingertip vector F and the first intersection point p, for example, line segment f t The length of p.

[0109] (z) case, the extension line of the fingertip vector F intersects with the extension line of the target edge e at the second intersection point p (for the convenience of mathematical expression, the second intersection point here is also marked with p), as Fig. 9 As shown, the endpoint corresponding to the extension line of the target edge e is P e 1 At this time, according to the endpoint P e 1 Specifically, the distance between the fingertip vector F and the target edge e of the candidate target can be the distance between the fingertip vector F and the endpoint P e 1 The distance (for example, line segment f t P e 1 length), or the distance between the tip vector F and the second intersection point p and the distance between the second intersection point p and the endpoint P e 1 The sum of the distances of the line segment f t p and line segment pP e 1 The sum of the lengths of t p+p e 1 >f t P e 1 , so it is easier to exclude the target corresponding to this situation (z), which is beneficial to improve the matching accuracy).

[0110] In the above three cases, the distance between the fingertip vector F and the target edge e of the candidate target can be expressed as:

[0111]

[0112] In the above embodiment, since the target edge is selected in step S701, the advantage of the directionality of the intention vector can be further utilized, which is conducive to matching the real user intention.

[0113] In another embodiment, the distance between the intention vector and all edges of a candidate target may be calculated, and the minimum distance may be selected as the distance between the intention vector and the candidate target. For example, for a target polygon, the distance between the fingertip vector and all edges of the target polygon may be calculated, and the minimum value may be selected as the distance between the fingertip vector and the target polygon. The disadvantage of this embodiment is that the matching result may not match the user's actual intention. For example, in Figure 2 In the example, if the method of this embodiment is used, since the fingertip vector is closer to the target 204, the final matching result may be the target 204, rather than the user's true intention, which is the target 203.

[0114] In another embodiment, the centroid of each candidate target may be determined first, and then the distance between the fingertip vector and the centroid is calculated as the distance between the intention vector and the corresponding candidate target, and finally the minimum distance is selected as the distance between the intention vector and the candidate target. The disadvantage of this embodiment is that the matching result may not match the user's actual intention. For example, Figure 2 If the horizontal length of target 204 is longer and that of target 203 is shorter, it is possible to match the user's true intention, target 203; conversely, if the horizontal length of target 204 is shorter and that of target 203 is longer, the final matching result may be target 204, rather than the user's true intention, target 203.

[0115] In step S703, in response to the minimum distance being less than a predetermined distance, the candidate target corresponding to the minimum distance is selected as the target matching the intention vector, wherein the predetermined distance is pre-set according to actual conditions, for example, the predetermined distance can be set to 200 (pixels).

[0116] Fig.10 One of the typical example diagrams according to the embodiment of the present invention is exemplarily shown. Fig.10 As shown, according to step S402, the angles between the targets represented by the white boxes and the intention vector F are all greater than 20°, while the angles between targets 1001, 1002 and 1003 and the intention vector F are less than 20°, so the selected candidate targets include targets 1001, 1002 and 1003. According to step S403, the distances between targets 1001 and 1002 and the intention vector are greater than 200 (pixels), and the distance between target 1003 and the intention vector F is less than 200 (pixels), so the final result of the match is target 1003. In addition, there may be a situation where the distances between multiple candidate targets and the intention vector F are less than the predetermined distance at the same time, in which case the target with the smallest distance to the intention vector F can be selected.

[0117] Fig.11 FIG2 is a typical example diagram according to an embodiment of the present invention. Fig.11 As shown, according to step S402, the angles between the targets represented by the white boxes and the intention vector F are all greater than 20°, while the angles between targets 1101, 1102 and 1103 and the intention vector F are less than 20°, so the selected candidate targets include targets 1101, 1102 and 1103. According to step S403, the distances between targets 1101, 1102 and 1103 and the intention vector F are all greater than 200 (pixels), indicating that the user's intention is unclear at this time, so no suitable target is finally matched.

[0118] According to the method of the embodiment of the present invention, the calculation of the angle and distance between the target and the intention vector can be performed sequentially. In other embodiments, the angle and distance between the target and the intention vector can also be calculated in parallel. As an example, Fig.10 The eight targets are included, and the angles between the eight targets and the intention vector, as well as the distances between the eight targets and the intention vector can be calculated in parallel, and then matched according to the corresponding angle threshold and distance threshold, thereby greatly improving the calculation efficiency in the implementation process of the scheme of the present invention. Moreover, all the operations involved can be efficiently completed on the CPU without involving any complex operations.

[0119] Exemplary Devices

[0120] After introducing the method of the exemplary embodiment of the present invention, next, refer to Fig.12 A device for a method of matching a target based on pointing information of a pointer according to an exemplary embodiment of the present invention is described.

[0121] Fig.12 The schematic block diagram of the device 1200 for matching a target based on the pointing information of a pointer according to an embodiment of the present invention is schematically shown. Fig.12 As shown, the device 1200 for matching a target based on pointing information of a pointer may include a processor 1201 and a memory 1202. The memory 1202 stores computer instructions for executing the method for matching a target based on pointing information of a pointer according to an embodiment of the present invention. When the computer instructions are executed by the processor 1201, the device 1200 executes the method described above.

[0122] In addition, in practical applications, the device 1200 for matching a target based on the pointing information of a pointer may also include a collection device for acquiring an image. In order to display the result, the device 1200 for matching a target based on the pointing information of a pointer may also include a display and other devices. It should be noted that the embodiments of the present invention do not limit the structure that the device 1200 for matching a target based on the pointing information of a pointer may have.

[0123] It should be noted that although several steps of the method for matching a target based on the pointing information of an indicator are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more steps described above can be concretized in one step. Conversely, the features and functions of one step described above can be further divided into multiple steps to be concretized.

[0124] In addition, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0125] The use of the verbs "comprise", "include" and their conjugations mentioned in the application documents does not exclude the presence of elements or steps other than those recorded in the application documents. The article "a" or "an" before an element does not exclude the presence of a plurality of such elements.

[0126] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and this division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. A method for matching a target based on pointing information of an indicator, characterized in that: include: Acquire an intention vector of the indicator, wherein the intention vector reflects a position and an indication direction of the indicator; Screening out multiple candidate targets in the direction of the intention vector; and Selecting a target matching the intention vector from the multiple candidate targets according to the distance between the intention vector and the multiple candidate targets; Wherein, screening out a plurality of candidate targets in the direction of the intention vector comprises: calculating an angle between the target and the intention vector, and taking a target whose angle with the intention vector is less than a predetermined angle as a candidate target; The angle between the target and the intention vector is the minimum absolute value of the angle between the vectors of the edges from the indicator to the target area and the intention vector, and the target area is the target convex polygon contour area; Wherein, for any edge of the target convex polygonal contour area, its first vertex corresponds to a first angle, and its second vertex corresponds to a second angle, wherein the first angle is: the angle between the vector from the indicator to the first vertex and the intention vector, and the second angle is: the angle between the vector from the indicator to the second vertex and the intention vector; In response to the product of the first angle and the second angle being less than or equal to zero, the angle between the vector from the pointer to the edge of the target convex polygonal contour area and the intention vector is zero; In response to the product of the first angle and the second angle being greater than zero, the angle between the vector from the pointer to the edge of the target convex polygonal contour area and the intention vector is the minimum value between the first angle and the second angle; and According to the angle between the vector from the pointer to an edge of the target convex polygonal contour area and the intention vector, the minimum absolute value of the angles between the vectors from the pointer to each edge of the target convex polygonal contour area and the intention vector is obtained.

2. The method according to claim 1, characterized in that Screening out multiple candidate targets in the direction of the intention vector also includes: A target intersecting with the intention vector is taken as a candidate target, wherein the target intersects with the intention vector when: the intention vector or an extension line of the intention vector intersects with a region where the target is located.

3. The method according to claim 1, characterized in that The predetermined angle is less than or equal to 20°.

4. The method according to claim 1, characterized in that: Selecting according to the distance between the intention vector and the multiple candidate targets includes: Calculating the distance between the intention vector and multiple candidate targets to obtain the minimum distance therebetween; In response to the minimum distance being less than a predetermined distance, a candidate target corresponding to the minimum distance is selected as a target matching the intention vector.

5. The method according to claim 4, characterized in that Calculating the distance between the intention vector and multiple candidate targets includes: The distances between the intention vector and all edges of a candidate target are calculated, and the minimum distance therebetween is selected as the distance between the intention vector and the candidate target.

6. The method according to claim 4, characterized in that Calculating the distance between the intention vector and multiple candidate targets includes: Determine one or more target edges in a candidate target, wherein the target edge is an edge of the candidate target that faces the intention vector; The distance between the intention vector and the target edge of the candidate target is calculated, and the minimum distance is selected as the distance between the intention vector and the candidate target.

7. The method according to claim 6, characterized in that Determining one or more target edges in a candidate target includes: Traverse each edge of a candidate target; In response to not intersecting with other edges when extending from a predetermined point on an edge in the opposite direction of the intention vector, the edge is the target edge.

8. The method according to claim 7, characterized in that The predetermined point is the midpoint.

9. The method according to claim 6, characterized in that Calculating the distance between the intention vector and the target edge of the candidate target includes: In response to the intention vector being parallel to the target edge, the distance between the intention vector and the target edge of the candidate target is: a minimum value of the distance between the indicator and the target edge; In response to the extension line of the intention vector intersecting the target edge at a first intersection point, the distance between the intention vector and the target edge of the candidate target is: the distance between the indicator and the first intersection point; In response to the extension line of the intention vector and the extension line of the target edge intersecting at a second intersection point, the distance between the intention vector and the target edge of the candidate target is determined according to the positions of the endpoints corresponding to the extension line of the target edge.

10. The method according to claim 9, characterized in that Determining the distance between the intention vector and the target edge of the candidate target according to the position of the endpoint corresponding to the extension line of the target edge includes: Using the distance between the pointer and the endpoints corresponding to the extension line of the target edge as the distance between the intention vector and the target edge of the candidate target; or The sum of the distance between the indicator and the second intersection point and the distance between the second intersection point and the endpoints corresponding to the extension line of the target edge is used as the distance between the intention vector and the target edge of the candidate target.

11. The method according to claim 4, characterized in that Calculating the distance between the intention vector and multiple candidate targets includes: Determine the centroid of each candidate object, and The distance between the indicator and the center of mass is calculated as the distance between the intention vector and the corresponding candidate target.

12. The method according to claim 1, characterized in that The angle and distance between the target and the intention vector are calculated in parallel.

13. The method according to any one of claims 1 to 12, characterized in that: The intent vector includes: The vector from the center of the pointer to the tip of the pointer.

14. The method according to claim 13, characterized in that The indicator includes a fingertip, a pen tip or a pointer.

15. The method according to claim 13, characterized in that The target includes a title or a paragraph.

16. A device for matching a target based on pointing information of an indicator, characterized in that: include: a processor configured to execute program instructions; as well as A memory configured to store the program instructions, and when the program instructions are loaded and executed by the processor, the processor executes the method for matching a target based on pointing information of a pointer according to any one of claims 1 to 15.

17. A computer-readable storage medium, characterized in that: Program instructions are stored therein, and when the program instructions are loaded and executed by a processor, the processor executes the method for matching a target based on pointing information of an indicator according to any one of claims 1 to 15.

Images