Intelligent mobile device pairing method using camera

Abstract

The invention discloses a target tracking-based pairing method by using a camera of intelligent mobile equipment, which is used for safe pairing of the intelligent mobile equipment and comprises the following steps: 1) a key generation method and a key sharing method based on a finger tracking trajectory can generate a key locally and encrypt and decrypt information, and the key can be shared by the key generation method and the key sharing method; determining a target device; 2) a key correction method based on a finger tracking trajectory can improve the robustness of a key generation method; and 3) a feature vector generation method based on a finger tracking trajectory can quickly find target equipment to finish quick pairing. The pairing method of the intelligent mobile equipment is designed from a new angle, can be realized on the existing equipment as software, and is a user-friendly and safe equipment pairing method oriented to the intelligent mobile equipment.

Description

Technical Field

[0001] The present invention relates to a method for pairing a smart mobile device, and in particular to a method for pairing a smart mobile device based on target tracking using a camera of the smart mobile device, which is used for pairing devices when two devices are first introduced and connected to work together before the two devices are set up, and belongs to the field of mobile computing application technology. Background Art

[0002] With the popularity of smart mobile devices, such as smartphones and smart glasses, many new application scenarios have emerged, many of which require smart mobile users to interact with other users or smart mobile devices, such as file sharing and team collaboration games. A prerequisite for realizing such application scenarios is device pairing, which is the first introduction and connection between two devices so that they can work together without pre-settings. Currently, there are two key issues in device pairing between two smart mobile devices (initiator device and target device): one is how the initiator device identifies the target device among the surrounding devices, and the other is how to achieve key sharing between the initiator device and the target device.

[0003] To solve these problems, a common method is to require users to create a unique ID on the device, such as a barcode, or a unique attribute of the device, also known as an identifier or fingerprint, and then the two users ask each other for ID or fingerprint matching. This method requires sharing a common fingerprint database before the devices are connected, and cannot provide spatial perception during the user's selection of the target device, thus affecting the user experience, especially in the application scenario of augmented reality environment. Another common method is the proximity-based method, which derives a shared key from the signal for pairing by sharing environmental signals, such as the channel state information of wireless signals. Although this type of method can identify a group of devices that are close to each other, it cannot identify a specific target device among them, and the devices need to be at a short distance. In addition, there is a method that uses audio signals for pairing. This method requires users to make a specific pointing gesture by holding a smartphone or shaking the device to select the target device, but this gesture is not applicable to all smart mobile devices.

[0004] In summary, the existing mainstream methods have some shortcomings, and new methods are urgently needed to overcome their limitations. The present invention allows users to directly select the device they want to pair with by waving their fingers in the air. The method can use the key features in the finger tracking trajectory to discover the user's intention to interact with which device, that is, identify the target device, and then design key generation and key correction algorithms to enhance the security of information interaction after device pairing. Summary of the invention

[0005] The purpose of the present invention is to realize a user-friendly and safe intelligent mobile device pairing method, and to propose an intelligent mobile device pairing method based on target tracking using a camera.

[0006] The idea of ​​the present invention is to track the user's waving fingers by using the camera equipped with an existing smart mobile device, utilize and analyze the trajectory of finger tracking, identify the target device, and then generate a shared key and distribute it between the two devices, so that the two devices can be paired and the security of information interaction after the devices are paired is guaranteed.

[0007] An embodiment of the present invention provides a method for pairing a smart mobile device based on target tracking, characterized in that the method comprises the following steps: 1) When a user (initiator device) wants to interact with another user (target device), the user waves his finger towards the target device and makes a specific gesture in the air, that is, draws a shape in the air; 2) The initiator device and several surrounding devices (including the target device) use existing cameras to track the initiator user's gestures and record the tracking trajectory; 3) The initiator device and surrounding devices extract the key from the finger tracking trajectory; 4) The surrounding device encrypts the message using the key extracted in step 3) and sends it to the initiator device together with the device ID; 5) The initiator device uses the key extracted in step 3) to decrypt the received message, find and identify the target device, obtain the ID of the target device, and can use the key to encrypt the message to establish a secure pairing connection.

[0008] In a possible implementation, step 1) the initiator user makes a gesture in the air, that is, draws a graphic, and the graphic is drawn in one stroke; the graphic can be defined by the system or customized by the user.

[0009] In a possible implementation, step 2) generates a finger trajectory for the gesture made by the initiator user in the air, and the trajectory result can be represented as a trajectory point graph in a two-dimensional plane.

[0010] In a possible implementation, in step 3), during the execution of the graphic gesture, the initiator device tracks the movement of the finger and obtains a graphic formed by the tracking trajectory as a first-person view, denoted as G. At the same time, each surrounding device d i ,i∈{1,2,…n}, n=the number of surrounding devices, and also track the movement of the finger, and obtain a graph formed by the tracking trajectory from other views, denoted by G iSince the graphic is being executed by the initiator user on the target device, the graphics from the target device d t The trajectory graph G t The trajectory graph G from the initiator device is a similarity graph. Similar graphs have the same shape despite different sizes. In similar graphs, the corresponding edges are proportional, and the ratio of the corresponding edges of each edge is the same. Therefore, using the properties of similar graphs, the initiator device and the target device can generate a shared key. Given a device G or d i , and its finger tracking trajectory is recorded as P (i) =[X (i) ,Y (i) ] T , P (i) Divide into l segments evenly, and get a sequence of partition points. In order to make each partition point smooth, each partition point is further replaced by the average value of its neighbors (such as the average value of the two neighboring points before and after and itself). Among the partition points, any two points can construct a vector. According to the properties of similar figures, G or d i The ratio of the lengths of the two vectors in G t The length ratios of the corresponding vectors in are the same. Therefore, the combination of ratios can be used as the shared key K i In addition, to reduce the error caused by the difference in the graphs, the values ​​of the ratios are rounded.

[0011] In a possible implementation, step 4) may send information in the form of wireless channel broadcasting.

[0012] In a possible implementation, if the initiator device decrypts successfully in step 5), the target device is identified. During the decryption process, since the finger tracking trajectory cannot have absolute confidence, the captured image may be distorted. Therefore, G and G t It is not a completely similar graph. To further solve this problem, if decryption is unsuccessful, the initiator device is allowed to try to change the numbers in the key to decrypt the message. There are two rules for this process: one is that only one number can be changed at a time, and at most b numbers can be changed; the other is that the numbers in the key can only be corrected to the non-negative integers of their nearest neighbors e.

[0013] If encryption is not considered, the pairing method can be simplified to the following steps: 1) When a user (the initiator user) wants to interact with another user (the target device), the user waves his finger towards the target device he wants to interact with and directly clicks the device in the air; 2) Several surrounding devices (including the target device) use existing cameras to track the initiator user's gesture and record the tracking trajectory; 3) The surrounding devices extract feature vectors from the finger tracking trajectory; 4) The surrounding devices send the properties of the extracted feature vectors together with the device ID to the initiator device; 5) The initiator device checks the received feature vector, searches for and identifies the target device, obtains the target device ID, and establishes a fast pairing connection.

[0014] In a possible implementation, step 1) the initiator user directly clicks the target device that he wants to interact with in the air; clicking is an intuitive and natural gesture, and is often used to express the selection of an object in people's daily interactive activities.

[0015] In a possible implementation, step 2) generates a finger trajectory for the gesture made by the initiator user in the air, and the trajectory result can be represented as a trajectory point graph in a two-dimensional plane.

[0016] In a possible implementation, in step 3), during the execution of the graphic gesture, each surrounding device d i ,i∈{1,2,…n}, n=the number of surrounding devices, track the movement of the finger to get the tracking trajectory. Given a device d i And its finger tracking trajectory P (i) =[X (i) ,Y (i) ] T , extract a feature vector representing the trajectory trend from the trajectory in Refers to Y (i) The average value of the k points with the largest median value, Refers to Y (i) The average value of the k points with the smallest median value. Then calculate the eigenvector v (i) The absolute slope S(v (i) ), this slope represents v (i) The absolute tangent value of the angle between the direction of and the positive direction of the x-axis.

[0017] In a possible implementation, step 4) may send information in the form of wireless channel broadcasting.

[0018] In one possible implementation, in step 5), after tracking the finger motion and extracting the feature vector, the participating surrounding devices report their calculated S(v (i) According to the characteristics of the click gesture, the target device should capture a linear trajectory perpendicular to the x-axis, so the target device should have the largest S(v (i) The initiator device can compare the received value S(v (i)) to identify the target device and obtain the target device ID for quick pairing and connection. Beneficial Effects

[0019] The beneficial effect of the present invention is that by designing a user-friendly and secure device pairing system for smart mobile devices, users can achieve wireless connection of devices by simply waving their fingers at the target device. The invention uses the existing camera of the smart mobile device, designs a key generation and correction algorithm based on the finger tracking trajectory, and realizes the key negotiation of secure device pairing in the mobile augmented reality environment. The method designs the pairing method of smart mobile devices from a new perspective and can be implemented as software on existing devices. If the encryption function is not considered, the method can be simplified, and the target device can be found and quick pairing can be completed only by extracting the feature vector of the finger tracking trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is an application scenario diagram of an embodiment of the present invention.

[0021] Figure 2 It is a flow chart of a method for pairing a smart mobile device based on target tracking according to an embodiment of the present invention.

[0022] Figure 3 This is a flow chart of a method for fast pairing of smart mobile devices based on target tracking, without considering the encryption function in an embodiment of the present invention.

[0023] Figure 4 This is an experimental scenario of an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following is a further description with reference to examples and accompanying drawings.

[0025] The application scenario of the embodiment of the present invention is shown in FIG. Figure 1 As shown in FIG, a pairing method of smart mobile devices based on target tracking (denoted as WVC-S) is proposed. The overall framework is shown in Figure 2 , including the following steps:

[0026] Step 1: A user (initiator user) waves his finger towards the target device he wants to interact with and makes a specific gesture in the air, that is, draws a shape in the air. The shape is drawn in one stroke, including but not limited to a circle, multiple continuous circles, a heart shape, a triangle, a four-petal flower, a square or a rectangle, etc., and the shape can be defined by the system or customized by the user;

[0027] Step 2: The initiator device and surrounding devices all turn on the existing cameras, use the TLD method and manually assist in marking the starting point to track the finger movement, and record the tracking trajectory;

[0028] Step 3: The initiator device and surrounding devices extract the key from the finger tracking trajectory;

[0029] Step 3.1 The initiator device sends P (i) The average is divided into l segments, and a sequence of partition points is obtained. In order to make each partition point smooth, each partition point is further replaced with the average value of its neighbors (such as the average value of the two neighboring points before and after and itself). Then, d i The dividing point is One of the points Expressed as Where j′ is P (i) The corresponding index in , H is a 1×s kernel, N = {j'! -! s, ..., j', ... j'! +! s}. Round the key length ratio to the nearest integer and record it as Where q = (l! -1)! is the maximum length of the key. The obtained key is recorded as K = {r1, r2, ..., r q};

[0030] Step 3.2: Surrounding Equipment i ,i∈{1,2,…n}, n=the number of surrounding devices, and the finger tracking trajectory is recorded as P=[X,Y] T , firstly, mirror-flip the obtained finger tracking trajectory, and then use the same method as step 3.1 to obtain the key;

[0031] Step 4: The surrounding device encrypts the message using the key extracted in step 3.2 and sends it to the initiator device along with the device ID;

[0032] Step 5: The initiator device decrypts the received message using the key extracted in step 3.1. If the decryption is successful, it is the target device, thus establishing a secure pairing connection. During the decryption process, if the first decryption is unsuccessful, the initiator device is allowed to try to change the numbers in the key to decrypt the message. There are two rules in this process: one is that only one number can be changed at a time, and a maximum of b numbers can be changed; the other is that the numbers in the key can only be corrected to the non-negative integer of its nearest neighbor e. This rule can be implemented using a function AvailableBitNum(). Given a starting number, the function returns a set of correctable numbers that conform to the above rules;

[0033] Step 5.1: Given K = {r1, r2, …, r q}, b, e;

[0034] Step 5.2: Traverse from b'=1 to b;

[0035] Step 5.3: From i=1 to q (r1 to r q )Traverse;

[0036] Step 5.4: Call the AvailableBitNum(r i ,b′,e), return value C i ;

[0037] Step 5.5: In the return value C i Check in sequence whether the decryption can be successfully completed.

[0038] If the encryption function is not considered, the pairing method can be simplified (denoted as WVC). The overall framework can be seen in Figure 3 , including the following steps:

[0039] Step 1: A user (the initiator user) waves his finger towards the target device he wants to interact with and clicks the device in the air;

[0040] Step 2: The surrounding device turns on the existing camera, uses the TLD method and manually assists in marking the starting point to track the finger movement, and records the tracking trajectory;

[0041] Step 3: The surrounding devices extract feature vectors from the finger tracking trajectory;

[0042] Step 3.1: Surrounding Equipment i ,i∈{1,2,…n}, n=the number of surrounding devices, track the movement of the finger to get the tracking trajectory. Given a device d i And its finger tracking trajectory P (i) =[X (i) ,Y (i) ] T , extract a feature vector from the trajectory in Refers to Y (i) The average value of the k points with the largest median value, Refers to Y (i) The average value of the k points with the smallest median value;

[0043] Step 3.2: Calculate the feature vector v of each surrounding device (i) The absolute slope S(v (i) ), that is, v (i) The absolute tangent value of the angle between the direction of and the positive direction of the x-axis;

[0044] Step 4: The surrounding device sends the absolute slope and device ID of the feature vector extracted in step 3) to the initiator device;

[0045] Step 5: The initiator device finds the maximum slope value and its corresponding device ID among the absolute slopes of all received feature vectors. Here i* represents the ID of the target device.

[0046] The specific implementation of the present invention is described above with reference to the accompanying drawings. A person skilled in the art will appreciate that several improvements and equivalent substitutions may be made without departing from the spirit and principle of the present invention. The technologies and solutions after improvements and equivalent substitutions in the claims of the present invention all fall within the protection scope of the present invention. Experimental design:

[0047] In order to verify the effectiveness of the proposed method, the camera raw data are obtained using smart mobile devices such as Galaxy Note II, Galaxy Note III, Mate 8, iPhone 6s, and iPad 4. The parameters used in the method are set to l = 6, b = 2, and e = 1 according to experience.

[0048] A total of 10 volunteers (5 males and 5 females) were recruited for the experiment. The volunteers' ages ranged from 20 to 28 years old (average age 24.3). Before the experiment, the volunteers were introduced to the method. Figure 4 As shown, recorded as ground Figure 1 . The map consists of concentric circles. The difference in radius of all adjacent semicircles is 10 cm. The blue lines divide each semicircle into 12 parts at an angle. So the angle between adjacent blue lines is 15°. Five devices are placed randomly, and their positions are represented by green dots (points P1-P5). During the experiment, each volunteer was asked to perform a click gesture and a graphic gesture on any target device among the placed devices. The process was repeated 5 times, and a total of 100 samples were collected (10 volunteers × 2 gestures × 5 repetitions).

[0049] As shown in Table 1, the experimental results show that the average device selection accuracy of WVC is high, which can reach 94%. The average device selection accuracy of WVC-S is 79%, but it increases to 86% after the second attempt. From the experiment, we observed that the confidence of finger tracking is the main reason affecting the performance of WVC-S. Therefore, we use physical methods to assist in improving the accuracy of finger tracking results, thereby improving the accuracy of WVC-S. First, for predefined graphic gestures, we add an augmented reality interface shape, also known as a canvas, to the screen of the smart mobile device. The graphics on this screen can guide users to draw more standardized graphics in the air. Second, we provide a confirmation process. We set the selected device to flash its flash light to provide visual feedback to the initiator user, who can confirm the selection by checking whether the target device flashes. Therefore, if the correct target device cannot be identified in the first attempt, it can also avoid wrong device pairing and start the second attempt to establish a successful connection with the correct device.

[0050] In addition, we evaluated three factors that may affect the accuracy of device selection, including the device angle between devices, the distance between the device and the initiator's device, and the height of the device. We changed one factor at a time and kept the rest of the factors unchanged in each experiment. As shown in Table 1, when the device angle of the surrounding devices is small, such as 30°, WVC and WVC-S can achieve an average device selection accuracy of 80% and 85%, respectively; when the target device is far away from the initiator's device, such as 2m, WVC and WVC-S can achieve an average device selection accuracy of 100% and 89%, respectively; when the surrounding devices are at different heights, WVC and WVC-S can achieve an average device selection accuracy of 90% and 80%, respectively. Therefore, in the above situations, the methods proposed in the present invention can achieve good results. Table 1 Experimental results set up WVC accuracy (%) WVC-S accuracy (%) land Figure 1 94 86 Equipment angle 30° 80 85 The distance between the target device and the initiator device is 2m 100 89 Different equipment heights 90 80

Claims

1. A method for pairing smart mobile devices based on target tracking, characterized in that: The specific steps are 1) When a user (initiator device) wants to interact with another user (target device), the user waves his finger towards the target device and makes a specific gesture in the air, that is, draws a shape in the air; 2) The initiator device and several surrounding devices (including the target device) use existing cameras to track the initiator user's gestures and record the tracking trajectory; 3) The initiator device and surrounding devices extract the key from the finger tracking trajectory; 4) The surrounding device encrypts the message using the key extracted in step 3) and sends it to the initiator device together with the device ID; 5) The initiator device uses the key extracted in step 3) to decrypt the received message, find and identify the target device, obtain the ID of the target device, and can use the key to encrypt the message to establish a secure pairing connection. In a possible implementation, step 1) the initiator user makes a gesture in the air, that is, draws a graphic, and the graphic is drawn in one stroke; the graphic can be defined by the system or customized by the user. In a possible implementation, step 2) generates a finger trajectory for the gesture made by the initiator user in the air, and the trajectory result can be represented as a trajectory point graph in a two-dimensional plane. In a possible implementation, in step 3), during the execution of the graphic gesture, the initiator device tracks the movement of the finger and obtains a graphic formed by the tracking trajectory as a first-person view, denoted as G. At the same time, each surrounding device d i ,i∈{1,2,…n}, n=the number of surrounding devices, and also track the movement of the finger, and obtain a graph formed by the tracking trajectory from other views, denoted by G i Since the graphic is being executed by the initiator user on the target device, the graphics from the target device d t The trajectory graph G t The trajectory graph G from the initiator device is a similarity graph. Similar graphs have the same shape despite different sizes. In similar graphs, the corresponding edges are proportional, and the ratio of the corresponding edges of each edge is the same. Therefore, using the properties of similar graphs, the initiator device and the target device can generate a shared key. Given a device G or d i , and its finger tracking trajectory is recorded as P (i) =[X (i) ,Y (i) ] T , P (i) Divide into l segments evenly, and get a sequence of partition points. In order to make each partition point smooth, each partition point is further replaced by the average value of its neighbors (such as the average value of the two neighboring points before and after and itself). Among the partition points, any two points can construct a vector. According to the properties of similar figures, G or d i The ratio of the lengths of the two vectors in G t The length ratios of the corresponding vectors in are the same. Therefore, the combination of ratios can be used as the shared key K i In addition, to reduce the error caused by the difference in the graphs, the values ​​of the ratios are rounded. In a possible implementation, step 4) may send information in the form of wireless channel broadcasting. In a possible implementation, if the initiator device decrypts successfully in step 5), the target device is identified. During the decryption process, since the finger tracking trajectory cannot have absolute confidence, the captured image may be distorted. Therefore, G and G t It is not a completely similar graph. To further solve this problem, if decryption is unsuccessful, the initiator device is allowed to try to change the numbers in the key to decrypt the message. There are two rules for this process: one is that only one number can be changed at a time, and at most b numbers can be changed; the other is that the numbers in the key can only be corrected to the non-negative integers of their nearest neighbors e. If encryption is not considered, the pairing method can be simplified to the following steps: 1) When a user (the initiator user) wants to interact with another user (the target device), the user waves his finger towards the target device he wants to interact with and directly clicks the device in the air; 2) Several surrounding devices (including the target device) use existing cameras to track the initiator user's gesture and record the tracking trajectory; 3) The surrounding devices extract feature vectors from the finger tracking trajectory; 4) The surrounding devices send the properties of the extracted feature vectors together with the device ID to the initiator device; 5) The initiator device checks the received feature vector, finds and identifies the target device, obtains the ID of the target device, and establishes a fast pairing connection. In a possible implementation, step 1) the initiator user directly clicks the target device that he wants to interact with in the air; clicking is an intuitive and natural gesture, and is often used to express the selection of an object in people's daily interactive activities. In a possible implementation, step 2) generates a finger trajectory for the gesture made by the initiator user in the air, and the trajectory result can be represented as a trajectory point graph in a two-dimensional plane. In a possible implementation, in step 3), during the execution of the graphic gesture, each surrounding device d i ,i∈{1,2,…n}, n=the number of surrounding devices, track the movement of the finger to get the tracking trajectory. Given a device d i And its finger tracking trajectory P (i) =[X (i) ,Y (i) ] T , extract a feature vector representing the trajectory trend from the trajectory in Refers to Y (i) The average value of the k points with the largest median value, Refers to Y (i) The average value of the k points with the smallest median value. Then calculate the eigenvector v (i) The absolute slope S(v (i) ), this slope represents v (i) The absolute tangent value of the angle between the direction of and the positive direction of the x-axis. In a possible implementation, step 4) may send information in the form of wireless channel broadcasting. In one possible implementation, in step 5), after tracking the finger motion and extracting the feature vector, the participating surrounding devices report their calculated S(v (i) According to the characteristics of the click gesture, the target device should capture a linear trajectory perpendicular to the x-axis, so the target device should have the largest S(v (i) The initiator device can compare the received value S(v (i) ) to identify the target device and obtain the target device ID for quick pairing and connection.

Images