Interaction method and device of smart wearable device and mobile terminal, equipment and medium

Abstract

The application provides an interaction method and device between a smart wearable device and a mobile terminal, equipment and a storage medium, when the smart wearable device and the mobile terminal are in an extended screen interaction state, the method triggers an operation recognition process of the extended screen component when it is detected that the current posture of the mobile device matches a preset interaction operation action, improving the component operation accuracy. By collecting the corner point coordinates of the target corner point on the mobile terminal, the target extended screen component that the current user wants to operate is determined, realizing accurate identification and selection of the extended screen component; by collecting the current gesture action of the current user, and generating corresponding component operation instructions according to the gesture operation action matched with the current gesture action, the target operation is executed on the target extended screen component, improving the interaction efficiency between the smart wearable device and the mobile terminal, and enhancing the user's use experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent wearing, in particular to an interaction method and device of an intelligent wearing device and a mobile terminal, a computer device and a storage medium. BACKGROUND

[0002] The extended screen is a fast application engine running on a smart large screen device, which provides a new multi-screen interaction mode. It can be pulled up and interacted with by mobile client applications and applets in a way similar to screen projection operation, and can also be directly operated and interacted with by a large screen remote control. The extended screen is a mapping of the mobile terminal application interface on the display interface of the intelligent wearing device, which facilitates the quick uploading of mobile terminal services and content to the large screen, greatly reducing the cost of development, promotion and maintenance of large screen applications.

[0003] The intelligent wearing device can realize extended screen interaction with the mobile terminal based on augmented reality technology, and map the desktop application of the mobile terminal in the display interface of the intelligent wearing device. Although the intelligent wearing device currently realizes the extended screen display of the mobile terminal desktop application, it cannot directly operate the extended screen components in the display interface, so that the interaction effect of the intelligent wearing device and the mobile terminal cannot meet the user's demand, and the user's experience of the interaction function of the intelligent wearing device and the mobile terminal is poor.

[0004] Therefore, how to improve the use experience of the interaction function of the intelligent wearing device and the mobile terminal has become a technical problem to be solved. SUMMARY

[0005] The present application provides an interaction method and device of an intelligent wearing device and a mobile terminal, a computer device and a storage medium, which aims to improve the use experience of the interaction function of the intelligent wearing device and the mobile terminal.

[0006] In a first aspect, the present application provides an interaction method of an intelligent wearing device and a mobile terminal, comprising:

[0007] The intelligent wearing device detects the current posture of the mobile terminal when the intelligent wearing device and the mobile terminal are in an extended screen interaction state;

[0008] When the current posture of the mobile terminal matches the preset interaction operation action, the intelligent wearing device collects the corner point coordinates of the target corner point on the mobile terminal;

[0009] Based on the extended screen component region corresponding to the corner point coordinates, the intelligent wearing device determines the target extended screen component;

[0010] When the current gesture action of the current user matches the preset gesture operation action, the intelligent wearing device executes the target operation on the target extended screen component based on the component operation instruction corresponding to the gesture operation action.

[0011] In a second aspect, the present application provides an interaction device of a smart wearable device and a mobile terminal, comprising:

[0012] a posture detection module, configured to detect a current posture of the mobile terminal when the smart wearable device is in an extended screen interaction state with the mobile terminal;

[0013] an angle point coordinate collection module, configured to collect angle point coordinates of a target angle point on the mobile terminal when it is detected that the current posture of the mobile terminal matches a preset interaction operation action;

[0014] a component determination module, configured to determine a target extended screen component based on an extended screen component region corresponding to the angle point coordinates;

[0015] a component operation module, configured to execute a target operation on the target extended screen component based on a component operation instruction corresponding to a preset gesture operation action when a current gesture action of a current user matches the gesture operation action.

[0016] In a third aspect, the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the computer program is executed by the processor to implement the steps of the interaction method of the smart wearable device and the mobile terminal as described above.

[0017] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the interaction method of the smart wearable device and the mobile terminal as described above.

[0018] The application provides an interaction method and device of a smart wearable device and a mobile terminal, equipment and a storage medium, the method comprises that the smart wearable device detects a current posture of the mobile terminal when the smart wearable device and the mobile terminal are in an extended screen interaction state; when it is detected that the current posture of the mobile terminal matches a preset interaction operation action, the smart wearable device collects an angle point coordinate of a target angle point on the mobile terminal; based on an extended screen component region corresponding to the angle point coordinate, the smart wearable device determines a target extended screen component; when a current gesture action of a current user matches a preset gesture operation action, the smart wearable device executes a target operation on the target extended screen component based on a component operation instruction corresponding to the gesture operation action. In the above manner, when the smart wearable device and the mobile terminal are in the extended screen interaction state, the smart wearable device matches the current posture of the mobile terminal with the preset interaction operation action by detecting the current posture of the mobile terminal; and when the current posture matches the preset interaction operation action, the smart wearable device triggers an operation recognition process of the extended screen component, improves the component operation accuracy, and avoids operation mis-triggering. By collecting the angle point coordinate of the target angle point on the mobile terminal, the smart wearable device determines the target extended screen component that the current user wants to operate according to the extended screen component region where the angle point coordinate is located, and realizes accurate identification and selection of the extended screen component; by comparing the current gesture action of the current user with the preset gesture operation action, the smart wearable device determines the gesture operation action that matches the current gesture action, and then generates the component operation instruction corresponding to the gesture operation action, and executes the target operation on the target extended screen component according to the component operation instruction, so that the current user can directly operate the extended screen component on the display interface of the smart wearable device, improves the interaction efficiency of the smart wearable device and the mobile terminal, and enhances the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creating laborious work.

[0020] Figure 1 A flowchart of a first embodiment of an interaction method of a smart wearable device and a mobile terminal provided by the embodiments of the present application;

[0021] Figure 2 A flowchart of a second embodiment of an interaction method of a smart wearable device and a mobile terminal provided by the embodiments of the present application;

[0022] Figure 3 A detailed flowchart of a first step of a second embodiment of an interaction method of a smart wearable device and a mobile terminal provided by the embodiments of the present application;

[0023] Figure 4 A flowchart of a third embodiment of a method for interaction between a smart wearable device and a mobile terminal provided by the present application;

[0024] Figure 5 A distribution diagram of an extended screen component in an extended screen mode activation state provided by the present application;

[0025] Figure 6 A distribution diagram of a relative coordinate relationship between an extended screen component region and a mobile terminal provided by the present application;

[0026] Figure 7 A distribution diagram of a relative size relationship between second size information of an extended screen component and first size information of a mobile terminal provided by the present application;

[0027] Figure 8 A distribution diagram of another relative size relationship between second size information of an extended screen component and first size information of a mobile terminal provided by the present application;

[0028] Figure 9 A structural diagram of a first embodiment of an interaction device between a smart wearable device and a mobile terminal provided by the present application;

[0029] Figure 10 A structural schematic diagram of a computer device provided by the present application.

[0030] The object, function features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0032] The flowchart shown in the drawings is only an example, and does not necessarily include all the contents and operations / steps, nor does it necessarily be executed in the described order. For example, some operations / steps can be decomposed, combined or partially combined, so that the actual execution order can be changed according to the actual situation.

[0033] Some embodiments of the present application will be described in detail below with reference to the drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

[0034] Please refer toFigure 1 , Figure 1 A flowchart of a first embodiment of an interaction method between a smart wearable device and a mobile terminal provided by an embodiment of the present application is shown.

[0035] As shown in the figure, the interaction method between the smart wearable device and the mobile terminal includes steps S101-S104. Figure 1

[0036] Step S101, the smart wearable device detects the current posture of the mobile terminal when the smart wearable device and the mobile terminal are in an extended screen interaction state.

[0037] In an embodiment, the smart wearable device refers to a wearable glasses device with an independent operating system, which can realize various functions through software installation. For example, AR glasses (Augmented Reality goggles), VR glasses (Virtual Reality), AR helmet, etc.

[0038] In an embodiment, the mobile terminal or mobile communication terminal refers to a computer device that can be used in mobile, including mobile phones, notebooks, tablets, etc.

[0039] In an embodiment, the smart wearable device can detect the current posture of the mobile terminal through an image acquisition device.

[0040] In an embodiment, the image acquisition device refers to a device component that can realize image acquisition and image processing, such as a camera, an infrared sensor, a laser scanner, etc.

[0041] In an embodiment, the image acquisition device can be carried by the smart wearable device itself, or can be an independent device installed on the smart wearable device.

[0042] In an embodiment, the image acquisition device is in communication connection with the smart wearable device, used to acquire image or video data of the mobile device and the user's hand, and according to the acquired image and video data, to identify the posture, coordinates and actions of the mobile device and the user's hand, etc. When the image acquisition device identifies that the mobile terminal moves into the acquisition picture of the image acquisition device, it can identify the contour information and corner point coordinate information of the mobile terminal, and can also track and identify the moving track information of the mobile terminal, etc.

[0043] Step S102, when the current posture of the mobile terminal matches the preset interaction operation action, the corner point coordinates of the target corner point on the mobile terminal are acquired.

[0044] ​In an embodiment, the display interface of the smart wearable device can be a two-dimensional coordinate system, the upper left corner of the display interface can be the origin of the coordinate system, the two sides of the display interface can be the coordinate axes of the two-dimensional coordinate system, and the scales of the coordinate axes can be set according to the length and width of the display interface. Thus, the contour coordinates and the corner point coordinates of the mobile terminal can be tracked in the display interface of the smart wearable device.

[0045] In an embodiment, the image acquisition device of the smart wearable device can track and calculate the contour and the corner point coordinates of the mobile terminal in real time. If it is detected that the current posture of the mobile terminal matches the interactive operation action, i.e., the similarity between the current posture and the interactive operation action is greater than or equal to a similarity threshold (e.g., 90%), the current corner point coordinates of the mobile terminal are acquired by the image acquisition device.

[0046] In an embodiment, the interactive operation action can include double-clicking, remaining in a stationary state for a preset time length, sliding, etc.

[0047] In an embodiment, the corner point coordinates of the mobile terminal can be the edge lines of the sides of the mobile terminal according to the contour coordinates of the mobile terminal. The intersection of two intersecting edge lines can be taken as the corner point coordinates of the mobile terminal.

[0048] In an embodiment, the four corners of the mobile terminal are generally designed to be arc-shaped. The intersection coordinates of the edge lines of the mobile terminal can be calculated according to the contour coordinates of the mobile terminal. The four intersections correspond to the four arc corners of the mobile terminal, respectively. The arc point closest to the corresponding intersection coordinates on each arc corner can be taken as the corner point.

[0049] In an embodiment, the mobile terminal generally has four sides, i.e., the corner point coordinates of the four corners of the mobile terminal can be tracked. In order to improve the recognition accuracy of the expansion screen assembly, one of the four corners can be preset as a target corner, such as the upper left corner of the mobile terminal.

[0050] In an embodiment, the smart wearable device can recognize the contour of the mobile terminal by the image acquisition device and activate the expansion screen function of the mobile terminal, so as to realize the expansion screen interaction between the smart wearable device and the mobile terminal. In the expansion screen interaction state, the smart wearable device acquires the corner point coordinates of the mobile terminal, such as the corner point coordinates of the upper right corner, by the image acquisition device.

[0051] In step S103, a target expansion screen assembly is determined based on the expansion screen assembly region corresponding to the corner point coordinates.

[0052] In an embodiment, when it is detected that the current posture of the mobile terminal matches the preset interaction operation action, the corner point coordinates of the target corner point of the mobile terminal are collected, the extended screen component region in which the corner point coordinates of the target corner point are located is determined as a target region, and the target extended screen component corresponding to the target region is the extended screen component on which the current user wants to perform an operation.

[0053] The extended screen component is a high-performance cross-terminal development framework for a smart large-screen device, and is a fast application engine running on a smart wearable device, which can help developers quickly build application services of the smart wearable device and provide multi-screen interaction capability.

[0054] In an embodiment, the extended screen component can be a mapping of the application interface of the mobile terminal in the display interface of the smart wearable device, so as to facilitate the quick display of the services and content of the mobile terminal in the display interface of the smart wearable device. The extended screen component includes an application component of the mobile terminal, which is used to display the application display content of the mobile terminal, such as a video played by a video software, a real-time barrage, a reading page of a novel software, and the like.

[0055] In an embodiment, the extended screen component further includes a function module component of the smart wearable device, which is used to facilitate the user to query and call the functions and services of the smart wearable device, such as product instructions, a list of connectable devices, a network list, and the like.

[0056] Further, based on the corner point coordinates, the current interaction action of the current user operating the mobile terminal is identified, and when the current interaction action satisfies a trigger condition corresponding to the preset interaction operation action, the current interaction action is determined as the interaction operation action.

[0057] In an embodiment, the interaction operation action can be identified according to the coordinates of the preset corner of the mobile terminal. For example, when the current user operates the mobile terminal to make a double-click action, that is, the coordinates of the target corner point of the mobile terminal point to the same coordinate position within a preset time; or when the image collection device identifies that the corner of the mobile terminal has an obvious knocking action, the landing point coordinates of the target corner point in the knocking action are collected, and if the landing point coordinates are located in a certain extended screen component region, it is determined that the extended screen component corresponding to the extended screen component region is the target extended screen component; or when it is identified that the corner point coordinates of the target corner point stay in a certain extended screen component region for a time reaching a preset threshold, it can be determined that the extended screen component region corresponding to the extended screen component is the target extended screen component on which the current user wants to view or perform an operation.

[0058] In an embodiment, if it is detected that the current user operates the mobile terminal to make an interaction operation action, but the coordinate region corresponding to the corner point coordinates does not belong to the extended screen component region, it can be determined that the interaction operation action is invalid, and no operation instruction is generated.

[0059] In an embodiment, if it is detected that the current user operates the mobile terminal to make an interactive operation action, but the coordinate region corresponding to the corner point coordinates does not belong to the extended screen component region, the distance between the corner point coordinates and the midpoint positions in each extended screen component region can also be calculated, the extended screen component region corresponding to the shortest distance is determined as the landing region of the corner point coordinates, and thus the extended screen component corresponding to the landing region is determined as the target extended screen component.

[0060] In an embodiment, the extended screen component region is the display region of each extended screen component in the display interface of the smart wearable device. With the display interface as a two-dimensional coordinate system, the four corner point coordinates of the extended screen component region can be determined, and thus the side length of the extended screen component region can be calculated according to the four corner point coordinates, such as the coordinate distance between two corner point coordinates in the same row as the length of the extended screen component region and the coordinate distance between two corner point coordinates in the same column as the width of the extended screen component region. Thus, whether the coordinate region corresponding to the corner point coordinates belongs to the extended screen component region can be determined according to the coordinate relationship between the corner point coordinates and the region coordinates of the extended screen component region. If the corner point coordinates are located within the range of any extended screen component region, it is determined that the coordinate region corresponding to the corner point coordinates belongs to the extended screen component region. If the corner point coordinates are not located within the range of any extended screen component region, it is determined that the coordinate region corresponding to the corner point coordinates does not belong to the extended screen component region.

[0061] In step S104, when the current gesture action of the current user matches the preset gesture operation action, a target operation is performed on the target extended screen component based on the component operation instruction corresponding to the gesture operation action.

[0062] In an embodiment, the current gesture action of the current user can be collected by an image collection module. The current user can make the preset gesture operation action in the image collection picture of the image collection module, so that the smart wearable device can recognize the operation of the current user on the target extended screen component.

[0063] In an embodiment, the preset gesture operation action can include a gesture action of a single or multiple fingers or the whole hand, such as a double-click action of the thumb, a sliding action of the fingertip of the index finger, and the like.

[0064] Further, a preset gesture instruction library is obtained, wherein the gesture instruction library includes at least one gesture operation action and the component operation instruction corresponding to each gesture operation action. The current gesture action is compared with each gesture operation action in the gesture instruction library based on a gesture recognition algorithm to determine the similarity between the current gesture action and each gesture operation action. When the similarity is greater than or equal to a preset similarity, the gesture operation action that matches the current gesture action is determined based on the similarity.

[0065] In an embodiment, each gesture operation action corresponding to each component operation instruction can be preset according to at least one component operation instruction of the extended screen assembly. The smart wearable device can pre-record gesture operation actions through the image acquisition device, and uniquely associate each gesture operation action with a corresponding component operation instruction, so that each component operation instruction corresponds to a gesture operation action, and store the gesture instruction library in the data storage module of the smart wearable device.

[0066] In an embodiment, one component operation instruction can also correspond to multiple different gesture operation actions. For example, the component detail page viewing instruction can correspond to the gesture double-click operation action, which can be a double-click action of the thumb tip of the hand or a double-click action of the index finger tip.

[0067] In an embodiment, the same gesture operation action can also correspond to multiple component operation instructions. For example, the component detail page viewing instruction and the component detail page exiting instruction can both correspond to the gesture double-click operation action. The gesture double-click operation action detected outside the component detail page (i.e., not in the component detail page) generates the component detail page viewing instruction, and the gesture double-click operation action detected inside the component detail page (i.e., already in the component detail page) generates the component detail page exiting instruction.

[0068] In an embodiment, the gesture recognition algorithm can be used to calculate the action trajectory of the current gesture action, and then compare the action trajectory of the current gesture action with the action trajectory of each gesture operation action in the gesture instruction library to calculate the similarity between the action trajectory of the current gesture action and the action trajectory of each gesture operation action.

[0069] If the similarity between the action trajectory of the current gesture action and the action trajectory of a certain gesture operation action is greater than or equal to a preset similarity, it is determined that the current gesture action is the gesture operation action, and the two are matched. If the similarity between the action trajectory of the current gesture action and the action trajectory of each gesture operation action in the gesture instruction library is less than the preset similarity, it is determined that the current gesture action does not match each gesture operation action, and no component operation instruction can be generated.

[0070] In an embodiment, if the similarity between the action trajectory of the current gesture action and the action trajectory of multiple gesture operation actions is greater than or equal to a preset similarity, it is determined that the gesture operation action with the highest similarity matches the current gesture action.

[0071] In an embodiment, the action trajectory of the current gesture action can be the coordinate trajectory of the hand key point (joint point, fingertip, etc.) over time.

[0072] In an embodiment, upon detecting that the current gesture action matches the preset gesture operation action, the smart wearable device generates a corresponding component operation instruction according to the gesture operation action, and performs a corresponding operation on the target extended component according to the component operation instruction.

[0073] In an embodiment, upon detecting that the current gesture action matches the preset gesture operation action, the smart wearable device can determine the component operation instruction corresponding to the current gesture operation action according to the preset gesture instruction library, and then generate the component operation instruction. For example, a click action can represent entering a component detail page, and a detail page opening instruction of the extended screen component is generated at this time; a swipe action represents exiting the component detail page, and a detail page exit instruction of the extended screen component is generated at this time, and the like.

[0074] For example, if the double-click operation action corresponds to the component viewing instruction of the extended screen component, the viewing instruction of the target extended screen component is generated according to the target extended screen component pointed by the double-click operation action, such as the component operation instruction "music player--view", and the component operation instruction can be used to enter the detail page of the music player extended screen component.

[0075] If the swipe operation action corresponds to the switching instruction of the extended screen component, upon detecting that the mobile terminal makes the swipe operation action in the current extended screen component detail page, the component detail page switching instruction is generated, and the current extended screen component detail page is switched to another extended screen component detail page.

[0076] In a specific embodiment, the process of performing the target extended screen component operation according to the current gesture action of the current user can be as follows: the thumb double-clicks the extended screen component to indicate that the detail page of the extended screen component is to be viewed, upon recognizing by the image acquisition module that the user's thumb makes the double-click action in the component page, the gesture operation action (thumb-double-click) corresponding to the double-click action is matched from the gesture instruction library, the smart wearable device generates the component operation instruction (entering the component detail page instruction) corresponding to the gesture operation action, and enters the detail page of the target extended screen component according to the component operation instruction. Upon capturing the swipe action (corresponding to the component switching instruction action) of the user gesture in the detail page of the target extended screen component, the smart wearable device generates the component switching instruction, and switches to display the detail page of the next extended screen component. Upon capturing the double-click action of the user's thumb in the detail page of the extended screen component, the back instruction can be matched according to "component detail page-double-click action-exit component detail page", and the current displayed detail page of the extended screen component is exited according to the back instruction.

[0077] This embodiment provides an interaction method between a smart wearable device and a mobile terminal. When the smart wearable device and the mobile terminal are in an extended screen interaction state, the method uses an image acquisition device to detect the current posture of the mobile terminal. When the current posture matches a preset interactive action, an operation recognition process for the extended screen component is triggered, improving the accuracy of component operation and avoiding accidental triggering. The image acquisition device collects the corner coordinates of a target corner point on the mobile terminal. Based on the extended screen component area where the corner coordinates are located, the target extended screen component that the user wants to operate is determined, achieving accurate identification and selection of the extended screen component. Based on the identified interactive action, a corresponding component operation command is generated. Then, based on the component operation command, the target operation is executed on the target extended screen component. This allows the user to directly operate the extended screen component on the display interface of the smart wearable device, improving the interaction efficiency between the smart wearable device and the mobile terminal and enhancing the user experience.

[0078] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a second embodiment of an interaction method between a smart wearable device and a mobile terminal, provided in this application.

[0079] like Figure 2 As shown, based on the above Figure 1 In the illustrated embodiment, prior to step S101, the method further includes:

[0080] Step S201: Based on the image acquisition device, identify the preset unbinding action made by the current user and generate a device unbinding command;

[0081] In one embodiment, in extended screen mode, the mobile terminal and the extended screen component are bound together. When the mobile terminal moves, the extended screen component moves synchronously and is anchored around the perimeter of the mobile terminal. Currently, the user can only control the extended screen component by operating the desktop application on the mobile terminal screen. To control the extended screen component on the display interface of the smart wearable device, the mobile terminal needs to be unbound from the extended screen component.

[0082] In one embodiment, the binding state between the mobile terminal and the extended screen component can be released by designing a specific preset unbinding action. For example, double-tapping within the screen area of ​​the mobile terminal.

[0083] In one embodiment, when the image acquisition device of the smart wearable device recognizes that the current user has made a preset unbinding action, it triggers an unbinding event between the mobile terminal and the extended screen component, and the smart wearable device generates a device unbinding command to release the binding state between the mobile terminal and the extended screen component.

[0084] Furthermore, such as Figure 3As shown, the step S201 specifically includes:

[0085] Step S2011, based on the image acquisition device, acquiring the hand action pictures of the current user, obtaining a hand picture group, wherein the hand action picture group includes at least two hand action pictures;

[0086] In an embodiment, the hand action pictures of the current user can be acquired by the image acquisition device of the smart wearable device, and a hand action picture group is obtained. The picture group can be in the form of a video stream, that is, a group of hand action pictures taken in time sequence.

[0087] In an embodiment, the shooting frame rate of the image acquisition device can be set according to the tracking effect of the hand action, such as setting the shooting frame rate to 25FPS (frames per second), that is, shooting 25 frames of pictures per second. In this way, it can be ensured that the hand action of the user can be completely captured, and the processing amount of picture data can be reduced, and the recognition efficiency and action response efficiency of the hand action can be improved.

[0088] Step S2012, based on a preset key point tracking algorithm, extracting the point coordinates of the target key points in the hand action pictures, obtaining time sequence coordinate data of the target key points;

[0089] In an embodiment, the time sequence coordinate data refers to the time sequence data of the point coordinates of the target key points. The time sequence data is a series of data obtained at different time periods or time points in time sequence, which describes the change of the observed object with time.

[0090] In an embodiment, according to the preset key point tracking algorithm, the target key points in each hand action picture are extracted, such as finger tips, joint points, etc., and the point coordinates of the target key points are tracked and calculated, and a group of coordinate change data of the target key points in time sequence is obtained.

[0091] Step S2013, when detecting that there are at least two coincident point coordinates in the time sequence coordinate data, calculating the acquisition time interval of each of the coincident point coordinates based on the acquisition time corresponding to each of the coincident point coordinates;

[0092] In an embodiment, when it is detected that there are at least two coincident point coordinates in the time sequence coordinate data, it means that the current user has clicked at least twice at the same position on the mobile phone. At this time, according to the time sequence coordinate data, the coordinate acquisition time of each coincident point coordinate is determined, and then the acquisition time interval of each coincident point coordinate is calculated.

[0093] In an embodiment, the coincident point coordinates can be two point coordinates with the same coordinates, or two point coordinates with a distance less than a set value, for example, two point coordinates with a distance less than 5 pixel points can be considered as the coincident point coordinates.

[0094] In step S2014, when the collection time interval is less than the preset time interval, it is determined that the hand movement of the current user is the preset unbinding action, and the device unbinding instruction is generated.

[0095] In an embodiment, according to the preset unbinding action, the preset time interval is set. If the preset unbinding action is double-clicking, the preset time interval for determining double-clicking can be set as 1s, that is, if the current user clicks the same position twice within 1s, the preset unbinding action is triggered.

[0096] In an embodiment, when the collection time interval of each coincident point coordinate is less than the preset time interval, it is determined that the hand movement of the current user triggers the preset unbinding action. At this time, the smart wearable device generates the device unbinding instruction, and according to the unbinding instruction, the binding state between the mobile terminal and the extended screen assembly is released.

[0097] For example, double-clicking is fast and continuous clicking twice in the same area range. The hand key point algorithm can obtain 21 key points of a human hand. The position coordinates of the thumb tip and the first joint are tracked in each frame. Assuming that the interval time is defined as t frames, if the thumb tip and the first joint of the thumb appear twice in the same position within t frames and the thumb is in the screen area of the mobile terminal, the preset unbinding action is triggered, and the binding state between the mobile terminal and the extended screen assembly is released.

[0098] In an embodiment, the area range can be a range area radiating a certain coordinate distance outward with the falling point coordinates of the first click as the center, that is, the coordinate distance of the falling point coordinates of the continuous two clicks is determined. For example, taking the falling point coordinates of the first click as the reference point, if the coordinate distance between the falling point coordinates of the second click and the reference point is less than or equal to a limited value (such as five pixel point distances), and the time interval of the two clicks is within t frames (including t frames), it can be determined as a double-click action.

[0099] In an embodiment, when the collection time interval is greater than the preset time interval, it is determined that the hand movement of the current user does not trigger the preset unbinding action. At this time, the smart wearable device does not respond, and continues to collect the hand movement picture of the user through the image collection module.

[0100] Further, after detecting that there are at least two coincident point coordinates in the time-series coordinate data, the method further includes: determining, based on the time-series coordinate data, interval frame numbers between the hand action pictures corresponding to the coincident point coordinates; and determining that the hand action of the current user is the preset unbinding action and generating the device unbinding instruction when the interval frame numbers are less than a preset interval frame number.

[0101] In an embodiment, the pictures collected by the image collection device are arranged in time sequence, and the collection time intervals of adjacent two pictures are the same. For example, if the frame rate is set to 25 FPS, the collection time interval between adjacent two pictures in the collected pictures is 1 / 25 second. Therefore, the collection interval time of two pictures can be calculated according to the interval frame number.

[0102] In an embodiment, when it is detected that there are at least two coincident point coordinates in the time-series coordinate data, the interval frame numbers between the hand action pictures corresponding to the coincident point coordinates can be calculated. For example, if the preset unbinding action is double-clicking, the interval frame numbers between the hand action pictures corresponding to adjacent two coincident point coordinates are calculated. If the interval frame numbers are less than a preset interval frame number, it is determined that the hand action of the current user triggers the preset unbinding action. When the preset unbinding action is triggered, the smart wearable device generates a device unbinding instruction. If the interval frame numbers are greater than the preset interval frame number, it is determined that the hand action of the current user does not trigger the preset unbinding action, and the smart wearable device does not need to respond to this hand action.

[0103] In step S202, the binding state between the mobile terminal and the extended screen component is released based on the device unbinding instruction, so that the smart wearable device and the mobile terminal are in the extended screen interaction state.

[0104] In an embodiment, after the mobile terminal and the extended screen component are released from the binding state, the extended screen component is fixedly displayed on the display screen of the smart wearable device and does not move with the mobile terminal. At this time, the mobile terminal can be used as a point-touch device. When the user holds the mobile terminal to move or tap, the smart wearable device tracks and calculates a corner point coordinate (for example, a left upper corner coordinate) of the mobile terminal in real time and recognizes the action of the mobile terminal.

[0105] In this embodiment, the hand action made by the current user is recognized through the image acquisition device, so that when it is determined that the hand action made by the current user is a preset unbinding action, a device unbinding instruction is triggered, and the binding state of the mobile terminal and the extended screen component is released according to the device unbinding instruction, so that the extended screen component is fixedly mapped in the display interface of the smart wearable device, and then the smart wearable device can track and identify the feature points of the mobile terminal, identify the interaction action made by the current user in controlling the mobile terminal, so that the current user can directly control the extended screen component through the display interface of the smart wearable device, and the convenience of the interaction function of the smart wearable device and the mobile terminal is improved.

[0106] Please refer to Figure 4 , Figure 4 The flowchart of a third embodiment of an interaction method between a smart wearable device and a mobile terminal provided by the embodiments of the present application is shown.

[0107] As Figure 4 shown, based on the above Figure 3 embodiment, before the step S201, the method further comprises:

[0108] Step S301, acquiring the contour coordinates of the mobile terminal based on the image acquisition device;

[0109] In an embodiment, the smart wearable device identifies the contour of the mobile terminal through the image acquisition device, and then calculates the contour coordinates of the contour of the mobile terminal in the two-dimensional coordinate system of the smart wearable device through a computer vision algorithm.

[0110] In an embodiment, computer vision refers to enabling computers and systems to obtain meaningful information from images, videos and other visual inputs, and taking actions or providing suggestions based on the information. Computer vision trains machines to recognize objects, object distances, object motion or stillness, and whether there are problems in images, etc. through cameras, data and computer vision algorithms.

[0111] In an embodiment, the computer vision algorithm can include feature extraction, feature point tracking and coordinate calculation, etc.

[0112] In an embodiment, feature extraction and matching are an important task in many computer vision applications, and are widely used in motion structure, image retrieval, target detection and other fields. The feature can be a specific structure in an image, such as a point, an edge or an object.

[0113] In one embodiment, the feature point tracking algorithm does not track the target as a whole, but rather matches and tracks a set of feature points (such as boundary lines, centroids, corner points, etc.) in consecutive frames of the image. Feature-based tracking methods mainly include two stages: feature extraction and matching. The main advantage of this type of tracking method is that even if the target in the scene is partially occluded, as long as the feature points are visible, continuous tracking of the target can be achieved.

[0114] In one embodiment, coordinate calculation means calculating the position coordinates of the target feature points in the coordinate system. Since the images captured by the image acquisition device of the smart wearable device are all within the display interface of the smart wearable device, the images captured by the image acquisition device all correspond to the same coordinate system. As long as the contour feature points of the mobile terminal are extracted, and then the position of the contour feature points in the image is calculated, the contour coordinates of the mobile terminal can be obtained.

[0115] Step S302: Obtain the relative positional relationship between at least one extended screen component and the mobile terminal;

[0116] In one embodiment, such as Figure 5 As shown, the relative position of the extended screen components and the mobile terminal depends on the position of the application corresponding to each extended screen component on the mobile terminal screen. The extended screen components can be distributed around the mobile terminal.

[0117] In one embodiment, when mapping the extended screen components, the distribution positions of the extended screen components are arranged sequentially according to the distribution positions and arrangement order of the applications on the mobile terminal. When multiple applications exist, their distribution on the mobile terminal is generally arranged in rows and columns, meaning the distribution position of each application on the mobile terminal can be represented by rows and columns. For example, Application 1 - first row, third column, etc.

[0118] In one embodiment, the extended screen components can be directly mapped onto the display screen of the smart wearable device in their original positions according to the distribution of applications, or they can be mapped onto the sides of the mobile terminal in a column and row manner. For example, the extended screen components corresponding to the applications in the first row are mapped onto the top of the mobile terminal, and the extended screen components are arranged according to the order of the applications; the extended screen components corresponding to the applications in the first column are mapped onto the left side of the mobile terminal, and so on.

[0119] In an embodiment, in the display interface of the smart wearable device, the mobile terminal outline can be taken as the center, and the desktop applications corresponding to the extended screen components can be arranged in sequence on the side of the mobile terminal according to the positions of the desktop applications on the mobile terminal screen. For example, if the first desktop application is located at the middle position on the left side of the mobile terminal screen, the extended screen component corresponding to the first desktop application is enlarged and displayed at the middle position on the left side of the mobile terminal outline in the display interface of the smart wearable device.

[0120] In step S303, the extended screen components are mapped and displayed in the display interface of the smart wearable device based on the outline coordinates of the mobile terminal and the relative position relationship, so as to activate the extended screen mode, wherein the mobile terminal and the extended screen components are in a binding state in the extended screen mode.

[0121] In an embodiment, according to the outline coordinates of the mobile terminal and the relative position relationship, the area coordinates of the extended screen component area corresponding to each extended screen component in the display interface of the smart wearable device are calculated, and then the extended screen components are mapped and displayed at the corresponding extended screen component areas in the display interface, so as to complete the activation of the extended screen mode. For example, as shown in FIG. 3, the first extended screen component is located on the left side of the mobile terminal, and the left side of the mobile terminal only has the first extended screen component. At this time, the outline coordinates of the mobile terminal can be ((100, 10), (150, 10), (100, 110), (150, 110)). According to the relative position relationship between the mobile terminal and the first extended screen component, the area coordinates of the extended screen component area corresponding to the first extended screen component can be calculated as ((10, 10), (100, 10), (10, 110), (100, 110)). Figure 6

[0122] Further, the first size information of the mobile terminal and the second size information of the extended screen components are obtained; based on the first size information, the second size information, the relative position relationship and the outline coordinates of the mobile terminal, the diagonal coordinate information of the extended screen components in the display interface is calculated; and based on the diagonal coordinate information, the extended screen component areas corresponding to the extended screen components are determined.

[0123] In an embodiment, the first size information of the mobile terminal includes the actual device size of the mobile terminal, including the length size and the width size.

[0124] In an embodiment, the first size information of the mobile terminal is known, which can be obtained by a size measuring tool or by querying the official website of the manufacturer of the mobile terminal. The user can preset the second size information of each extended screen component according to the first size information of the mobile terminal, for example, the display size of the extended screen components can be enlarged in proportion according to the number of the extended screen components, so that the extended screen components can be displayed completely and reasonably around the mobile terminal.​

[0125] For example, if the first row of the mobile terminal screen has four applications, the expansion screen components of the four applications are mapped on the upper side of the mobile terminal, the first size information of the mobile terminal is 8x16 cm, i.e., the length of the upper side of the mobile terminal is 8 cm, and the second size information of the four expansion screen components can be maximally 2xncm, i.e., the length of the four expansion screen components can be maximally 2 cm, and the width can be adjusted according to actual application (e.g. Figure 7 as shown); or the length of the four expansion screen components can be maximally 4xncm, i.e., the length of the four expansion screen components can be maximally 4 cm (e.g. Figure 8 as shown), wherein the two expansion screen components corresponding to the first and fourth applications are respectively distributed on the two upper diagonal positions of the mobile terminal.

[0126] In an embodiment, according to the relative position relationship between the mobile terminal and the expansion screen components, the magnification of the expansion screen components can be set, for example, if the mobile terminal is rectangular, the magnification of the expansion screen components on the long side can be set to be larger, and the magnification of the expansion screen components on the short side can be set to be smaller.

[0127] In an embodiment, according to the first size information, the second size information, the relative position relationship, and the outline coordinates of the mobile terminal, the diagonal coordinate information of the component frame of each expansion screen component in the display interface of the smart wearable device, such as the upper left corner coordinate and the lower right corner coordinate, can be calculated; and according to the diagonal coordinate information, the area range of each expansion screen component in the display interface of the smart wearable device can be represented.

[0128] For example, according to the outline coordinates of the mobile terminal, the pixel size of the mobile terminal in the two-dimensional coordinate system, i.e., the number of pixels occupied by the length and the number of pixels occupied by the width, can be calculated. The ratio of the first size information to the pixel size is the size ratio of the actual size of the device to the pixel size in the two-dimensional coordinate system, and the ratio of the second size information to the size ratio is the pixel size of the expansion screen component. According to the relative position relationship and the outline coordinates of the mobile terminal, the starting point coordinates of each expansion screen component in the display interface can be obtained, and according to the starting point coordinates and the pixel size corresponding to each expansion screen component, the diagonal coordinate information of the component frame of each expansion screen component can be calculated, so as to obtain the expansion screen component area corresponding to each expansion screen component.

[0129] In this embodiment, the profile coordinates of the mobile terminal are collected by the image collection device to determine the position of the mobile terminal in the display interface of the smart wearable device, so as to determine the position of the extended screen assembly in the display interface of the smart wearable device according to the relative position relationship between the extended screen assembly and the mobile terminal, and then accurately map the extended screen assembly, avoid mapping position disorder of the extended screen assembly, adapt to the use habit of the user, and improve the use experience of the user for the interactive function of the smart wearable device and the mobile terminal.

[0130] Please refer to Figure 9 , Figure 9 is a structural schematic diagram of a first embodiment of an interactive device of a smart wearable device and a mobile terminal provided by the present application, and the interactive device of the smart wearable device and the mobile terminal is used to execute the foregoing interactive method of the smart wearable device and the mobile terminal.

[0131] As Figure 9 shown, the interactive device 400 of the smart wearable device and the mobile terminal includes a posture detection module 401, an angle point coordinate collection module 402, an assembly determination module 403, and an assembly operation module 404.

[0132] The posture detection module 401 is configured to detect the current posture of the mobile terminal when the smart wearable device is in an extended screen interaction state with the mobile terminal.

[0133] The angle point coordinate collection module 402 is configured to collect angle point coordinates of a target angle point on the mobile terminal when it is detected that the current posture of the mobile terminal matches a preset interactive operation action.

[0134] The assembly determination module 403 is configured to determine a target extended screen assembly based on an extended screen assembly region corresponding to the angle point coordinates.

[0135] The assembly operation module 404 is configured to execute a target operation on the target extended screen assembly based on an assembly operation instruction corresponding to the gesture operation action when the current gesture action of the current user matches the preset gesture operation action.

[0136] In an embodiment, the interactive device 400 of the smart wearable device and the mobile terminal further includes an extended screen interaction state activation module, which includes:

[0137] A device unbinding instruction generation unit is configured to generate a device unbinding instruction based on the image collection device and identify a preset unbinding action made by the current user.

[0138] A binding state release unit is configured to release the binding state of the mobile terminal and the extended screen assembly based on the device unbinding instruction, so that the smart wearable device and the mobile terminal are in the extended screen interaction state.

[0139] In an embodiment, the device unbundling instruction generation unit comprises:

[0140] a hand action picture acquisition subunit configured to acquire hand action pictures of the current user based on the image acquisition device, and obtain a hand picture group, wherein the hand action picture group comprises at least two hand action pictures;

[0141] a time sequence coordinate data obtaining subunit configured to extract point coordinates of a target key point in the hand action pictures based on a preset key point tracking algorithm, and obtain time sequence coordinate data of the target key point;

[0142] a collection time interval calculation subunit configured to, when detecting that there are at least two coincident point coordinates in the time sequence coordinate data, calculate a collection time interval of each of the coincident point coordinates based on a collection time corresponding to each of the coincident point coordinates;

[0143] a device unbundling instruction generation subunit configured to, when the collection time interval is less than a preset time interval, determine that the hand action of the current user is the preset unbundling action, and generate the device unbundling instruction.

[0144] In an embodiment, the device unbundling instruction generation unit further comprises:

[0145] an interval frame number determination subunit configured to determine an interval frame number between the hand action pictures corresponding to each of the coincident point coordinates based on the time sequence coordinate data;

[0146] a device unbundling instruction generation subunit configured to, when the interval frame number is less than a preset interval frame number, determine that the hand action of the current user is the preset unbundling action, and generate the device unbundling instruction.

[0147] In an embodiment, the interaction device 400 of the smart wearable device and the mobile terminal further comprises an extended screen mode activation module, and the extended screen mode activation module comprises:

[0148] a contour coordinate acquisition unit configured to acquire contour coordinates of the mobile terminal based on the image acquisition device;

[0149] a relative position relationship obtaining unit configured to obtain a relative position relationship between at least one extended screen component and the mobile terminal;

[0150] an extended screen mode activation unit configured to, based on the contour coordinates of the mobile terminal and the relative position relationship, display the extended screen component in the display interface of the smart wearable device to activate an extended screen mode, wherein the mobile terminal and the extended screen component are in a bound state in the extended screen mode.

[0151] In an embodiment, the extended screen mode activating module further comprises:

[0152] a size information obtaining unit, configured to obtain first size information of the mobile terminal and second size information of the extended screen component;

[0153] a diagonal coordinate information calculating unit, configured to calculate diagonal coordinate information of the extended screen component in the display interface based on the first size information, the second size information, the relative position relationship and profile coordinates of the mobile terminal;

[0154] an extended screen component region determining unit, configured to determine the extended screen component region corresponding to each of the extended screen components based on the diagonal coordinate information.

[0155] In an embodiment, the interaction device 400 between the smart wearable device and the mobile terminal further comprises a gesture action matching module, and the gesture action matching module comprises:

[0156] a gesture instruction library obtaining unit, configured to obtain a preset gesture instruction library, wherein the gesture instruction library comprises at least one gesture operation action and the component operation instruction corresponding to each gesture operation action;

[0157] an action similarity determining unit, configured to compare the current gesture action with each gesture operation action in the gesture instruction library based on a gesture recognition algorithm, and determine the similarity between the current gesture action and each gesture operation action;

[0158] a gesture operation action determining unit, configured to determine the gesture operation action matched with the current gesture action based on the similarity when the similarity is greater than or equal to a preset similarity.

[0159] It should be noted that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the foregoing embodiment of the interaction method between the smart wearable device and the mobile terminal, and will not be described herein.

[0160] The device provided in the foregoing embodiments can be implemented as a computer program in the form of a program product, and can run on a computer device such as the computer device 1000 shown in FIG. 10. Figure 10

[0161] Please refer to Figure 10 , Figure 10 is a structural schematic block diagram of a computer device provided in the embodiments of the present application. The computer device can be a server.

[0162] Refer to​Figure 10 The computer device comprises a processor, a memory and a network interface connected through a system bus, wherein the memory can comprise a non-volatile storage medium and an internal memory.

[0163] The non-volatile storage medium can store an operating system and a computer program. The computer program comprises program instructions which, when executed, can cause the processor to perform any one of the interaction methods of the smart wearable device and the mobile terminal.

[0164] The processor is used to provide computing and control capabilities to support the operation of the entire computer device.

[0165] The internal memory provides an environment for the running of the computer program in the non-volatile storage medium, which, when executed by the processor, can cause the processor to perform any one of the interaction methods of the smart wearable device and the mobile terminal.

[0166] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that Figure 10 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0167] It should be understood that the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0168] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0169] The smart wearable device detects the current posture of the mobile terminal when the smart wearable device is in an extended screen interaction state with the mobile terminal;

[0170] When it is detected that the current posture of the mobile terminal matches a preset interaction operation action, the corner point coordinates of a target corner point on the mobile terminal are collected;

[0171] determine a target extended screen component based on the region of the extended screen component corresponding to the corner point coordinates;

[0172] When the current gesture action of the current user matches the preset gesture operation action, perform a target operation on the target extended screen component based on a component operation instruction corresponding to the gesture operation action.

[0173] In one embodiment, before the processor implements the detection of the current posture of the mobile terminal when the smart wearable device is in the extended screen interaction state with the mobile terminal, the processor is further configured to implement:

[0174] identify a preset unbinding action made by the current user based on the image acquisition device, and generate a device unbinding instruction;

[0175] release the binding state of the mobile terminal and the extended screen component based on the device unbinding instruction, so that the smart wearable device and the mobile terminal are in the extended screen interaction state.

[0176] In one embodiment, when the processor implements the identification of the preset unbinding action made by the current user based on the image acquisition device, the processor is configured to implement:

[0177] acquire hand action pictures of the current user based on the image acquisition device, and obtain a hand picture group, wherein the hand action picture group includes at least two hand action pictures;

[0178] extract point coordinates of a target key point in the hand action pictures based on a preset key point tracking algorithm, and obtain time sequence coordinate data of the target key point;

[0179] When at least two coincident point coordinates are detected in the time sequence coordinate data, calculate the acquisition time interval of each coincident point coordinate based on the acquisition time corresponding to each coincident point coordinate;

[0180] When the acquisition time interval is less than a preset time interval, determine that the hand action of the current user is the preset unbinding action, and generate the device unbinding instruction.

[0181] In one embodiment, after the processor implements the detection of at least two coincident point coordinates in the time sequence coordinate data, the processor is further configured to implement:

[0182] determine the interval frame number between the hand action pictures corresponding to each coincident point coordinate based on the time sequence coordinate data;

[0183] When the interval frame number is less than the preset interval frame number, it is determined that the hand movement of the current user is the preset unbinding action, and the device unbinding instruction is generated.

[0184] In one embodiment, before the processor implements the image acquisition device-based identification of the preset unbinding action made by the current user and the generation of the device unbinding instruction, the processor is further configured to implement:

[0185] acquiring contour coordinates of the mobile terminal based on the image acquisition device;

[0186] acquiring a relative position relationship between at least one extended screen component and the mobile terminal;

[0187] mapping and displaying the extended screen component in a display interface of the smart wearable device based on the contour coordinates of the mobile terminal and the relative position relationship, to activate an extended screen mode, wherein the mobile terminal and the extended screen component are in a bound state in the extended screen mode.

[0188] In one embodiment, before the processor implements the contour coordinates of the mobile terminal and the relative position relationship, mapping and displaying the extended screen component in the display interface of the smart wearable device to activate the extended screen mode, the processor is further configured to implement:

[0189] acquiring first size information of the mobile terminal and second size information of the extended screen component;

[0190] calculating diagonal coordinate information of the extended screen component in the display interface based on the first size information, the second size information, the relative position relationship, and the contour coordinates of the mobile terminal;

[0191] determining the extended screen component region corresponding to each of the extended screen components based on the diagonal coordinate information.

[0192] In one embodiment, before the processor implements the current gesture action of the current user matching the preset gesture operation action, and performing a target operation on the target extended screen component based on the component operation instruction corresponding to the gesture operation action, the processor is further configured to implement:

[0193] acquiring a preset gesture instruction library, wherein the gesture instruction library includes at least one gesture operation action and the component operation instruction corresponding to each gesture operation action;

[0194] comparing the current gesture action with each gesture operation action in the gesture instruction library based on a gesture recognition algorithm, to determine the similarity between the current gesture action and each gesture operation action;

[0195] When the similarity is greater than or equal to a preset similarity, the gesture operation action matching the current gesture action is determined based on the similarity.

[0196] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement any of the interaction methods between the intelligent wearable device and the mobile terminal provided by the embodiments of the present application.

[0197] The computer readable storage medium can be an internal storage unit of the computer device, for example, a hard disk or a memory of the computer device. The computer readable storage medium can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0198] The above merely describes the specific embodiments of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for interaction between a smart wearable device and a mobile terminal, the method comprising: The method comprises: The smart wearable device detects the current posture of the mobile terminal when the smart wearable device is in an extended screen interaction state with the mobile terminal; When it is detected that the current posture of the mobile terminal matches a preset interaction operation action, the coordinates of a target corner point on the mobile terminal are collected; Based on the extended screen component area corresponding to the corner point coordinates, a target extended screen component is determined; wherein the extended screen component is a mapping of the application interface of the mobile terminal on the display interface of the smart wearable device, and / or the extended screen component includes a functional module component of the smart wearable device; When the current gesture action of the current user matches a preset gesture operation action, a target operation is performed on the target extended screen component based on the component operation instruction corresponding to the gesture operation action. 2.The method of claim 1, wherein, Before the smart wearable device detects the current posture of the mobile terminal when the smart wearable device is in an extended screen interaction state with the mobile terminal, the method further comprises: Based on the image acquisition device, a preset unbinding action made by the current user is recognized to generate a device unbinding instruction; Based on the device unbinding instruction, the binding state of the mobile terminal and the extended screen component is released, so that the smart wearable device and the mobile terminal are in the extended screen interaction state. 3.The method of claim 2, wherein, The method of recognizing the preset unbinding action made by the current user based on the image acquisition device to generate a device unbinding instruction comprises: Based on the image acquisition device, hand action pictures of the current user are collected to obtain a hand picture group, wherein the hand action picture group includes at least two hand action pictures; Based on a preset key point tracking algorithm, the point coordinates of a target key point in the hand action picture are extracted to obtain time sequence coordinate data of the target key point; When it is detected that there are at least two coincident point coordinates in the time sequence coordinate data, the collection time interval of each coincident point coordinate is calculated based on the collection time corresponding to each coincident point coordinate; When the collection time interval is less than a preset time interval, the hand action of the current user is determined to be the preset unbinding action, and the device unbinding instruction is generated. 4.The method of claim 3, wherein, After detecting that there are at least two coincident point coordinates in the time sequence coordinate data, the method further comprises: Based on the time sequence coordinate data, the interval frame number between the hand action pictures corresponding to each coincident point coordinate is determined; When the interval frame number is less than a preset interval frame number, the hand action of the current user is determined to be the preset unbinding action, and the device unbinding instruction is generated. 5.The method of claim 2, wherein, Before recognizing the preset unbinding action made by the current user based on the image acquisition device to generate a device unbinding instruction, the method further comprises: Based on the image acquisition device, the contour coordinates of the mobile terminal are collected; The relative position relationship between at least one extended screen component and the mobile terminal is obtained; Based on the contour coordinates of the mobile terminal and the relative position relationship, the extended screen component is displayed in the display interface of the smart wearable device to activate the extended screen mode, wherein the mobile terminal and the extended screen component are in a binding state in the extended screen mode. 6.The method of claim 5, wherein, The mapping and displaying of the extended screen components in the display interface of the smart wearable device based on the contour coordinates of the mobile terminal and the relative position relationship further comprises, before the activation of the extended screen mode: obtaining first size information of the mobile terminal and second size information of the extended screen components; calculating diagonal coordinate information of the extended screen components in the display interface based on the first size information, the second size information, the relative position relationship, and the contour coordinates of the mobile terminal; determining the extended screen component region corresponding to each of the extended screen components based on the diagonal coordinate information. 7.The method of claim 1, wherein, The method further comprises, before the execution of the target operation on the target extended screen component based on the component operation instruction corresponding to the gesture operation action when the current gesture action of the current user matches the preset gesture operation action: obtaining a preset gesture instruction library, wherein the gesture instruction library comprises at least one gesture operation action and the component operation instruction corresponding to each gesture operation action; comparing the current gesture action with each gesture operation action in the gesture instruction library based on a gesture recognition algorithm to determine the similarity of the current gesture action and each gesture operation action; determining the gesture operation action matching the current gesture action based on the similarity when the similarity is greater than or equal to a preset similarity.

8. An interaction device of a smart wearable device and a mobile terminal, characterized in that, The interaction device of the smart wearable device and the mobile terminal comprises: a posture detection module configured to detect the current posture of the mobile terminal when the smart wearable device is in an extended screen interaction state with the mobile terminal; an angle point coordinate acquisition module configured to acquire the angle point coordinates of a target angle point on the mobile terminal when the current posture of the mobile terminal matches a preset interaction operation action; a component determination module configured to determine a target extended screen component based on the extended screen component region corresponding to the angle point coordinates; wherein the extended screen component is the mapping of the application interface of the mobile terminal in the display interface of the smart wearable device, and / or the extended screen component comprises a functional module component of the smart wearable device; a component operation module configured to execute a target operation on the target extended screen component based on the component operation instruction corresponding to the gesture operation action when the current gesture action of the current user matches the preset gesture operation action.

9. A computer device, comprising: The computer device comprises a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program is executed by the processor to implement the steps of the interaction method of the smart wearable device and the mobile terminal according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to implement the steps of the interaction method of the smart wearable device and the mobile terminal according to any one of claims 1 to 7.

Images