Interaction method and device of intelligent wearable device and mobile terminal, equipment and medium

Abstract

The invention provides an interaction method and device of intelligent wearable equipment and a mobile terminal, equipment and a storage medium, and the method comprises the steps: when the intelligent wearable equipment and the mobile terminal are in an extended screen interaction state, triggering an operation recognition process of an extended screen assembly when the current posture of the mobile equipment is detected to be matched with a preset interaction operation action; the assembly operation accuracy is improved. The method comprises the following steps of: determining a target extension screen component which a current user wants to operate by collecting a corner point coordinate of a target corner point on a mobile terminal, and realizing accurate identification and selection of the extension screen component; according to the embodiment of the invention, the current gesture action of the current user is collected, and the corresponding component operation instruction is generated according to the gesture operation action matched with the current gesture action, so that the target operation is executed on the target extension screen component, the interaction efficiency of the intelligent wearable device and the mobile terminal is improved, and the use experience of the user is enhanced.

Description

Technical Field

[0001] The present application relates to the field of smart wearable technology, and in particular to a method, device, computer equipment and storage medium for interaction between a smart wearable device and a mobile terminal. Background Art

[0002] The extended screen is a fast application engine running on smart large-screen devices, providing a new multi-screen interactive mode. It can be pulled up and interacted with by mobile client applications and mini-programs in a similar way to screen casting, or it can be directly operated and interacted with using a large-screen remote control. The extended screen is a mapping of the mobile terminal application interface on the display interface of the smart wearable device, which facilitates the rapid upload of mobile terminal services and content to the large screen, greatly reducing the cost of developing, promoting and maintaining large-screen applications.

[0003] Smart wearable devices can realize extended screen interaction with mobile terminals based on augmented reality technology, and map the desktop applications of mobile terminals on the display interface of smart wearable devices. Although smart wearable devices currently realize extended screen display of mobile terminal desktop applications, they cannot directly operate the extended screen components in the display interface, making the interaction effect between smart wearable devices and mobile terminals unable to meet user needs, and users have a poor experience in using the interactive functions of smart wearable devices and mobile terminals.

[0004] Therefore, how to improve the user experience of the interactive functions between smart wearable devices and mobile terminals has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a method, apparatus, device and storage medium for interaction between a smart wearable device and a mobile terminal, aiming to improve the user experience of the interaction function between the smart wearable device and the mobile terminal.

[0006] In a first aspect, the present application provides an interaction method between a smart wearable device and a mobile terminal, comprising:

[0007] When the smart wearable device is in an extended screen interaction state with the mobile terminal, detecting the current posture of the mobile terminal;

[0008] When it is detected that the current posture of the mobile terminal matches the preset interactive operation action, collecting the corner point coordinates of the target corner point on the mobile terminal;

[0009] Determining a target extended screen component based on an extended screen component area corresponding to the corner point coordinates;

[0010] When the current gesture action of the current user matches the preset gesture operation action, the target operation is performed on the target extended screen component based on the component operation instruction corresponding to the gesture operation action.

[0011] In a second aspect, the present application further provides an interactive device between a smart wearable device and a mobile terminal, comprising:

[0012] The posture detection module is used to detect the current posture of the mobile terminal when the smart wearable device is in an extended screen interaction state with the mobile terminal;

[0013] A corner point coordinate acquisition module, configured to acquire the corner point coordinates of a target corner point on the mobile terminal when detecting that the current posture of the mobile terminal matches a preset interactive operation action;

[0014] A component determination module, used to determine a target extended screen component based on an extended screen component area corresponding to the corner point coordinates;

[0015] The component operation module is used to perform a target operation on the target extended screen component based on the component operation instruction corresponding to the gesture operation when the current gesture action of the current user matches the preset gesture operation action.

[0016] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the method for interaction between the smart wearable device and the mobile terminal as described above are implemented.

[0017] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the method for interaction between a smart wearable device and a mobile terminal as described above are implemented.

[0018] The present invention provides an interaction method, device, equipment and storage medium between a smart wearable device and a mobile terminal, the method comprising: when the smart wearable device is in an extended screen interaction state with the mobile terminal, detecting the current posture of the mobile terminal; when detecting that the current posture of the mobile terminal matches a preset interactive operation action, collecting the corner point coordinates of the target corner point on the mobile terminal; determining the target extended screen component based on the extended screen component area corresponding to the corner point coordinates; when the current gesture action of the current user matches the preset gesture operation action, performing the target operation on the target extended screen component based on the component operation instruction corresponding to the gesture operation action. Through the above manner, when the smart wearable device and the mobile terminal are in an extended screen interaction state, the smart wearable device matches the current posture of the mobile terminal with the preset interactive operation action by detecting the current posture of the mobile terminal; and when the current posture matches the preset interactive operation action, triggering the operation recognition process of the extended screen component, improving the accuracy of component operation, and avoiding mis-triggering of operation. By collecting the corner point coordinates of the target corner point on the mobile terminal, the target extended screen component that the current user wants to operate is determined according to the extended screen component area where the corner point coordinates are located, thereby realizing accurate identification and selection of the extended screen component; by comparing the current gesture action of the current user with the preset gesture operation action, the gesture operation action that matches the current gesture action is determined, and then the component operation instruction corresponding to the gesture operation action is generated, and according to the component operation instruction, the target operation is performed on the target extended screen component, so that the current user can directly operate the extended screen component on the display interface of the smart wearable device, thereby improving the interaction efficiency between the smart wearable device and the mobile terminal, and enhancing the user's usage experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of a first embodiment of an interaction method between a smart wearable device and a mobile terminal provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a flow chart of a second embodiment of a method for interaction between a smart wearable device and a mobile terminal provided in an embodiment of the present application;

[0022] Figure 3 A detailed flow chart of the first step of a second embodiment of a method for interaction between a smart wearable device and a mobile terminal provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a flow chart of a third embodiment of an interaction method between a smart wearable device and a mobile terminal provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the distribution of extended screen components in an extended screen mode activated state provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the relative coordinate relationship between an extended screen component area and a mobile terminal provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a relative size relationship distribution between second size information of an extended screen assembly and first size information of a mobile terminal provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of another relative size relationship distribution between the second size information of an extended screen assembly and the first size information of a mobile terminal provided in an embodiment of the present application;

[0028] Fig. 9 It is a structural schematic diagram of a first embodiment of an interactive device between a smart wearable device and a mobile terminal provided by the present application;

[0029] Fig.10 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0030] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0033] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0034] Please refer to Figure 1 , Figure 1 A flowchart of a first embodiment of a method for interaction between a smart wearable device and a mobile terminal provided in an embodiment of the present application.

[0035] like Figure 1 As shown, the interaction method between the smart wearable device and the mobile terminal includes steps S101 to S104.

[0036] Step S101: When the smart wearable device is in an extended screen interaction state with the mobile terminal, the smart wearable device detects the current posture of the mobile terminal;

[0037] In one embodiment, the smart wearable device refers to a wearable eyewear device with an independent operating system that can realize various functions through software installation, such as AR glasses (Augmented Reality goggles), VR glasses (Virtual Reality), AR helmets, etc.

[0038] In one embodiment, a mobile terminal or a mobile communication terminal refers to a computer device that can be used on the move, including a mobile phone, a notebook, a tablet computer, etc.

[0039] In one embodiment, the smart wearable device can detect the current posture of the mobile terminal through an image acquisition device.

[0040] In one embodiment, the image acquisition device refers to a device component that can realize image acquisition and image processing, such as a camera, an infrared sensor, a laser scanner, etc.

[0041] In one embodiment, the image acquisition device may be carried by the smart wearable device itself, or may be an independent device installed on the smart wearable device.

[0042] In one embodiment, the image acquisition device is connected to the smart wearable device for acquiring images or video data of the mobile device and the user's hand, and identifying the posture, coordinates, and movements of the mobile device and the user's hand based on the acquired images and video data. When the image acquisition device recognizes that the mobile terminal moves into the acquisition screen of the image acquisition device, it can identify the contour information and corner point coordinate information of the mobile terminal, and can also track and identify the movement trajectory of the mobile terminal and other information.

[0043] Step S102: when it is detected that the current posture of the mobile terminal matches the preset interactive operation action, the corner point coordinates of the target corner point on the mobile terminal are collected;

[0044] In one embodiment, the display interface of the smart wearable device can be used as a two-dimensional coordinate system. The upper left corner of the display interface can be used as the origin of the coordinate system, and the two sides of the display interface can be used as coordinate axes of the two-dimensional coordinate system. The scale of the coordinate axis is set according to the length and width of the display interface. In this way, the outline coordinates and corner point coordinates of the mobile terminal can be tracked in the display interface of the smart wearable device.

[0045] In one embodiment, the image acquisition device of the smart wearable device can track and calculate the outline and corner point coordinates of the mobile terminal in real time. If it is detected that the current posture of the mobile terminal matches the interactive operation action, that is, the similarity between the current posture and the interactive operation action is greater than or equal to the similarity threshold (such as 90%), the current corner point coordinates of the mobile terminal are collected by the image acquisition device.

[0046] In one embodiment, the interactive operation action may include double-clicking, maintaining a stationary state for a preset time, sliding, etc.

[0047] In one embodiment, the corner coordinates of the mobile terminal may be edge lines representing the sides of the mobile terminal based on the outline coordinates of the mobile terminal, wherein the intersection of two intersecting edge lines may be used as the corner coordinates of the mobile terminal.

[0048] In one embodiment, the four corners of the mobile terminal are generally designed to be arc-shaped. The intersection coordinates of the edges of the mobile terminal can be calculated based on the contour coordinates of the mobile terminal. The four intersections correspond to the four arc angles of the mobile terminal respectively. The arc point on each arc angle that is closest to the corresponding intersection coordinates can be used as the corner point.

[0049] In one embodiment, a mobile terminal generally has four sides, that is, the coordinates of the four corner points of the mobile terminal can be tracked. In order to improve the recognition accuracy of the extended screen component, one of the four corner points can be pre-set as the target corner point, such as the upper left corner of the mobile terminal.

[0050] In one embodiment, the smart wearable device can recognize the outline of the mobile terminal through the image acquisition device and activate the extended screen function of the mobile terminal, thereby realizing the extended screen interaction between the smart wearable device and the mobile terminal. In the extended screen interaction state, the smart wearable device collects the corner point coordinates of the mobile terminal through the image acquisition device, such as the corner point coordinates of the upper right corner.

[0051] Step S103, determining a target extended screen component based on the extended screen component area corresponding to the corner point coordinates;

[0052] In one embodiment, when it is detected that the current posture of the mobile terminal matches the preset interactive operation action, the corner point coordinates of the target corner point of the mobile terminal are collected, and the extended screen component area where the corner point coordinates of the target corner point are located is determined as the target area. The target extended screen component corresponding to the area is the extended screen component on which the current user wants to perform the operation.

[0053] Among them, the extended screen component is a high-performance cross-terminal development framework for smart large-screen devices, a fast application engine running on smart wearable devices, which can help developers quickly build application services for smart wearable devices and provide multi-screen interaction capabilities.

[0054] In one embodiment, the extended screen component can be a mapping of the mobile terminal application interface in the display interface of the smart wearable device, which facilitates the quick display of the mobile terminal's services and content in the display interface of the smart wearable device. The extended screen component includes an application component of the mobile terminal, which is used to display the application display content of the mobile terminal, such as the video played by the video software, the real-time bullet screen; the reading page of the novel software, etc.

[0055] In one embodiment, the extended screen component also includes a functional module component of the smart wearable device, which facilitates users to query and call the functions and services of the smart wearable device, such as product descriptions, connectable device lists, network lists, etc.

[0056] Further, based on the corner point coordinates, a current interaction action of the current user operating the mobile terminal is identified; when the current interaction action meets a trigger condition corresponding to the preset interaction operation action, the current interaction action is determined to be the interaction operation action.

[0057] In one embodiment, the interactive operation action can be identified based on the coordinates of the preset corners of the mobile terminal. For example, the current user operates the mobile terminal to perform a double-click action, that is, the coordinate value of the target corner point of the mobile terminal points to the same coordinate position within a preset time; or when the image acquisition device recognizes that there is an obvious tapping action at the corner of the mobile terminal, the landing point coordinates of the target corner point in the tapping action are collected. If the landing point coordinates are located in a certain extended screen component area, the extended screen component corresponding to the extended screen component area is determined to be the target extended screen component; or when it is identified that the time that the corner point coordinates of the target corner point stay in a certain extended screen component area reaches a preset threshold, the extended screen component corresponding to the extended screen component area can be determined to be the target extended screen component that the current user wants to view or perform a certain operation.

[0058] In one embodiment, if it is detected that the current user operates the mobile terminal to perform an interactive operation, but the coordinate area corresponding to the corner point coordinates does not belong to the extended screen component area, it can be determined that the interactive operation is invalid and no operation instruction is generated.

[0059] In one embodiment, if it is detected that the current user operates the mobile terminal to perform an interactive operation, but the coordinate area corresponding to the corner point coordinates does not belong to the extended screen component area, the distance between the corner point coordinates and the midpoint position of each extended screen component area can also be calculated, and the extended screen component area corresponding to the shortest distance is determined to be the landing area of ​​the corner point coordinates, thereby determining that the extended screen component corresponding to the landing area is the target extended screen component.

[0060] In one embodiment, the extended screen component area is the display area of ​​each extended screen component in the display interface of the smart wearable device. With the display interface as a two-dimensional coordinate system, the coordinates of the four corner points of the extended screen component area can be determined, so that the side length of the extended screen component area can be calculated according to the four corner point coordinates. For example, the coordinate distance between the two corner point coordinates in the same row is the length of the extended screen component area, and the coordinate distance between the two corner point coordinates in the same column is the width of the extended screen component area. Therefore, according to the coordinate relationship between the corner point coordinates and the area coordinates of the extended screen component area, it can be judged whether the coordinate area corresponding to the corner point coordinates belongs to the extended screen component area. If the corner point coordinates are within the range of any extended screen component area, it is determined that the coordinate area corresponding to the corner point coordinates belongs to the extended screen component area; and if the corner point coordinates are not within the range of any extended screen component area, it is determined that the coordinate area corresponding to the corner point coordinates does not belong to the extended screen component area.

[0061] Step S104: when the current gesture action of the current user matches the preset gesture operation action, based on the component operation instruction corresponding to the gesture operation action, perform a target operation on the target extended screen component;

[0062] In one embodiment, the current gesture of the current user can be collected through the image acquisition module. The current user can make a preset gesture operation in the image acquisition screen of the image acquisition module so that the smart wearable device can recognize the current user's operation on the target extended screen component.

[0063] In one embodiment, the preset gesture operation action may include a single finger or multiple fingers, or a gesture action of the entire hand, such as a thumb double-click action, an index finger tip sliding action, etc.

[0064] Furthermore, a preset gesture instruction library is obtained, wherein the gesture instruction library includes at least one of the gesture operation actions and the component operation instructions corresponding to each of the gesture operation actions; based on a gesture recognition algorithm, the current gesture action is compared with each of the gesture operation actions in the gesture instruction library to determine the similarity between the current gesture action and each of the gesture operation actions; when the similarity is greater than or equal to a preset similarity, the gesture operation action that matches the current gesture action is determined based on the similarity.

[0065] In one embodiment, gesture operation actions corresponding to each component operation instruction can be preset according to at least one component operation instruction of the extended screen component. The smart wearable device can pre-record gesture operation actions through an image acquisition device, and uniquely associate each gesture operation action with the corresponding component operation instruction, so that each component operation instruction corresponds to a gesture operation action, and store it in a data storage module of the smart wearable device to obtain a gesture instruction library.

[0066] In one embodiment, a component operation instruction may also correspond to multiple different gesture operation actions. For example, a component details page viewing instruction may correspond to a double-click gesture operation action. The double-click gesture operation action may be a double-click action of the thumb or a double-click action of the index finger.

[0067] In one embodiment, the same gesture operation action may also correspond to multiple component operation instructions. For example, a component details page viewing instruction and a component details page exit instruction may both correspond to a gesture double-click operation action. If a gesture double-click operation action is detected outside the component details page (i.e., without entering the component details page), a component details page viewing instruction is generated. If a gesture double-click operation action is detected within the component details page (i.e., already in the component details page), a component details page exit instruction is generated.

[0068] In one embodiment, the motion trajectory of the current gesture action can be calculated through a gesture recognition algorithm, and then the motion trajectory is compared with the motion trajectory of each gesture operation action in the gesture instruction library to calculate the similarity between the motion trajectory of the current gesture action and the motion trajectory of each gesture operation action.

[0069] If the similarity between the motion trajectory of the current gesture action and the motion trajectory of a certain gesture operation action is greater than or equal to the preset similarity, the current gesture action is determined to be the gesture operation action, and the two are matched. If the similarity between the motion trajectory of the current gesture action and the motion trajectory of each gesture operation action in the gesture instruction library is less than the preset similarity, the current gesture action is determined to be unmatched with each gesture operation action, and the component operation instruction cannot be generated.

[0070] In one embodiment, if the similarity between the motion trajectory of the current gesture action and the motion trajectory of multiple gesture operation actions is greater than or equal to a preset similarity, the gesture operation action with the highest similarity is determined to match the current gesture action.

[0071] In one embodiment, the motion trajectory of the current gesture action may be a coordinate trajectory of a key point of the hand (joint points, fingertips, etc.) over a time series.

[0072] In one embodiment, when it is detected that the current gesture action matches the preset gesture operation action, the smart wearable device generates a corresponding component operation instruction according to the gesture operation action, and performs a corresponding operation on the target extended component according to the component operation instruction.

[0073] In one embodiment, when it is detected that the current gesture action matches the preset gesture operation action, the smart wearable device can determine the component operation instruction corresponding to the current gesture operation action according to the preset gesture instruction library, and then generate the component operation instruction. For example, a click action can indicate entering the component details page, and at this time, an instruction to open the details page of the extended screen component is generated; a swiping action indicates exiting the component details page, and at this time, an instruction to exit the details page of the extended screen component is generated, etc.

[0074] Exemplarily, if the double-click operation action corresponds to a component viewing instruction of an extended screen component, a viewing instruction of the target extended screen component is generated according to the target extended screen component pointed to by the double-click operation action, such as the component operation instruction "Music Player--View", then you can enter the details page of the music player extended screen component according to the component operation instruction.

[0075] If the sliding operation corresponds to a switching instruction of the extended screen component, when it is detected that the mobile terminal makes a sliding operation in the current extended screen component details page, a component details page switching instruction is generated to switch the current extended screen component details page to another extended screen component details page.

[0076] In a specific embodiment, the process of executing the target extended screen component operation according to the current gesture action of the current user may be: when the thumb double-clicks the extended screen component, it indicates that the details page of the extended screen component is to be viewed. When the image acquisition module recognizes that the user's thumb makes a double-click action on the component page, the gesture operation action (thumb-double-click) corresponding to the double-click action is matched from the gesture instruction library, and the smart wearable device generates a component operation instruction (enter component details page instruction) corresponding to the gesture operation action, and enters the details page of the target extended screen component according to the component operation instruction. When the user's gesture is captured in the details page of the target extended screen component to make a sliding action (corresponding to the component switching instruction action), the smart wearable device generates a component switching instruction to switch to display the details page of the next extended screen component. When the user's thumb is captured to make a double-click action in the details page of the extended screen component, at this time, the back-off instruction can be matched according to "component details page-double-click action-exit component details page", and the details page of the currently displayed extended screen component is exited according to the back-off instruction.

[0077] This embodiment provides an interaction method between a smart wearable device and a mobile terminal. When the smart wearable device and the mobile terminal are in an extended screen interaction state, the method detects the current posture of the mobile terminal through an image acquisition device, and when the current posture matches a preset interactive operation action, triggers the operation recognition process of the extended screen component, thereby improving the accuracy of component operation and avoiding mis-triggering of operation. The image acquisition device acquires the corner point coordinates of the target corner point on the mobile terminal, thereby determining the target extended screen component that the current user wants to operate according to the extended screen component area where the corner point coordinates are located, and realizing accurate identification and selection of the extended screen component; according to the identified interactive operation action, a corresponding component operation instruction is generated, thereby performing the target operation on the target extended screen component according to the component operation instruction, so that the current user can directly operate the extended screen component on the display interface of the smart wearable device, thereby improving the interaction efficiency between the smart wearable device and the mobile terminal and enhancing the user's usage experience.

[0078] Please refer to Figure 2 , Figure 2 A flowchart of a second embodiment of a method for interaction between a smart wearable device and a mobile terminal provided in an embodiment of the present application.

[0079] like Figure 2 As shown, based on the above Figure 1 In the illustrated embodiment, before step S101, the following steps are further included:

[0080] Step S201: Based on the image acquisition device, identify the preset unbinding action made by the current user and generate a device unbinding instruction;

[0081] In one embodiment, in the extended screen mode, the mobile terminal and the extended screen component are in a binding state. At this time, when the mobile terminal moves, the extended screen component moves synchronously and is anchored around the mobile terminal. At this time, the current user can only control the extended screen component by operating the desktop application on the mobile terminal screen. To control the extended screen component on the display interface of the smart wearable device, the mobile terminal and the extended screen component need to be unbound.

[0082] In one embodiment, the binding state between the mobile terminal and the extended screen component can be accessed by designing a specific preset unbinding action, for example, double-clicking within the screen range of the mobile terminal.

[0083] In one embodiment, when the image acquisition device of the smart wearable device recognizes that the current user makes a preset unbinding action, an unbinding event between the mobile terminal and the extended screen component is triggered, and the smart wearable device generates a device unbinding instruction to release the binding state between the mobile terminal and the extended screen component.

[0084] Furthermore, if Figure 3As shown, the step S201 specifically includes:

[0085] Step S2011: Based on the image acquisition device, collect the hand movement pictures of the current user to obtain a hand movement picture group, wherein the hand movement picture group includes at least two of the hand movement pictures;

[0086] In one embodiment, the image acquisition device of the smart wearable device can be used to collect hand movement pictures of the current user to obtain a hand movement picture group. The picture group can be in the form of a video stream, that is, a group of hand movement pictures taken in chronological order.

[0087] In one embodiment, the shooting frame rate of the image acquisition device can be set according to the tracking effect of the hand movement, for example, the shooting frame rate is set to 25FPS (frames per second), that is, 25 frames of pictures are taken per second. In this way, it can ensure that the user's hand movements can be fully captured, and the amount of image data processing can be reduced, thereby improving the recognition efficiency and action response efficiency of hand movements.

[0088] Step S2012: extracting the point coordinates of the target key points in the hand action picture based on a preset key point tracking algorithm to obtain the time series coordinate data of the target key points;

[0089] In one embodiment, the time series coordinate data refers to the time series data of the coordinates of the target key points. The time series data is a series of data obtained in different time periods or time points in chronological order, describing the changes of the observed object over time.

[0090] In one embodiment, according to a preset key point tracking algorithm, target key points in each hand action image, such as fingertips, joints, etc., are extracted, and the point coordinates of the target key points are tracked and calculated to obtain a set of coordinate change data of the target key points in time sequence.

[0091] Step S2013: when it is detected that there are at least two coincident point coordinates in the time series coordinate data, based on the collection time corresponding to each of the coincident point coordinates, calculating the collection time interval of each of the coincident point coordinates;

[0092] In one embodiment, when at least two coincident point coordinates are detected in the time series coordinate data, it means that the current user has clicked the same position at least twice by operating the mobile phone. At this time, the coordinate collection time of each coincident point coordinate is determined according to the time series coordinate data, and then the collection time interval of each coincident point coordinate is calculated.

[0093] In one embodiment, the coincident point coordinates may be two point coordinates with exactly the same coordinates, or two point coordinates with a point coordinate distance less than a set value. For example, two point coordinates with a point coordinate distance less than 5 pixels may be considered coincident point coordinates.

[0094] Step S2014: When the collection time interval is less than the preset time interval, determine that the hand movement of the current user is the preset unbinding movement, and generate the device unbinding instruction.

[0095] In one embodiment, a preset time interval is set according to a preset unbinding action. If the preset unbinding action is double-clicking, the preset time interval for determining a double-click can be set to 1 second, which means that if the current user clicks the same position twice within 1 second, the preset unbinding action is triggered.

[0096] In one embodiment, when the collection time interval of the coordinates of each overlapping point is less than the preset time interval, it can be determined that the current user's hand movement has triggered a preset unbinding action. At this time, the smart wearable device generates a device unbinding instruction and, according to the unbinding instruction, releases the binding status of the mobile terminal and the extended screen assembly.

[0097] Exemplarily, a double-click is two quick and consecutive clicks within the same area. The 21 key points of the human hand can be obtained through the hand key point algorithm. The position coordinates of the thumb tip and the first joint point are tracked in each frame. Assuming that the interval time is defined as t frames, if the thumb tip and the first joint of the thumb appear twice in the same position within t frames and the thumb is in the screen area of ​​the mobile terminal, the preset unbinding action is triggered, and the binding state between the mobile terminal and the extended screen component is released.

[0098] In one embodiment, the area range may be a range area radiating a certain coordinate distance outward with the coordinates of the landing point of the first click as the center, that is, the determination is made based on the coordinate distance of the coordinates of the landing points of two consecutive clicks. For example, with the coordinates of the landing point of the first click as the reference point, if the coordinate distance between the coordinates of the landing point of the second click and the reference point is less than or equal to a limited value (such as a distance of five pixels), and the time interval between the two clicks is within t frames (including t frames), it can be determined as a double-click action.

[0099] In one embodiment, when the collection time interval is greater than the preset time interval, it is determined that the current user's hand movement has not triggered the preset unbinding action. At this time, the smart wearable device does not respond and continues to collect the user's hand movement pictures through the image collection module.

[0100] Furthermore, after detecting that there are at least two overlapping point coordinates in the time series coordinate data, it also includes: determining the interval frame number between the hand movement pictures corresponding to each of the overlapping point coordinates based on the time series coordinate data; when the interval frame number is less than the preset interval frame number, determining that the hand movement of the current user is the preset unbinding action, and generating the device unbinding instruction.

[0101] In one embodiment, the images captured by the image acquisition device are arranged in order of acquisition time, and the acquisition time interval between two adjacent frames of images is the same. For example, if the frame rate is set to 25FPS, the acquisition time interval between two adjacent frames of images in the captured images is 1 / 25 second. Therefore, the acquisition interval time between the two frames of images can be calculated based on the interval frame number.

[0102] In one embodiment, when at least two coincident point coordinates are detected in the time series coordinate data, the interval frame number between the hand action pictures corresponding to the coincident point coordinates can be calculated. For example, if the preset unbinding action is double-clicking, the interval frame number between the hand action pictures corresponding to two adjacent coincident point coordinates is calculated. If the interval frame number is less than the preset interval frame number, it is determined that the hand action of the current user triggers the preset unbinding action. When the preset unbinding action is triggered, the smart wearable device generates a device unbinding instruction. If the interval frame number is greater than the preset interval frame number, it is determined that the hand action of the current user does not trigger the preset unbinding action, and the smart wearable device does not need to respond to this hand action.

[0103] Step S202: based on the device unbinding instruction, unbinding the mobile terminal from the extended screen component so that the smart wearable device and the mobile terminal are in the extended screen interactive state.

[0104] In one embodiment, after the mobile terminal and the extended screen component are unbound, the extended screen component is fixedly displayed on the display screen of the smart wearable device and does not move with the movement of the mobile terminal. At this time, the mobile terminal can be used as a touch device. When the user moves or taps the mobile terminal, the smart wearable device tracks and calculates the coordinates of a corner point of the mobile terminal (such as the upper left corner coordinates) in real time, and recognizes the action of the mobile terminal.

[0105] In this embodiment, the hand movement made by the current user is identified through the image acquisition device, so when it is determined that the hand movement made by the current user is a preset unbinding action, the device unbinding instruction is triggered to generate, and according to the device unbinding instruction, the binding state of the mobile terminal and the extended screen component is released, so that the extended screen component is fixedly mapped in the display interface of the smart wearable device, thereby facilitating the smart wearable device to track and identify the feature points of the mobile terminal, identify the interactive actions made by the current user to manipulate the mobile terminal, so that the current user can directly manipulate the extended screen component through the display interface of the smart wearable device, thereby improving the convenience of the interactive function between the smart wearable device and the mobile terminal.

[0106] Please refer to Figure 4 , Figure 4 A flowchart of a third embodiment of a method for interaction between a smart wearable device and a mobile terminal provided in an embodiment of the present application.

[0107] like Figure 4 As shown, based on the above Figure 3 In the illustrated embodiment, before step S201, the following steps are further included:

[0108] Step S301: Based on the image acquisition device, the outline coordinates of the mobile terminal are acquired;

[0109] In one embodiment, the smart wearable device recognizes the outline of the mobile terminal through an image acquisition device, and then calculates the outline coordinates of the outline of the mobile terminal in the two-dimensional coordinate system of the smart wearable device through a computer vision algorithm.

[0110] In one embodiment, computer vision refers to enabling computers and systems to extract meaningful information from images, videos, and other visual inputs and to take actions or provide recommendations based on that information. Computer vision training machines rely on cameras, data, and computer vision algorithms to distinguish objects, their distance, whether they are moving, and whether there are problems with the image.

[0111] In one embodiment, the computer vision algorithm may include feature extraction, feature point tracking, coordinate calculation, etc.

[0112] In one embodiment, feature extraction and matching is an important task in many computer vision applications, and is widely used in the fields of motion structure, image retrieval, object detection, etc. A feature may be a specific structure in an image, such as a point, edge, or object.

[0113] In one embodiment, the feature point tracking algorithm does not track the target as a whole, but matches and tracks a set of feature points (such as boundary lines, centroids, corner points, etc.) in continuous frame images. The feature-based tracking method mainly includes two steps: feature extraction and matching. The main advantage of this type of tracking method is that even if the target in the scene is partially blocked, as long as the feature points are visible, the target can be continuously tracked.

[0114] In one embodiment, coordinate calculation is to calculate the position coordinates of the target feature point in the coordinate system. Because the pictures collected by the image acquisition device of the smart wearable device are all within the display interface of the smart wearable device, the pictures collected by the image acquisition device all correspond to the same coordinate system. As long as the contour feature points of the mobile terminal are extracted and then the positions of the contour feature points in the picture are calculated, the contour coordinates of the mobile terminal can be obtained.

[0115] Step S302: obtaining a relative position relationship between at least one extended screen component and the mobile terminal;

[0116] In one embodiment, if Figure 5 As shown, the relative position relationship between the extended screen components and the mobile terminal depends on the position of the applications corresponding to each extended screen component on the screen of the mobile terminal, and the extended screen components can be distributed around the mobile terminal.

[0117] In one embodiment, according to the distribution position and arrangement order of the applications on the mobile terminal, when implementing the mapping of the extended screen components, the distribution position of the extended screen components is arranged in sequence according to the distribution position and arrangement order of the applications. When there are multiple applications, the distribution of the multiple applications on the mobile terminal is generally arranged in rows and columns, that is, the distribution position of each application on the mobile terminal can be represented by rows and columns. For example, application 1-first row and third column, etc.

[0118] In one embodiment, the extended screen components can be directly mapped on the display screen of the smart wearable device according to the distribution position of the applications, or they can be mapped on the side of the mobile terminal in columns and rows. For example, the extended screen components corresponding to the first row of applications are mapped above the mobile terminal, and the extended screen components are arranged in the order of the application arrangement; the extended screen components corresponding to the first column of applications are mapped on the left side of the mobile terminal, etc.

[0119] In one embodiment, in the display interface of the smart wearable device, the outline of the mobile terminal can be taken as the center, and the desktop applications corresponding to the extended screen components can be arranged in sequence on the side of the mobile terminal according to the positions of the desktop applications on the mobile terminal screen. For example, if the first desktop application is located in the middle of the left side of the mobile terminal screen, the extended screen component corresponding to the first desktop application is enlarged and displayed in the middle of the left side of the outline of the mobile terminal in the display interface of the smart wearable device.

[0120] Step S303: Based on the outline coordinates of the mobile terminal and the relative position relationship, the extended screen component is mapped and displayed in the display interface of the smart wearable device to activate the extended screen mode, wherein in the extended screen mode, the mobile terminal and the extended screen component are in a bound state.

[0121] In one embodiment, according to the outline coordinates and relative position relationship of the mobile terminal, the regional coordinates of the corresponding extended screen component area of ​​each extended screen component in the display interface of the smart wearable device are calculated, and then the extended screen component is mapped and displayed in the corresponding extended screen component area in the display interface to complete the activation of the extended screen mode. Figure 6 As shown, the first extended screen assembly is located on the left side of the mobile terminal, and there is only the first extended screen assembly on the left side of the mobile terminal. At this time, the outline coordinates of the mobile terminal can be ((100,10), (150,10), (100,110), (150,110)). According to the relative position relationship between the mobile terminal and the first extended screen assembly, the area coordinates of the extended screen assembly area corresponding to the first extended screen assembly can be calculated as ((10,10), (100,10), (10,110), (100,110)).

[0122] Furthermore, first size information of the mobile terminal and second size information of the extended screen component are obtained; based on the first size information, the second size information, the relative position relationship and the contour coordinates of the mobile terminal, the diagonal coordinate information of the extended screen component in the display interface is calculated; based on the diagonal coordinate information, the extended screen component area corresponding to each extended screen component is determined.

[0123] In one embodiment, the first size information of the mobile terminal includes the actual device size of the mobile terminal, including length size, width size, etc.

[0124] In one embodiment, the first size information of the mobile terminal is known, and can be obtained by measuring with a size measuring tool, or by querying from the official website of the mobile terminal manufacturer. The current user can preset the second size information of each extended screen component according to the first size information of the mobile terminal, such as proportionally enlarging the display size according to the number of extended screen components, so that the extended screen components can be completely and reasonably displayed around the mobile terminal.

[0125] For example, if there are four applications in the first row of the mobile terminal screen, the extended screen components of these four applications are all mapped to the upper side of the mobile terminal, and the first size information of the mobile terminal is 8×16 cm, that is, the length of the upper side of the mobile terminal is 8 cm, then the second size information of the above four extended screen components can be a maximum of 2×ncm, that is, the length of the four extended screen components can be a maximum of 2 cm, and the width can be adjusted according to the actual application situation (such as Figure 7 or the length of the four extended screen components can be up to 4×ncm, that is, the length of the four extended screen components can be up to 4cm (as shown in FIG. Figure 8 As shown), wherein the two extended screen components corresponding to the first application and the fourth application are respectively distributed at two upper diagonal positions of the mobile terminal.

[0126] In one embodiment, the enlargement ratio of the extended screen assembly can be set according to the relative position relationship between the mobile terminal and the extended screen assembly. For example, if the mobile terminal is rectangular, the enlargement ratio of the extended screen assembly on the long side can be set larger, and the enlargement ratio of the extended screen assembly on the short side can be set smaller.

[0127] In one embodiment, based on the first size information, the second size information, the relative position relationship and the outline coordinates of the mobile terminal, the diagonal coordinate information of the component frame of each extended screen component in the display interface of the smart wearable device can be calculated, such as the upper left corner coordinates and the lower right corner coordinates; based on the diagonal coordinate information, the area range of each extended screen component in the display interface of the smart wearable device can be indicated.

[0128] Exemplarily, the pixel size of the mobile terminal in the two-dimensional coordinate system is calculated based on the contour coordinates of the mobile terminal, that is, the number of pixels occupied by the length of the mobile terminal and the number of pixels occupied by the width. The ratio of the first size information and the pixel size is the size ratio of the actual size of the device and the pixel size of the two-dimensional coordinate system, and the ratio of the second size information and the size ratio is the pixel size of the extended screen component. According to the relative position relationship and the contour coordinates of the mobile terminal, the starting point coordinates of each extended screen component in the display interface can be known. According to the starting point coordinates and pixel size corresponding to each extended screen component, the diagonal coordinate information of the component frame of each extended screen component can be calculated, thereby obtaining the extended screen component area corresponding to each extended screen component.

[0129] In this embodiment, the outline coordinates of the mobile terminal are collected through the image acquisition device to determine the position of the mobile terminal in the display interface of the smart wearable device. Therefore, according to the relative position relationship between the extended screen component and the mobile terminal, the position of the extended screen component in the display interface of the smart wearable device is determined, thereby achieving accurate mapping of the extended screen component to avoid confusion in the mapping position of the extended screen component, adapting to the user's usage habits, and improving the user's experience of the interactive function between the smart wearable device and the mobile terminal.

[0130] See also Fig. 9 , Fig. 9 It is a structural schematic diagram of a first embodiment of an interactive device between a smart wearable device and a mobile terminal provided in the present application, and the interactive device between the smart wearable device and the mobile terminal is used to execute the aforementioned interactive method between the smart wearable device and the mobile terminal.

[0131] like Fig. 9 As shown, the interactive device 400 for the smart wearable device and the mobile terminal includes: a posture detection module 401, a corner point coordinate acquisition module 402, a component determination module 403 and a component operation module 404.

[0132] The posture detection module 401 is used to detect the current posture of the mobile terminal when the smart wearable device is in an extended screen interaction state with the mobile terminal;

[0133] A corner point coordinate acquisition module 402 is used to acquire the corner point coordinates of a target corner point on the mobile terminal when it is detected that the current posture of the mobile terminal matches a preset interactive operation action;

[0134] A component determination module 403, configured to determine a target extended screen component based on an extended screen component area corresponding to the corner point coordinates;

[0135] The component operation module 404 is used to perform a target operation on the target extended screen component based on the component operation instruction corresponding to the gesture operation when the current gesture action of the current user matches the preset gesture operation action.

[0136] In one embodiment, the interactive device 400 for the smart wearable device and the mobile terminal further includes an extended screen interactive state activation module, and the extended screen interactive state activation module includes:

[0137] A device unbinding instruction generating unit, configured to identify a preset unbinding action performed by the current user based on the image acquisition device, and generate a device unbinding instruction;

[0138] The binding state releasing unit is used to release the binding state between the mobile terminal and the extended screen component based on the device unbinding instruction, so that the smart wearable device and the mobile terminal are in the extended screen interactive state.

[0139] In one embodiment, the device unbinding instruction generating unit includes:

[0140] A hand motion picture acquisition subunit, configured to acquire hand motion pictures of the current user based on the image acquisition device to obtain a hand picture group, wherein the hand motion picture group includes at least two of the hand motion pictures;

[0141] A time series coordinate data acquisition subunit is used to extract the point coordinates of the target key points in the hand action picture based on a preset key point tracking algorithm to obtain the time series coordinate data of the target key points;

[0142] A collection time interval calculation subunit, configured to calculate the collection time interval of each of the coincident point coordinates based on the collection time corresponding to each of the coincident point coordinates when detecting that there are at least two coincident point coordinates in the time series coordinate data;

[0143] The device unbinding instruction generating subunit is used to determine that the hand movement of the current user is the preset unbinding movement when the collection time interval is less than the preset time interval, and generate the device unbinding instruction.

[0144] In one embodiment, the device unbinding instruction generating unit further includes:

[0145] An interval frame number determination subunit, used to determine the interval frame number between the hand action pictures corresponding to the coordinates of each of the coincident points based on the time series coordinate data;

[0146] The device unbinding instruction generating subunit is used to determine that the hand action of the current user is the preset unbinding action and generate the device unbinding instruction when the interval frame number is less than the preset interval frame number.

[0147] In one embodiment, the interactive device 400 for the smart wearable device and the mobile terminal further includes an extended screen mode activation module, and the extended screen mode activation module includes:

[0148] A contour coordinate acquisition unit, used to acquire contour coordinates of the mobile terminal based on the image acquisition device;

[0149] A relative position relationship acquisition unit, used to acquire a relative position relationship between at least one extended screen assembly and the mobile terminal;

[0150] An extended screen mode activation unit is used to map and display the extended screen component in the display interface of the smart wearable device based on the outline coordinates of the mobile terminal and the relative position relationship to activate the extended screen mode, wherein in the extended screen mode, the mobile terminal and the extended screen component are in a bound state.

[0151] In one embodiment, the extended screen mode activation module further includes:

[0152] A size information acquisition unit, used to acquire first size information of the mobile terminal and second size information of the extended screen assembly;

[0153] a diagonal coordinate information calculation unit, configured to calculate the diagonal coordinate information of the extended screen component in the display interface based on the first size information, the second size information, the relative position relationship, and the outline coordinates of the mobile terminal;

[0154] The extended screen component area determination unit is used to determine the extended screen component area corresponding to each extended screen component based on the diagonal coordinate information.

[0155] In one embodiment, the interactive device 400 for the smart wearable device and the mobile terminal further includes a gesture matching module, and the gesture matching module includes:

[0156] A gesture instruction library acquisition unit, used to acquire a preset gesture instruction library, wherein the gesture instruction library includes at least one gesture operation action and the component operation instructions corresponding to each gesture operation action;

[0157] an action similarity determination unit, configured to compare the current gesture action with each of the gesture operation actions in the gesture instruction library based on a gesture recognition algorithm, and determine the similarity between the current gesture action and each of the gesture operation actions;

[0158] The gesture operation action determining unit is used to determine the gesture operation action matching the current gesture action based on the similarity when the similarity is greater than or equal to a preset similarity.

[0159] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned embodiment of the interaction method between the smart wearable device and the mobile terminal, and will not be repeated here.

[0160] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Fig.10 Runs on the computer device shown.

[0161] See also Fig.10 , Fig.10 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0162] See also Fig.10 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0163] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any method for interacting with the smart wearable device and the mobile terminal.

[0164] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0165] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any method for interaction between the smart wearable device and the mobile terminal.

[0166] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0167] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0168] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0169] When the smart wearable device is in an extended screen interaction state with the mobile terminal, detecting the current posture of the mobile terminal;

[0170] When it is detected that the current posture of the mobile terminal matches the preset interactive operation action, collecting the corner point coordinates of the target corner point on the mobile terminal;

[0171] Determining a target extended screen component based on an extended screen component area corresponding to the corner point coordinates;

[0172] When the current gesture action of the current user matches the preset gesture operation action, the target operation is performed on the target extended screen component based on the component operation instruction corresponding to the gesture operation action.

[0173] In one embodiment, before the processor detects the current posture of the mobile terminal when the smart wearable device is in an extended screen interaction state with the mobile terminal, it is further used to implement:

[0174] Based on the image acquisition device, identifying the preset unbinding action made by the current user, and generating a device unbinding instruction;

[0175] Based on the device unbinding instruction, the binding state between the mobile terminal and the extended screen component is released, so that the smart wearable device and the mobile terminal are in the extended screen interaction state.

[0176] In one embodiment, when the processor implements the image acquisition device, identifies the preset unbinding action made by the current user, and generates the device unbinding instruction, it is used to implement:

[0177] Based on the image acquisition device, the hand motion pictures of the current user are acquired to obtain a hand picture group, wherein the hand motion picture group includes at least two of the hand motion pictures;

[0178] Based on a preset key point tracking algorithm, extract the point coordinates of the target key points in the hand action picture to obtain the time series coordinate data of the target key points;

[0179] When it is detected that there are at least two coincident point coordinates in the time series coordinate data, based on the collection time corresponding to each of the coincident point coordinates, the collection time interval of each of the coincident point coordinates is calculated;

[0180] When the collection time interval is less than the preset time interval, the hand movement of the current user is determined to be the preset unbinding movement, and the device unbinding instruction is generated.

[0181] In one embodiment, after the processor detects that there are at least two coincident point coordinates in the time series coordinate data, it is further configured to implement:

[0182] Based on the time series coordinate data, determining the interval frame number between the hand action pictures corresponding to the coordinates of each of the coincident points;

[0183] When the interval frame number is less than the preset interval frame number, the hand action of the current user is determined to be the preset unbinding action, and the device unbinding instruction is generated.

[0184] In one embodiment, before the processor implements the image acquisition device to identify the preset unbinding action made by the current user and generates the device unbinding instruction, it is further used to implement:

[0185] Based on the image acquisition device, the outline coordinates of the mobile terminal are acquired;

[0186] Acquire a relative position relationship between at least one extended screen component and the mobile terminal;

[0187] Based on the outline coordinates of the mobile terminal and the relative position relationship, the extended screen component is mapped and displayed in the display interface of the smart wearable device to activate the extended screen mode, wherein in the extended screen mode, the mobile terminal and the extended screen component are in a bound state.

[0188] In one embodiment, before the processor implements mapping and displaying the extended screen component in the display interface of the smart wearable device based on the outline coordinates of the mobile terminal and the relative position relationship to activate the extended screen mode, it is also used to implement:

[0189] Acquire first size information of the mobile terminal and second size information of the extended screen assembly;

[0190] Calculate the diagonal coordinate information of the extended screen component in the display interface based on the first size information, the second size information, the relative position relationship, and the outline coordinates of the mobile terminal;

[0191] Based on the diagonal coordinate information, the extended screen component area corresponding to each extended screen component is determined.

[0192] In one embodiment, when the current gesture action of the current user matches the preset gesture operation action, the processor is further used to implement, before performing the target operation on the target extended screen component based on the component operation instruction corresponding to the gesture operation action:

[0193] Acquire a preset gesture instruction library, wherein the gesture instruction library includes at least one gesture operation action and the component operation instruction corresponding to each gesture operation action;

[0194] Based on a gesture recognition algorithm, the current gesture action is compared with each of the gesture operation actions in the gesture instruction library to determine the similarity between the current gesture action and each of the gesture operation actions;

[0195] When the similarity is greater than or equal to a preset similarity, the gesture operation action matching the current gesture action is determined based on the similarity.

[0196] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the interaction methods between a smart wearable device and a mobile terminal provided in the embodiments of the present application.

[0197] The computer-readable storage medium may be an internal storage unit of the computer device described in the above embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the computer device.

[0198] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for interaction between a smart wearable device and a mobile terminal, characterized in that: The method comprises: When the smart wearable device is in an extended screen interaction state with the mobile terminal, detecting the current posture of the mobile terminal; When it is detected that the current posture of the mobile terminal matches the preset interactive operation action, collecting the corner point coordinates of the target corner point on the mobile terminal; Determining a target extended screen component based on an extended screen component area corresponding to the corner point coordinates; When the current gesture action of the current user matches the preset gesture operation action, the target operation is performed on the target extended screen component based on the component operation instruction corresponding to the gesture operation action.

2. The method for interaction between a smart wearable device and a mobile terminal according to claim 1, characterized in that: When the smart wearable device is in an extended screen interaction state with the mobile terminal, before detecting the current posture of the mobile terminal, the smart wearable device further includes: Based on the image acquisition device, identifying the preset unbinding action made by the current user, and generating a device unbinding instruction; Based on the device unbinding instruction, the binding state between the mobile terminal and the extended screen component is released, so that the smart wearable device and the mobile terminal are in the extended screen interaction state.

3. The method for interaction between a smart wearable device and a mobile terminal according to claim 2, characterized in that: The image acquisition device is based on identifying the preset unbinding action made by the current user and generating a device unbinding instruction, including: Based on the image acquisition device, the hand motion pictures of the current user are acquired to obtain a hand picture group, wherein the hand motion picture group includes at least two of the hand motion pictures; Based on a preset key point tracking algorithm, extract the point coordinates of the target key points in the hand action picture to obtain the time series coordinate data of the target key points; When it is detected that there are at least two coincident point coordinates in the time series coordinate data, based on the collection time corresponding to each of the coincident point coordinates, the collection time interval of each of the coincident point coordinates is calculated; When the collection time interval is less than the preset time interval, the hand movement of the current user is determined to be the preset unbinding movement, and the device unbinding instruction is generated.

4. The method for interaction between a smart wearable device and a mobile terminal according to claim 3, characterized in that: After detecting that there are at least two coincident point coordinates in the time series coordinate data, the method further includes: Based on the time series coordinate data, determining the interval frame number between the hand action pictures corresponding to the coordinates of each of the coincident points; When the interval frame number is less than the preset interval frame number, the hand action of the current user is determined to be the preset unbinding action, and the device unbinding instruction is generated.

5. The method for interaction between a smart wearable device and a mobile terminal according to claim 2, characterized in that: Before the image acquisition device is used to identify the preset unbinding action performed by the current user and generate the device unbinding instruction, the method further includes: Based on the image acquisition device, the outline coordinates of the mobile terminal are acquired; Acquire a relative position relationship between at least one extended screen component and the mobile terminal; Based on the outline coordinates of the mobile terminal and the relative position relationship, the extended screen component is mapped and displayed in the display interface of the smart wearable device to activate the extended screen mode, wherein in the extended screen mode, the mobile terminal and the extended screen component are in a bound state.

6. The method for interaction between a smart wearable device and a mobile terminal according to claim 5, characterized in that: Before mapping and displaying the extended screen component in the display interface of the smart wearable device based on the outline coordinates of the mobile terminal and the relative position relationship to activate the extended screen mode, the method further includes: Acquire first size information of the mobile terminal and second size information of the extended screen assembly; Calculate the diagonal coordinate information of the extended screen component in the display interface based on the first size information, the second size information, the relative position relationship, and the outline coordinates of the mobile terminal; Based on the diagonal coordinate information, the extended screen component area corresponding to each extended screen component is determined.

7. The method for interaction between a smart wearable device and a mobile terminal according to claim 1, characterized in that: When the current gesture action of the current user matches the preset gesture operation action, before performing the target operation on the target extended screen component based on the component operation instruction corresponding to the gesture operation action, the method further includes: Acquire a preset gesture instruction library, wherein the gesture instruction library includes at least one gesture operation action and the component operation instruction corresponding to each gesture operation action; Based on a gesture recognition algorithm, the current gesture action is compared with each of the gesture operation actions in the gesture instruction library to determine the similarity between the current gesture action and each of the gesture operation actions; When the similarity is greater than or equal to a preset similarity, the gesture operation action matching the current gesture action is determined based on the similarity.

8. An interactive device between a smart wearable device and a mobile terminal, characterized in that: The interactive device between the smart wearable device and the mobile terminal includes: The posture detection module is used to detect the current posture of the mobile terminal when the smart wearable device is in an extended screen interaction state with the mobile terminal; A corner point coordinate acquisition module, configured to acquire the corner point coordinates of a target corner point on the mobile terminal when detecting that the current posture of the mobile terminal matches a preset interactive operation action; A component determination module, used to determine a target extended screen component based on an extended screen component area corresponding to the corner point coordinates; The component operation module is used to perform a target operation on the target extended screen component based on the component operation instruction corresponding to the gesture operation when the current gesture action of the current user matches the preset gesture operation action.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the method for interaction between a smart wearable device and a mobile terminal as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by the processor, the steps of the method for interaction between the smart wearable device and the mobile terminal as described in any one of claims 1 to 7 are implemented.

Images