Intelligent wearable device interface display method and device, intelligent wearable device and storage medium

Abstract

The invention provides an intelligent wearable device interface display method and device, an intelligent wearable device and a computer readable storage medium. The intelligent wearable device interface display method comprises the following steps: detecting displacement data of a wearer of the intelligent wearable device; when it is detected that the wearer has the target displacement data meeting the interaction condition, calling a camera of the intelligent wearable device to perform human body part detection; when a target human body part is detected, an interaction interface matched with the target human body part is displayed on the target human body part, and the part category of the target human body part is the same as the part category of the intent interaction part of the target displacement data. According to the method, the matching degree between the interface display form and the user intention display form can be improved, so that the interactive interface displays behavior actions conforming to user habits or behavior actions generally recognized by the user.

Description

Technical Field

[0001] The present application relates to the field of interface display technology, and specifically to a method and device for displaying an interface of a smart wearable device, a smart wearable device, and a computer-readable storage medium. Background Art

[0002] With the development of smart wearable technology, augmented reality devices and virtual reality devices have entered the lives of users. Interface display is the core technology of augmented reality devices and virtual reality devices. At present, the interface display of smart wearable devices such as augmented reality devices and virtual reality devices mainly projects the interactive interface according to the set projection parameters. However, the inventors of this application found in the actual research and development process that the current device display scheme has the problem of low matching between the interface display form and the user's intended display form. For example, the user uses the navigation interface provided by smart glasses to navigate during walking, but when the user wants to look down at the watch, the smart glasses still display the interactive interface according to the original projection position, and cannot adaptively move the interactive interface to the arm for display (such as according to the position and size of the arm). Summary of the invention

[0003] The present application provides a smart wearable device interface display method, device, smart wearable device and computer-readable storage medium, which can improve the matching degree between the interface display form and the user's intended display form, so that the interactive interface display conforms to the user's habitual behavior actions or the user's generally recognized behavior actions.

[0004] In a first aspect, the present application provides a method for displaying an interface of a smart wearable device, the method comprising:

[0005] Detect displacement data of the wearer of the smart wearable device;

[0006] When it is detected that the wearer has target displacement data that meets the interaction conditions, calling the camera of the smart wearable device to perform human body part detection;

[0007] When a target human body part is detected, an interaction interface matching the target human body part is displayed at the target human body part, wherein the part category of the target human body part is the same as the part category of the intended interaction part of the target displacement data.

[0008] In a second aspect, the present application provides an intelligent wearable device interface display device, the intelligent wearable device interface display device comprising:

[0009] A first detection module, used to detect displacement data of a wearer of the smart wearable device;

[0010] A second detection module is used to call the camera of the smart wearable device to detect human body parts when it is detected that the wearer has target displacement data that meets the interaction conditions;

[0011] A display module is used to display an interactive interface matching the target human body part at the target human body part when a target human body part is detected, wherein the part category of the target human body part is the same as the part category of the intended interactive part of the target displacement data.

[0012] In a third aspect, the present application also provides a smart wearable device, which includes a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, it executes any smart wearable device interface display method provided in the present application.

[0013] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the smart wearable device interface display method.

[0014] In the present application, on the first hand, when it is detected that the wearer of the smart wearable device has target displacement data that meets the interaction conditions, the camera is called to detect the human body part, and an interactive interface matching the target human body part is displayed on the target human body part. Since the target human body part displayed on the interactive interface has the same part category as the intended interaction part of the target displacement data, and the intended interaction part of the target displacement data can reflect the human body part that the user wants to display, the display position of the interactive interface is matched with the display position intended by the user (such as when the target displacement data of looking down at the watch is detected, the interactive interface is displayed on the arm), which can improve the matching degree between the interface display form and the user intended display form; on the second hand, by displaying the interactive interface matching the target human body part on the target human body part, different interface contents or interface layout forms can be set for different human body parts according to business needs, so that interactive interfaces with different contents or layout forms can be adaptively displayed for different human body parts, which can also improve the matching degree between the interface display form and the user intended display form to a certain extent. Thirdly, since the corresponding interaction interface is triggered only when the target displacement data that meets the interaction conditions is detected for the wearer of the smart wearable device, and the intended interaction part of the target displacement data can reflect the part of the human body that the user wants to use for display, the interaction interface can be displayed according to the user's habitual behavior or the user's generally recognized behavior (such as displaying the interaction interface on the arm when the target displacement data of looking down at the watch is detected), the interaction interface display is consistent with the user's habitual behavior or the user's generally recognized behavior, thereby improving the user experience of the interaction interface display. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 It is a schematic block diagram of the structure of a smart wearable device provided in an embodiment of the present application;

[0017] Figure 2 This is a flow chart of a method for displaying an interface of a smart wearable device provided in an embodiment of the present application;

[0018] Figure 3 It is a schematic diagram of a scene in which an interactive interface is displayed on a target human body part provided in an embodiment of the present application;

[0019] Figure 4 is a schematic diagram of a scenario of a user gesture action provided in an embodiment of the present application;

[0020] Figure 5 is a schematic diagram of changes in the interactive interface provided in an embodiment of the present application in response to user gestures;

[0021] Figure 6 This is a schematic diagram of an embodiment of the process of displaying the interface of a smart wearable device provided in an embodiment of the present application;

[0022] Figure 7 It is a schematic diagram of the structure of an embodiment of the smart wearable device interface display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0024] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0025] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0026] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art can recognize that the present application can also be implemented without using these specific details. In other examples, the known process will not be elaborated in detail to avoid unnecessary details that make the description of the present application embodiment obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in accordance with the embodiments of the present application.

[0027] The embodiments of the present application provide a method and device for displaying an interface of a smart wearable device, a smart wearable device, and a computer-readable storage medium. The smart wearable device interface display device can be integrated in a smart wearable device. The smart wearable device can be smart glasses, smart helmets, etc. The smart glasses can be AR (augmented reality) glasses, VR (Virtual Reality) glasses, MR (Mixed Reality) glasses, XR (eXtended Reality) glasses, etc. The smart helmet can be an AR helmet, a VR helmet, an MR helmet, an XR helmet, etc.

[0028] The executor of the smart wearable device interface display method of the embodiment of the present application may be the smart wearable device interface display device provided in the embodiment of the present application, or a smart wearable device integrated with the smart wearable device interface display device, wherein the smart wearable device interface display device may be implemented in hardware or software.

[0029] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0030] Figure 1 It is a schematic block diagram of the structure of a smart wearable device provided in an embodiment of the present application.

[0031] like Figure 1As shown, the smart wearable device 100 includes a processor 101 and a memory 102, and the processor 101 and the memory 102 are connected via a bus 103, such as an I2C (Inter-integrated Circuit) bus.

[0032] Specifically, the processor 101 is used to provide computing and control capabilities to support the operation of the entire smart wearable device 100. The processor 101 can be a central processing unit (CPU), and the processor 101 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0033] Specifically, the memory 102 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0034] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment scheme of the present application, and does not constitute a limitation on the smart wearable device to which the embodiment scheme of the present application is applied. The specific smart wearable device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0035] The processor 101 is used to run the computer program stored in the memory 102, and implement any one of the smart wearable device interface display methods provided in the embodiments of the present application when executing the computer program. For example, the processor 101 is used to run the computer program stored in the memory 102, and can implement the following steps when executing the computer program:

[0036] Detect displacement data of a wearer of a smart wearable device; when it is detected that the wearer has target displacement data that meets the interaction conditions, call the camera of the smart wearable device to detect human body parts; when the target human body part is detected, display an interaction interface matching the target human body part at the target human body part, wherein the part category of the target human body part is the same as the part category of the intended interaction part of the target displacement data.

[0037] It should be noted that technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the smart wearable device described above can refer to the corresponding process in the following smart wearable device interface display method embodiment, and will not be repeated here.

[0038] In the following, Figure 1 Taking the smart wearable device shown in as an example as the execution subject of the smart wearable device interface display method, the smart wearable device interface display method provided in the embodiment of the present application is described in detail. For the sake of simplicity and convenience of description, the execution subject will be omitted in the subsequent method embodiments.

[0039] See also Figure 2 , Figure 2 201 to 203, wherein:

[0040] 201. Detect displacement data of a wearer of a smart wearable device.

[0041] There are many ways to implement step 201, illustratively including:

[0042] (1) In some embodiments, the displacement data includes the head movement angle and the head movement speed. The head movement angle and the head movement speed of the wearer can be detected in turn to determine whether the wearer has target displacement data that meets the interaction conditions, so as to determine whether to trigger step 202. At this time, step 201 can specifically include the following steps 2011A to 2013A:

[0043] 2011A. Determine the head movement angle of the wearer according to the angle measurement data of the smart wearable device.

[0044] Among them, the inertial measurement unit (IMU) is a sensor system that can be used to measure the acceleration and angular velocity of an object. In the embodiment of the present application, the smart wearable device is equipped with an inertial measurement unit for detecting sensor data such as acceleration and angular velocity of the smart wearable device, so as to determine the head movement speed and movement angle of the wearer of the smart wearable device.

[0045] In some embodiments, the angle measurement data includes the angular velocity of the smart wearable device, and the velocity measurement data can be obtained through the inertial measurement unit of the smart wearable device. The inertial measurement unit can be equipped with a gyroscope to measure the angular velocity of the smart wearable device (that is, the rotation rate of the smart wearable device around a certain axis of the reference coordinate system, such as the x-axis; it can be understood that when the wearer wears the smart wearable device, the angular velocity of the smart wearable device is the angular velocity of the wearer's head, thereby providing information on the change in the angular velocity of the head around a certain direction). The wearer's head movement angle can be determined based on the angular velocity of the smart wearable device, the reference coordinate system of the smart wearable device and the relative position relationship between the wearer's head.

[0046] 2012A. When the head movement angle meets the preset angle condition, determine the wearer's head movement speed according to the speed measurement data of the smart wearable device.

[0047] In some embodiments, the speed measurement data includes the angular velocity and acceleration of the smart wearable device. The speed measurement data can be obtained through the inertial measurement unit of the smart wearable device. The inertial measurement unit can be equipped with a gyroscope to measure the angular velocity of the smart wearable device (i.e., the rotation rate of the smart wearable device around a certain axis of the reference coordinate system, such as the x-axis); the inertial measurement unit can be equipped with an accelerometer to measure the acceleration of the smart wearable device (i.e., the moving speed of the smart wearable device in the direction of a certain axis of the reference coordinate system, such as the x-axis. It can be understood that when the wearer wears the smart wearable device, the acceleration of the smart wearable device is the acceleration of the wearer's head, thereby providing information on the change in acceleration of the head in a certain direction); the wearer's head movement speed can be determined based on the angular velocity of the smart wearable device, the acceleration of the smart wearable device, and the relative position relationship between the reference coordinate system of the smart wearable device and the wearer's head.

[0048] There are many ways to implement step 2012A according to different settings of the preset angle conditions. For example, they include the following ways ①, ② and ③:

[0049] ① In some embodiments, the head movement angle includes the up-and-down movement angle of the wearer's head relative to the horizontal plane, and the preset angle condition is that the up-and-down movement angle is within a first preset angle range. At this time, step 2012A may specifically include: when the up-and-down movement angle of the wearer's head relative to the horizontal plane is within the first preset angle range, determining the head movement speed of the wearer according to the speed measurement data of the smart wearable device.

[0050] Among them, the specific value of the first preset angle range can be set according to the actual business scenario requirements. For example, when a user looks at a watch, the head is generally lowered about 15° to 20° relative to the horizontal plane. Therefore, when designing the interactive interface display for the user viewing the watch scenario, the first preset angle range can be set to [15°, 20°].

[0051] For example, in the scenario of "needing to determine whether the user has checked the watch to determine whether to display the interactive interface", it is necessary to determine whether the wearer has the target displacement data of checking the watch. In step 2011A, the angular velocity of the smart wearable device can be determined based on the angle measurement data of the inertial measurement unit of the smart wearable device, and the up and down movement angle of the wearer's head relative to the horizontal plane can be determined based on the angular velocity of the smart wearable device, the reference coordinate system of the smart wearable device and the relative position relationship between the wearer's head. When it is detected in step 2011A that the up and down movement angle is within the first preset angle range (such as [15°, 20°]), the process of detecting the head movement speed of the wearer in step 2012A is triggered. Otherwise, when it is detected in step 2011A that the up and down movement angle is outside the first preset angle range, it is determined that the wearer does not have the target displacement data that meets the interactive conditions.

[0052] ② In some embodiments, the head movement angle includes the left-right translation angle of the wearer's head relative to the horizontal plane, and the preset angle condition is that the left-right translation angle is within a second preset angle range. At this time, step 2012A may specifically include: when the left-right translation angle of the wearer's head relative to the horizontal plane is within the second preset angle range, determining the head movement speed of the wearer according to the speed measurement data of the smart wearable device.

[0053] Among them, the specific value of the second preset angle range can be set according to the actual business scenario requirements. For example, when a user views a watch, the head generally moves about 10° to 30° to the left or right relative to the horizontal plane. Therefore, when designing the interactive interface display for the user viewing the watch scenario, the second preset angle range can be set to [10°, 30°].

[0054] For example, in the scenario of "needing to determine whether the user has checked the watch to determine whether to display the interactive interface", it is necessary to determine whether the wearer has the target displacement data of checking the watch. In step 2011A, the angular velocity of the smart wearable device can be determined based on the angle measurement data of the inertial measurement unit of the smart wearable device, and the left and right translation angle of the wearer's head relative to the horizontal plane can be determined based on the angular velocity of the smart wearable device, the reference coordinate system of the smart wearable device and the relative position relationship between the wearer's head. When it is detected in step 2011A that the left and right translation angle is within the second preset angle range (such as [10°, 30°]), the process of detecting the head movement speed of the wearer in step 2012A is triggered. Otherwise, when it is detected in step 2011A that the left and right translation angle is outside the second preset angle range, it is determined that the wearer does not have the target displacement data that meets the interactive conditions.

[0055] ③ In some embodiments, the head movement angle includes the up-down movement angle and the left-right translation angle of the wearer's head relative to the horizontal plane, and the preset angle condition is that the up-down movement angle is in the first preset angle range, and the left-right translation angle is in the second preset angle range. At this time, step 2012A may specifically include: when the up-down movement angle of the wearer's head relative to the horizontal plane is in the first preset angle range, and the left-right translation angle of the wearer's head relative to the horizontal plane is in the second preset angle range, according to the speed measurement data of the smart wearable device, determine the head movement speed of the wearer. For example, in the scenario of "needing to determine whether the user has checked the watch action to determine whether to display the interactive interface", it is necessary to determine whether the wearer has the target displacement data of the watch checking action. In step 2011A, the angular velocity of the smart wearable device can be determined according to the angle measurement data of the inertial measurement unit of the smart wearable device, and the up-down movement angle of the wearer's head relative to the horizontal plane can be determined according to the angular velocity of the smart wearable device, and the left-right translation angle of the wearer's head relative to the horizontal plane can be determined according to the angular velocity of the smart wearable device, and the relative position relationship between the reference coordinate system of the smart wearable device and the wearer's head. When it is detected in step 2011A that the up-down movement angle is within the first preset angle range (e.g., [15°, 20°]) and the left-right translation angle is within the second preset angle range (e.g., [10°, 30°]), the process of detecting the head movement speed of the wearer in step 2012A is triggered. Otherwise, when it is detected in step 2011A that the up-down movement angle is outside the first preset angle range or the left-right translation angle is outside the second preset angle range, it is determined that the wearer does not have target displacement data that meets the interaction conditions.

[0056] 2013A. When the head movement speed meets the first preset speed condition, it is determined that the wearer has target displacement data that meets the interaction condition.

[0057] Among them, the first preset speed condition can be set according to the actual business scenario requirements, for example, the head movement speed can be within the first preset speed range.

[0058] Among them, the specific value of the first preset speed range can be set according to the actual business scenario requirements.

[0059] Among them, the target displacement data refers to the displacement data that meets the interaction conditions. The interaction conditions can be set according to the actual scene requirements. There is no specific restriction on the interaction conditions here. For example, in the implementation of steps 2011A to 2013A, the interaction conditions can be: the head movement angle meets the preset angle condition, and the head movement speed meets the first preset speed condition.

[0060] For example, in the scenario of "needing to determine whether the user has checked the watch to determine whether to display the interactive interface", it is necessary to determine whether the wearer has target displacement data for checking the watch. In step 2012A, the spatial position of the smart wearable device at each time point can be determined based on the positioning data of the smart wearable device. Then, the head movement speed of the wearer can be determined based on the angular velocity of the smart wearable device, the acceleration of the smart wearable device, and the relative position relationship between the reference coordinate system of the smart wearable device and the head of the wearer. When it is detected in step 2012A that the head movement speed is within the first preset speed range, step 2013 is triggered to determine that the wearer has target displacement data that meets the interactive conditions. Otherwise, when it is detected in step 2012A that the head movement speed is outside the first preset speed range, it is determined that the wearer does not have target displacement data that meets the interactive conditions.

[0061] (2) In some embodiments, the displacement data includes the overall movement speed, the head movement angle, and the head movement speed. The overall movement speed, the head movement angle, and the head movement speed of the wearer can be detected in sequence to determine whether the wearer has target displacement data that meets the interaction conditions, so as to determine whether to trigger step 202. At this time, step 201 can specifically include the following steps 2011B to 2014B:

[0062] 2011B. Determine the overall moving speed of the wearer according to the positioning data of the smart wearable device.

[0063] In some embodiments, the positioning data of the smart wearable device can be obtained through the positioning unit of the smart wearable device, wherein the positioning unit can be a GPS (Global Positioning System), which is a system that uses satellite signals for positioning, navigation and timing, and can provide accurate geographic location data. The positioning data collected by the positioning unit (such as GPS) of the smart wearable device can determine the spatial position of the smart wearable device at each time point, so that the overall movement speed of the wearer can be calculated based on the spatial position of the smart wearable device at each time point.

[0064] Since users will pause or slow down their movements when viewing or performing certain interactive interface operations, such as checking a watch, the embodiment of the present application uses positioning data to determine the wearer's overall movement speed to improve the accuracy of determining the viewing or interaction intention of the interactive interface.

[0065] 2012B. When the overall movement speed meets the second preset speed condition, determine the wearer's head movement angle according to the angle measurement data of the smart wearable device.

[0066] The second preset speed condition may be set according to actual business scenario requirements, for example, the overall moving speed may be within the second preset speed range.

[0067] The specific value of the second preset speed range may be set according to actual business scenario requirements, for example, it may be set to be less than or equal to 0.01 m / s.

[0068] For example, in the scenario of "needing to determine whether the user has checked the watch to decide whether to display the interactive interface", it is necessary to determine whether the wearer has target displacement data for checking the watch. In step 2011B, the spatial position of the smart wearable device at each time point can be determined based on the positioning data of the smart wearable device. Then, the overall movement speed of the wearer can be calculated based on the spatial position of the smart wearable device at each time point. When it is detected in step 2011B that the overall movement speed is within the second preset speed range (such as less than or equal to 0.01m / s), the process of detecting the head movement angle of the wearer in step 2012B is triggered. Otherwise, when it is detected in step 2012B that the overall movement speed is outside the second preset speed range, it is determined that the wearer does not have target displacement data that meets the interaction conditions.

[0069] 2013B. When the head movement angle meets the preset angle condition, determine the wearer's head movement speed according to the speed measurement data of the smart wearable device.

[0070] 2014B. When the head movement speed meets the first preset speed condition, it is determined that the wearer has target displacement data that meets the interaction condition.

[0071] (3) In some embodiments, the overall movement speed, head movement angle, and head movement speed of the wearer may be detected in parallel to determine whether the wearer has target displacement data that meets the interaction conditions, so as to determine whether to trigger step 202. In this case, step 201 may specifically include the following steps 2011C to 2014C:

[0072] 2011C. Determine the overall moving speed of the wearer according to the positioning data of the smart wearable device.

[0073] The implementation of step 2011C is similar to that of step 2011B. For details, please refer to the relevant instructions in the previous text, which will not be repeated here.

[0074] 2012C. Determine the head movement angle of the wearer according to the angle measurement data of the smart wearable device.

[0075] The implementation of step 2012C is similar to step 2011A. For details, please refer to the relevant instructions in the previous text and will not be repeated here.

[0076] 2013C. Determine the head movement speed of the wearer according to the speed measurement data of the smart wearable device.

[0077] The implementation of step 2013C is similar to step 2012A. For details, please refer to the relevant instructions in the previous text and will not be repeated here.

[0078] 2014C. When the overall movement speed meets the second preset speed condition, the head movement angle meets the preset angle condition, and the head movement speed meets the first preset speed condition, it is determined that the wearer has target displacement data that meets the interaction condition.

[0079] For the specific implementation of steps 2011C to 2014C, reference may be made to the relevant descriptions of steps 2011A to 2013A and steps 2011B to 2014B, which will not be repeated here.

[0080] 202. When the wearer has target displacement data that meets the interaction condition, call the camera of the smart wearable device to perform human body part detection.

[0081] Among them, smart wearable devices can be smart glasses, smart helmets, etc., smart glasses can be AR glasses, VR glasses, MR glasses, XR glasses, etc., and smart helmets can be AR helmets, etc.

[0082] In some embodiments, when there is target displacement data that meets the interaction conditions for the wearer, the camera of the smart wearable device can be called to collect a scene image; through a trained body part detection model, human body part detection is performed based on the scene image to obtain the human body part detection result of the scene image, where the human body part detection result includes the position information and part category of the target human body part.

[0083] Among them, human body key points refer to the key points of each part of the human body, such as head key points, eye key points, shoulder key points, elbow key points, wrist key points, knee key points, ankle key points, etc.

[0084] Exemplarily, performing human body area detection through a trained body part detection model includes the following steps:

[0085] 1. Key point calibration: Set one or more part category human body key points according to the scene requirements. For example, the set key points include: head key points, eye key points, shoulder key points, elbow key points, wrist key points, knee key points, ankle key points, etc.

[0086] 2. Collect sample images of the human body key points (such as wrist key points) of each part category i, and train a preset body part detection model so that the trained body part detection model can identify whether each pixel point in the image is a human body key point of part category i (such as whether it is a wrist key point), and according to the pixel points of the human body key points belonging to part category i in the image (such as the pixel points belonging to the wrist key point), divide the human body part of part category i from the image. Exemplarily, a convolutional neural network (CNN), a graph convolutional network (GCN), etc. can be used to construct a body part detection model for predicting whether each pixel point in the image belongs to the human body key point of part category i, and then divide the human body part of part category i.

[0087] 3. In step 203, when target displacement data that meets the interaction conditions is detected for the wearer of the smart wearable device, the camera of the smart wearable device is called to collect a scene image. Based on the scene image, keypoint detection for body part category i is performed through a trained body part detection model, and a pixel classification map i for indicating whether each pixel point in the scene image belongs to the body keypoints of body part category i is obtained (where the pixel value of each pixel point in the pixel classification map i is used to indicate whether the pixel point is located at the body keypoints of body part category i). Then, according to the pixel values of each pixel point in the pixel classification map i, it is determined whether the pixel point at the corresponding position in the scene image is the pixel point where the body keypoints of body part category i are located, so that the pixel points of all body keypoints of body part category i (such as wrist keypoints) in the scene image can be determined. By dividing the scene image according to the pixel points of all body keypoints of body part category i in the scene image, the body part of body part category i can be obtained. By analogy, all body parts of all body part categories in the scene image can be detected, such as head parts, eye parts, shoulder parts, elbow parts, wrist parts, knee parts, ankle parts, etc.

[0088] Furthermore, based on the body part detection result (such as according to the body part category of the body parts included in the scene image), it can be determined whether there is a target body part in the scene image whose body part category is the same as that of the intended interaction body part. If the scene image contains a body part whose body part category is the same as that of the intended interaction body part, it is determined that a target body part is detected, and the body part in the scene image whose body part category is the same as that of the intended interaction body part is used as the target body part. If the scene image does not contain a body part whose body part category is the same as that of the intended interaction body part, it is determined that no target body part is detected. For example, after detecting all body parts of all body part categories in the scene image, it is also necessary to compare whether there is a body part whose body part category is the same as that of the intended interaction body part. If there is, it is determined that a target body part is detected, and the body part whose body part category is the same as that of the intended interaction body part is used as the target body part. If not, it is determined that no target body part is detected. Among them, the intended interaction body part is the body part corresponding to the target displacement data for the user's intended interaction. For example, if the user's intended interaction body part is the left arm, target displacement data of "the overall movement speed is 0.1 m / s, the downward movement angle of the head is 15° and the leftward movement angle is 10°, and the movement speed of the head is within the first preset speed range" will be generated.

[0089] In this way, the camera of the smart wearable device can be called to capture scene images, and the scene images can be identified to determine whether there are human body parts within the scene field of view of the camera. When there are human body parts in the scene image, the scene image is further detected through the trained part detection model to determine the location information of the target human body part in the scene image, as well as the part category of the target human body part.

[0090] Furthermore, in order to prevent the device from heating and power consumption problems caused by the camera being turned on for a long time, after calling the camera of the smart wearable device, if no human body parts are detected for more than a preset time threshold, the camera is turned off. Among them, the specific value of the preset time threshold can be set according to the actual business scenario requirements. For example, in the scenario of "need to determine whether the user has checked the watch to decide whether to display the interactive interface", ordinary people will spend 2 to 5 seconds to confirm the time when checking the watch in most cases. If it is an emergency or a quick check of the time may only take 1 to 2 seconds, and if there are additional functions or more information is needed, it may take 5 to 10 seconds. Therefore, human body parts (such as arms) generally appear in the camera field of view within 10 seconds, so the preset time threshold can be set to 10 seconds.

[0091] 203. When a target human body part is detected, an interactive interface matching the target human body part is displayed on the target human body part.

[0092] The part category of the target human body part is the same as the part category of the intended interaction part of the target displacement data.

[0093] When a target human body part is detected, an interface projection area is determined from the target human body part according to the position information of the target human body part; and an interactive interface matching the target human body part is displayed in the interface projection area. In some embodiments, the interface projection area can be determined from the target human body part according to preset layout information and the position information of the target human body part. In some embodiments, the interface projection area can be determined from the target human body part according to the interface layout information matching the part category of the target human body part and the position information of the target human body part.

[0094] There are many ways to implement step 203, which include, for example:

[0095] (1) In some embodiments, each part category is correspondingly provided with an interface content, so as to adapt to different parts of the human body to display different contents of the interactive interface, thereby improving the interactive richness of the interface display. In this case, step 203 may include step 2031A:

[0096] 2031A. When a target human body part is detected, an interactive interface is displayed on the target human body part according to interface content that matches the part category of the target human body part.

[0097] For example, as shown in Table 1, assuming that there is a preset mapping relationship between each part category and the interface content as shown in Table 1, when a target human body part (such as an arm part, that is, the part category of the target human body part is an arm) is detected, an interactive interface containing interface content matching the part category of the target human body part (that is, displaying content 1) is displayed at the target human body part, such as Figure 3 As shown in (a) in FIG. 1 , different parts of the human body can be matched with interactive interfaces of different interface contents.

[0098] It is understandable that the interaction intention can be further identified based on the target displacement data; based on the interaction intention and the part category of the target human body part, the interface content that matches the part category of the target human body part and matches the interaction intention is determined, such as Figure 3 As shown in (a) and (b) in the figure, the same human body part can be matched with interactive interfaces of different interface contents to improve the richness of interaction.

[0099] Table 1

[0100] Part category Interface content arm Content 1 (such as timetable) palm Content 2 (such as input keyboard) … …

[0101] (2) In some embodiments, each part category is correspondingly provided with a kind of interface layout information, so as to adapt to different human body parts to display interactive interfaces of different sizes, thereby improving the fit between the interactive interface and the human body parts. In this case, step 203 may include step 2031B:

[0102] 2031B. When a target human body part is detected, an interactive interface is displayed on the target human body part according to interface layout information that matches the part category of the target human body part, wherein the interface layout information includes at least one of an interface ratio and an interface size.

[0103] Exemplarily, when a target body part is detected, an interface projection area is determined from the target body part based on the position information of the target body part and the interface layout information that matches the part category of the target body part; projection is performed on the interface projection area to display an interactive interface that matches the target body part on the target body part.

[0104] For example, as shown in Table 2, assuming that there is a preset mapping relationship between each part category and the interface layout information as shown in Table 2, when a target human body part (such as an arm, that is, the part category of the target human body part is an arm) is detected, the interactive interface is displayed on the target human body part according to the interface layout information matching the part category of the target human body part (that is, according to interface layout a). In this way, different human body parts can be matched with interactive interfaces with different interface layouts.

[0105] Table 2

[0106]

[0107] (3) In some embodiments, each part category is corresponding to a set of interface content, and each part category is corresponding to a set of interface layout information. In this case, step 203 may include step 2031C:

[0108] 2031C. When it is detected that the wearer of the smart wearable device has target displacement data that meets the interaction conditions, an interaction interface is displayed on the target human body part according to the interface content that matches the part category of the target human body part and the interface layout information that matches the part category.

[0109] Exemplarily, when a target body part is detected, an interface projection area is determined from the target body part based on the position information of the target body part and the interface layout information that matches the part category of the target body part; projection is performed on the interface projection area to display an interactive interface containing interface content that matches the part category of the target body part, so as to display an interactive interface that matches the target body part on the target body part.

[0110] Furthermore, the smart wearable device interface display method further includes: when a user gesture is detected, updating the display content of the interactive interface according to the user gesture. Figure 4 As shown, Figure 4 is a schematic diagram of a scene of user gestures provided in an embodiment of the present application. After the interactive interface is displayed on the target human body part, when the camera is turned on and recognizes a user gesture, such as a click or a double click, the smart wearable device can respond to the user gesture and update the display content of the interactive interface according to the corresponding user gesture. For example, Figure 5 As shown, Figure 5 is a schematic diagram of the change of the interactive interface in response to the user's gesture action in the embodiment of the present application, Figure 5 (a) is the SMS preview page. Figure 5 (b) is the SMS details page, and in step 203, the following is displayed on the arm: Figure 5 After the interactive interface shown in (a) in the figure, the user clicks on the SMS preview page and jumps to the following Figure 5 (b) is the SMS details page, which enables the display content of the interactive interface to be updated through user gestures.

[0111] For easier understanding, please refer to Figure 6 , the following takes "Inertial Measurement Unit Room IMU, Positioning Unit is GPS" as an example to explain the interface display process of the smart wearable device:

[0112] S1. Use IMU and GPS to detect whether the wearer has target displacement data that meets the interaction conditions. (Refer to Figure 6 "Detection of target displacement data that meets the interaction conditions" section)

[0113] At this point, the IMU and GPS can be judged by three conditions to determine whether preliminary step S2 is needed to turn on the camera to match the user's action:

[0114] The first condition: the current overall movement speed of the user meets the second preset speed condition. When the user needs to check the watch, he or she will generally pause or slow down the overall movement speed. At this time, GPS can be used to determine whether the user's overall movement speed meets the second preset speed condition. For example, when the user's overall movement speed is within the second preset speed range and approaches a certain value (such as less than or equal to 0.01m / s), it is considered that the overall movement speed meets the second preset speed condition, triggering the judgment of the second condition;

[0115] The second condition: the user's head movement angle meets the preset angle condition. At this time, the IMU can be used to determine whether the user's head movement angle meets the preset angle condition. For example, when the up and down movement angle is within the first preset angle range (for example, [15°, 20°]), and the left and right translation angle is within the second preset angle range (for example, [10°, 30°]), it is considered that the head movement angle meets the preset angle condition, triggering the judgment of the third condition;

[0116] The third condition: the user's head movement speed meets the first preset speed condition. At this time, the IMU can be used to determine whether the user's head movement speed meets the first preset speed condition. For example, when the head movement speed is within the first preset speed range, it is considered that the user's head movement speed meets the first preset speed condition, and step S2 is triggered.

[0117] S2, the camera is turned on for preview. (Refer to Figure 6 'Camera on' section)

[0118] In order to prevent heating and power consumption problems caused by the camera being turned on for a long time, if the camera is turned on and no human body area is detected for more than a preset time threshold, the camera will be turned off.

[0119] S3. Detect the target human body part (such as the arm). For detailed implementation, please refer to the previous description, which will not be repeated here. (Reference Figure 6 "Whether human body parts are detected" section)

[0120] S4. Project the relevant interactive interface on the target human body part (such as the arm). (Ref. Figure 6 (see "Device Display Related Interaction Interface" section)

[0121] Taking arm recognition as an example, based on the acquired arm position information, a corresponding interactive interface is displayed in the arm area.

[0122] S5. In response to the user's gesture, the interactive interface displayed on the target human body part (such as the arm) changes. (Refer to Figure 6 "Whether gestures are detected" and "Device responds to gestures" in the

[0123] From the above content, it can be seen that, on the first hand, when it is detected that the wearer of the smart wearable device has target displacement data that meets the interaction conditions, the camera is called to detect the human body part, and an interactive interface matching the target human body part is displayed on the target human body part. Since the target human body part displayed on the interactive interface is the same as the intended interaction part of the target displacement data, and the intended interaction part of the target displacement data can reflect the human body part that the user wants to display, the display position of the interactive interface is matched with the display position intended by the user (such as when the target displacement data of looking down at the watch is detected, the interactive interface is displayed on the arm), which can improve the matching degree between the interface display form and the user intended display form; on the second hand, by displaying the interactive interface matching the target human body part on the target human body part, different interface content or interface layout forms can be set for different human body parts according to business needs, so that different content or different layout forms of interactive interfaces can be adaptively displayed for different human body parts, which can also improve the matching degree between the interface display form and the user intended display form to a certain extent. Thirdly, since the corresponding interaction interface is triggered only when the target displacement data that meets the interaction conditions is detected for the wearer of the smart wearable device, and the intended interaction part of the target displacement data can reflect the part of the human body that the user wants to use for display, the interaction interface can be displayed according to the user's habitual behavior or the user's generally recognized behavior (such as displaying the interaction interface on the arm when the target displacement data of looking down at the watch is detected), the interaction interface display is consistent with the user's habitual behavior or the user's generally recognized behavior, thereby improving the user experience of the interaction interface display.

[0124] In addition, in order to better implement the smart wearable device interface display method in the embodiment of the present application, based on the smart wearable device interface display method, the embodiment of the present application also provides a smart wearable device interface display device, such as Figure 7 FIG. 1 is a schematic diagram of a structure of an embodiment of a smart wearable device interface display device provided in an embodiment of the present application. The smart wearable device interface display device 700 includes:

[0125] A first detection module 701, a first detection module, is used to detect displacement data of a wearer of a smart wearable device;

[0126] The second detection module 702 is used to call the camera of the smart wearable device to detect human body parts when it is detected that the wearer has target displacement data that meets the interaction conditions;

[0127] The display module 703 is used to display an interactive interface matching the target human body part at the target human body part when the target human body part is detected, wherein the part category of the target human body part is the same as the part category of the intended interactive part of the target displacement data.

[0128] In some embodiments, the display module 703 is specifically used for:

[0129] When a target human body part is detected, an interactive interface is displayed at the target human body part according to the interface content matched with the part category of the target human body part;

[0130] And / or, when a target human body part is detected, an interactive interface is displayed on the target human body part according to interface layout information that matches the part category of the target human body part, wherein the interface layout information includes at least one of an interface ratio and an interface size.

[0131] In some embodiments, the second detection module 702 is specifically used to:

[0132] When it is detected that the wearer has target displacement data that meets the interaction conditions, calling the camera of the smart wearable device to collect scene images;

[0133] By using the trained part detection model, human body part detection is performed based on the scene image to obtain a human body part detection result of the scene image, wherein the human body part detection result includes position information and part category of each human body part contained in the scene image.

[0134] In some embodiments, the second detection module 702 is specifically used to:

[0135] If the scene image contains a human body part with the same part category as the intended interaction part, it is determined that the target human body part is detected, and the human body part in the scene image with the same part category as the intended interaction part is used as the target human body part.

[0136] In some embodiments, the display module 703 is specifically configured to:

[0137] When the target human body part is detected, determine an interface projection area from the target human body part according to the position information of the target human body part;

[0138] Display an interaction interface matching the target human body part in the interface projection area.

[0139] In some embodiments, the displacement data includes the head movement angle and the head movement speed, and the first detection module 701 is specifically configured to:

[0140] Determine the head movement angle of the wearer according to the angle measurement data of the smart wearable device;

[0141] When the head movement angle meets the preset angle condition, determine the head movement speed of the wearer according to the speed measurement data of the smart wearable device;

[0142] When the head movement speed meets the first preset speed condition, it is determined that the wearer has target displacement data meeting the interaction condition.

[0143] In some embodiments, the displacement data further includes the overall movement speed, and the display module 703 is specifically configured to:

[0144] Determine the overall movement speed of the wearer according to the positioning data of the smart wearable device;

[0145] The determining the head movement angle of the wearer according to the angle measurement data of the smart wearable device includes:

[0146] When the overall movement speed meets the second preset speed condition, determine the head movement angle of the wearer according to the angle measurement data of the smart wearable device.

[0147] In some embodiments, the display module 703 is specifically configured to:

[0148] When a user gesture action is detected, update the display content of the interaction interface according to the user gesture action.

[0149] In some embodiments, the second detection module 702 is specifically configured to:

[0150] After the camera of the smart wearable device is called, if no human body part is detected for more than a preset time threshold, the camera is turned off.

[0151] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous smart wearable device interface display method embodiment, which will not be repeated here.

[0152] A person skilled in the art can understand that all or part of the steps in the above-mentioned smart wearable device interface display method can be completed by instructions, or by controlling related hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0153] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored. The computer programs can be loaded by a processor to execute any smart wearable device interface display method provided in the embodiment of the present application.

[0154] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0155] In the above-mentioned smart wearable device interface display device, computer-readable storage medium, and smart wearable device embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, refer to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-described smart wearable device interface display device, computer-readable storage medium, smart wearable device and its corresponding units can refer to the description of the smart wearable device interface display method in the above embodiment, and will not be repeated here.

[0156] The above is a detailed introduction to a smart wearable device interface display method, device, smart wearable device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for displaying an interface of a smart wearable device, characterized in that: The method comprises: Detect displacement data of the wearer of the smart wearable device; When it is detected that the wearer has target displacement data that meets the interaction conditions, calling the camera of the smart wearable device to perform human body part detection; When a target human body part is detected, an interaction interface matching the target human body part is displayed at the target human body part, wherein the part category of the target human body part is the same as the part category of the intended interaction part of the target displacement data.

2. The method for displaying an interface of a smart wearable device according to claim 1, characterized in that: When a target human body part is detected, an interactive interface matching the target human body part is displayed on the target human body part, including: When a target human body part is detected, an interactive interface is displayed at the target human body part according to the interface content matched with the part category of the target human body part; And / or, when a target human body part is detected, an interactive interface is displayed on the target human body part according to interface layout information that matches the part category of the target human body part, wherein the interface layout information includes at least one of an interface ratio and an interface size.

3. The method for displaying an interface of a smart wearable device according to claim 1, characterized in that: When it is detected that the wearer has target displacement data that meets the interaction condition, calling the camera of the smart wearable device to perform human body part detection includes: When it is detected that the wearer has target displacement data that meets the interaction conditions, calling the camera of the smart wearable device to collect scene images; By using the trained part detection model, human body part detection is performed based on the scene image to obtain a human body part detection result of the scene image, wherein the human body part detection result includes position information and part category of each human body part contained in the scene image.

4. The method for displaying an interface of a smart wearable device according to claim 3, characterized in that: The method further comprises: If the scene image contains a human body part of the same category as the intended interaction part, it is determined that a target human body part is detected, and the human body part of the same category as the intended interaction part in the scene image is used as the target human body part; When a target human body part is detected, an interactive interface matching the target human body part is displayed on the target human body part, including: When a target human body part is detected, determining an interface projection area from the target human body part according to position information of the target human body part; An interactive interface matching the target human body part is displayed in the interface projection area.

5. The method for displaying an interface of a smart wearable device according to claim 1, characterized in that: The displacement data includes a head movement angle and a head movement speed, and the detection of the displacement data of the wearer of the smart wearable device includes: Determining the head movement angle of the wearer according to the angle measurement data of the smart wearable device; When the head movement angle meets the preset angle condition, determining the head movement speed of the wearer according to the speed measurement data of the smart wearable device; When the head movement speed meets the first preset speed condition, it is determined that the wearer has target displacement data that meets the interaction condition.

6. The method for displaying an interface of a smart wearable device according to claim 5, characterized in that: The displacement data also includes the overall moving speed, and the method further includes: Determining the overall movement speed of the wearer according to the positioning data of the smart wearable device; The step of determining the head movement angle of the wearer according to the angle measurement data of the smart wearable device includes: When the overall movement speed meets the second preset speed condition, the head movement angle of the wearer is determined according to the angle measurement data of the smart wearable device.

7. The method for displaying an interface of a smart wearable device according to claim 1, characterized in that: The method further comprises: When a user gesture action is detected, the display content of the interactive interface is updated according to the user gesture action.

8. A smart wearable device interface display device, characterized in that: The smart wearable device interface display device comprises: A first detection module, used to detect displacement data of a wearer of the smart wearable device; A second detection module is used to call the camera of the smart wearable device to detect human body parts when it is detected that the wearer has target displacement data that meets the interaction conditions; A display module is used to display an interactive interface matching the target human body part at the target human body part when a target human body part is detected, wherein the part category of the target human body part is the same as the part category of the intended interactive part of the target displacement data.

9. A smart wearable device, characterized in that: It comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the smart wearable device interface display method according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the smart wearable device interface display method according to any one of claims 1 to 7.

Images