Handle positioning method and apparatus, wearable device, storage medium, and program product

Abstract

The present disclosure relates to a handle positioning method and device, a wearable device, a storage medium and a program product, which can reduce the false detection probability and the missed detection probability of optical handle positioning caused by environmental factors, and improve the recognition accuracy and robustness of optical handle positioning. The method comprises: acquiring a target image comprising a handle; detecting and positioning the handle in the target image based on a deep learning positioning model to obtain first region position information, the first region position information being used to indicate the position of the handle in the target image; wherein the deep learning positioning model is trained according to training sample data collected under different environments.

Description

Technical Field

[0001] This disclosure relates to the field of wearable technology, and more particularly to a handle positioning method, device, wearable device, storage medium, and program product. Background Technology

[0002] Positioning technology is widely used in fields such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), and is an important component of human-computer interaction. Among them, the positioning and tracking of optical controllers has been widely applied and developed in human-computer interaction fields such as interface interaction, games, and movies in wearable devices such as AR, VR, or MR.

[0003] Currently, most optical handle positioning methods for wearable devices are based on traditional image processing methods (such as image segmentation), which locate the handle's position in the image by using features such as the edge and brightness of the handle's light spot. Other methods are based on traditional machine learning methods (such as clustering), which classify the handle's light spot in the image by feature similarity, thereby obtaining a "cluster of points" to locate the handle's position.

[0004] The traditional optical handle positioning method is susceptible to environmental factors (e.g., lighting, shadows, and background natural light close to the handle's light spot), resulting in false detections and missed detections of the handle, and the algorithm has low robustness. Summary of the Invention

[0005] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a handle positioning method, apparatus, wearable device, storage medium, and program product, which can reduce the probability of false detection and missed detection of optical handle positioning caused by environmental factors, and improve the recognition accuracy and robustness of optical handle positioning.

[0006] A first aspect of this disclosure provides a handle positioning method, the method comprising: acquiring a target image including a handle; detecting and locating the handle in the target image based on a deep learning positioning model to obtain first region position information, the first region position information being used to indicate the position of the handle in the target image; wherein the deep learning positioning model is trained based on training sample data collected under different environments.

[0007] Optionally, before detecting and locating the handle in the target image based on the deep learning localization model to obtain the location information of the first region, the method further includes: acquiring the training sample data, each training sample in the training sample data including: a sample image including the handle and labeling information, the labeling information being used to indicate the position of the handle in the sample image; and training the deep learning model based on the training sample data to obtain the deep learning localization model.

[0008] Optionally, the deep learning model is a simplified version of the original lightweight network model MobileNet-v2.

[0009] Optionally, the backbone network of this deep learning model includes P network blocks-1 and Q network blocks-2, where P and Q are positive integers.

[0010] Optionally, P = Q, and the P network blocks-1 and Q network blocks-2 are connected alternately.

[0011] Optionally, after detecting and locating the handle in the target image based on the deep learning localization model to obtain the first region location information, the method further includes: tracking and locating the handle in at least one frame of images based on the first region location information and a tracking algorithm to obtain at least one second region location information, each second region location information being used to indicate the position of the handle in each frame of images, wherein the target image and the first frame of the at least one frame of images are two consecutively acquired frames of images.

[0012] Optionally, after detecting and locating the handle in the target image based on a deep learning model to obtain the first region position information, the method further includes: after obtaining the second region position information of each frame image through tracking and positioning, obtaining the final region position information of each frame image according to the following step S1; the final region position information is used to indicate the position of the handle in the corresponding frame image; wherein, S1 includes: when s is less than N, determining the second region position information corresponding to the s-th frame image as the final region position information of the s-th frame image; when s is greater than or equal to N, smoothing the N second region position information corresponding to the N consecutive frame images to obtain the final region position information of the s-th frame image, and the last frame image of the N consecutive frame images is the s-th frame image; wherein s is 0, 1, 2, 3..., and N is a positive integer.

[0013] A second aspect of this disclosure provides a handle positioning device, comprising: an acquisition module and a positioning module; the acquisition module is configured to acquire a target image including a handle; the positioning module is configured to detect and locate the handle in the target image acquired by the acquisition module based on a deep learning positioning model, thereby obtaining first region position information, the first region position information being used to indicate the position of the handle in the target image; wherein the deep learning positioning model is trained based on training sample data collected under different environments.

[0014] Optionally, the handle positioning device further includes: a training module; the acquisition module is further configured to acquire training sample data before detecting and locating the handle in the target image based on the deep learning positioning model to obtain the first region location information; each training sample in the training sample data includes: a sample image including the handle and labeling information, the labeling information being used to indicate the position of the handle in the sample image; the training module is configured to train the deep learning model based on the training sample data acquired by the acquisition module to obtain the deep learning positioning model.

[0015] Optionally, the deep learning model is a simplified version of the original lightweight network model MobileNet-v2.

[0016] Optionally, the backbone network of this deep learning model includes P network blocks-1 and Q network blocks-2, where P and Q are positive integers.

[0017] Optionally, P = Q, and the P network blocks-1 and Q network blocks-2 are connected alternately.

[0018] Optionally, the positioning module is further configured to detect and locate the handle in the target image based on a deep learning positioning model to obtain the first region position information, and then track and locate the handle in at least one frame of images based on the first region position information and a tracking algorithm to obtain at least one second region position information. Each second region position information is used to indicate the position of the handle in each frame of images. The target image and the first frame of the at least one frame of images are two consecutively acquired frames.

[0019] Optionally, the acquisition module is further configured to, after detecting and locating the handle in the target image based on a deep learning model to obtain the first region position information, and after obtaining the second region position information of each frame image through tracking and positioning, obtain the final region position information of each frame image according to the following step S1; the final region position information is used to indicate the position of the handle in the corresponding frame image; wherein, S1 includes: when s is less than N, determining the second region position information corresponding to the s-th frame image as the final region position information of the s-th frame image; when s is greater than or equal to N, smoothing the N second region position information corresponding to the N consecutive frame images to obtain the final region position information of the s-th frame image, and the last frame image of the N consecutive frame images is the s-th frame image; wherein, s is 0, 1, 2, 3..., and N is a positive integer.

[0020] A third aspect of this disclosure provides a wearable device, the wearable device including a processor, a memory, and a computer program stored in the memory and executable on the processor, the computer program implementing the handle positioning method as described in the first aspect when executed by the processor.

[0021] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the handle positioning method as described in the first aspect.

[0022] A fifth aspect of this disclosure provides a computer program product, wherein the computer program product includes a computer program that, when the computer program product is run on a processor, causes the processor to execute the computer program to implement the handle positioning method as described in the first aspect.

[0023] A sixth aspect of this disclosure provides a chip including a processor and a communication interface coupled to the processor, the processor being configured to execute program instructions to implement the handle positioning method as described in the first aspect.

[0024] Compared with the prior art, the technical solution provided in this disclosure has the following advantages: It can acquire a target image including a handle; based on a deep learning localization model, it detects and locates the handle in the target image to obtain first region location information, which is used to indicate the position of the handle in the target image; since the deep learning localization model is trained based on training sample data collected under different environments, it can detect and locate the handle in images collected under different environments. In this disclosure, the use of a deep learning localization model trained with training samples from different environments for multi-target image detection and localization can reduce the probability of false detection and missed detection caused by environmental factors such as complex backgrounds (especially since natural light and handle spot features are similar), rapid changes in handle spot features due to drastic changes in lighting conditions, and partial occlusion of the handle by objects, greatly improving the accuracy and robustness of the handle's position recognition in the image. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0026] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is one of the flowcharts illustrating the handle positioning method provided in this embodiment of the disclosure;

[0028] Figure 2 A second schematic flowchart illustrating the handle positioning method provided in this embodiment of the present disclosure;

[0029] Figure 3 This is a schematic diagram of the structure of a network block in a deep learning model provided in an embodiment of the present disclosure;

[0030] Figure 4 This is a schematic diagram of the structure of a deep learning model provided in an embodiment of the present disclosure;

[0031] Figure 5 The third schematic flowchart of the handle positioning method provided in the embodiments of this disclosure;

[0032] Figure 6 The fourth flowchart illustrates the handle positioning method provided in this embodiment of the disclosure.

[0033] Figure 7A structural block diagram of a handle positioning device provided in an embodiment of this disclosure;

[0034] Figure 8 This is a structural block diagram of a wearable device provided in an embodiment of the present disclosure. Detailed Implementation

[0035] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0036] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0037] The terms "first," "second," etc., used in this disclosure and in the claims are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0038] The detection and localization of the handle can be achieved by acquiring a target image including the handle; based on a deep learning localization model, the handle in the target image is detected and located to obtain first region location information, which is used to indicate the position of the handle in the target image; since the deep learning localization model is trained based on training sample data collected under different environments, the handle in images collected under different environments can be detected and located. In this embodiment of the present disclosure, the detection and localization of the handle in multiple target images using a deep learning localization model trained with training samples under different environments can reduce the probability of false detection and missed detection caused by environmental factors such as complex backgrounds (especially since natural light and handle spot features are similar), rapid changes in handle spot features due to drastic changes in lighting conditions, and partial occlusion of the handle by objects, which greatly improves the accuracy and robustness of the handle's position recognition in the image.

[0039] The wearable device in this disclosure can be a wearable device with VR function, augmented reality (AR) function, or MR function, such as VR headset, VR glasses, VR helmet, AR glasses, AR helmet, MR glasses, MR helmet, etc. The specific device can be determined according to the actual situation, and this disclosure does not limit it.

[0040] The execution subject of the handle positioning method provided in this embodiment can be the wearable device described above, or it can be a functional module and / or functional entity in the wearable device that can implement the handle positioning method. The specific implementation subject can be determined according to actual usage requirements, and this embodiment does not limit it.

[0041] The handle positioning method provided in this disclosure will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0042] like Figure 1 As shown, this disclosure provides a handle positioning method, which may include the following steps 101 to 102.

[0043] 101. Obtain the target image including the handle.

[0044] In this embodiment of the disclosure, the target image may be captured in real time by a camera, or it may be read from the device memory from a previously captured image, or it may be received from other devices. The specific method can be determined based on the actual situation, and this embodiment of the disclosure does not impose any limitations.

[0045] It should be noted that in this embodiment of the disclosure, the handle is an optical handle, and the target image includes a complete handle area image, and the clarity of the handle area in the target image is greater than or equal to a certain threshold.

[0046] The degree threshold can be determined according to the actual use situation, and this embodiment does not limit it.

[0047] Optionally, the images in the embodiments of this disclosure can be grayscale images, RGB images, YUV images, YCbCr images, HSI images, or HSV images, etc., which can be determined according to actual usage requirements, and this disclosure does not limit them.

[0048] 102. Based on a deep learning localization model, the handle in the target image is detected and located to obtain the location information of the first region.

[0049] The first region location information is used to indicate the position of the handle in the target image. The first region location information can be a range of pixel coordinates indicating the position of the handle in the target image.

[0050] The deep learning localization model was trained using training sample data collected under different environments.

[0051] In this embodiment of the disclosure, the training sample data collected under different environments includes training samples collected under different backgrounds, training samples collected under different lighting conditions, and other training samples under different environments. In particular, it includes training samples where the background or natural light and the characteristics of the handle's light spot are similar, as well as training samples under rapidly changing lighting conditions. The sample image in each training sample includes a complete handle region image, and the clarity of the handle region in the target image is greater than or equal to a certain threshold.

[0052] In this embodiment of the disclosure, a deep learning localization model can be used to detect and locate the handle in images acquired under different environments. This reduces the probability of false detection and missed detection of the handle, and improves the accuracy and robustness of the handle's position recognition in the image. Moreover, since the deep learning model locates the handle's position in the image by detecting a fixed-size image region, it does not suffer from significant time consumption due to high image resolution, achieving real-time localization. Furthermore, it can improve recognition accuracy even with high image resolution.

[0053] In this embodiment of the disclosure, a deep learning localization model needs to be obtained before using it for detection and localization. This model can be obtained from other devices or trained using training sample data. The model training is an offline process; in practical applications, it is not necessary to perform model training. It is sufficient to use the trained deep learning localization model to detect and locate the handle in the image.

[0054] Optionally, combined Figure 1 ,like Figure 2 As shown, prior to step 101 above, the handle positioning method provided in this embodiment may further include steps 103 to 104 as described below.

[0055] 103. Obtain the training sample data.

[0056] Each training sample in the training sample data includes: a sample image of the handle and labeling information. The labeling information indicates the position of the handle in the sample image, and the labeling information is the regional location information of the handle in the sample image. Each training sample in the training sample data may also include other information, which is not limited in this embodiment.

[0057] The training sample data consists of a large number of sample images containing optical handles collected in diverse environments. The optical handle region in each sample image is labeled to obtain the labeling information for each sample image.

[0058] It is understood that obtaining this training sample data includes: collecting a large number of sample images according to the requirements of the usage scenario and environment, and labeling the optical handle region information in each sample image to form training sample data for deep learning. It is important to note that during data collection, the integrity of the handle and its clarity must be greater than or equal to a certain threshold.

[0059] 104. Based on the training sample data, the deep learning model is trained to obtain the deep learning localization model.

[0060] The deep learning model consists of a backbone network and a loss function. The backbone network is the feature extraction part of the deep learning model, and the loss function is the target location regression part of the deep learning model.

[0061] It is understandable that a backbone network is used to train the deep learning localization model on training sample data collected in different environments, performing handle detection, localization, and learning. A loss function is then used to measure whether the deep learning localization model is properly trained, effectively ensuring that the trained deep learning localization model has high-accuracy localization results. The similarity between the detected and localized handles in the sample images and the corresponding labeled information in the sample images can be calculated using the loss function to verify the recognition accuracy of the deep learning localization model, thereby training a highly stable deep learning localization model.

[0062] In this embodiment of the disclosure, a deep learning localization model can be trained based on a large number of sample images collected according to usage scenario requirements and environmental requirements. This model is suitable for detecting and locating handles in images collected under different environments, and can improve detection accuracy and robustness.

[0063] Optionally, the deep learning model can be a convolutional neural network model. The specific convolutional neural network model can be determined according to the actual situation, and this disclosure does not limit it.

[0064] Optionally, the deep learning model is a simplified version of the original lightweight network model (MobileNet-v2).

[0065] It is understandable that this deep learning model is obtained by pruning MobileNet-v2. By simplifying MobileNet-v2, the resulting deep learning model can reduce algorithm complexity, improve detection efficiency, and increase detection accuracy.

[0066] Optionally, the backbone network of this deep learning model includes P network blocks-1 and Q network blocks-2, where P and Q are positive integers.

[0067] P and Q can be the same or different.

[0068] like Figure 3 As shown in (a), the structure of network block-1 includes: a first layer (input), a second layer (convolutional layer (conv1×1), activation function), a third layer (depthwise separable convolutional layer 1 (Dwise3×3), activation function), a fourth layer (convolutional layer (conv1×1), linear function), and a fifth layer (add function). Figure 3 As shown in (b), the structure of network block-2 includes: a first layer (input), a second layer (convolutional layer (conv1×1), activation function), a third layer (depthwise separable convolutional layer 2 (Dwise3×3, S=2), activation function), and a fourth layer (convolutional layer (conv1×1), linear function). Here, 1×1 and 3×3 represent the size of the convolutional kernel, S=2 represents the stride of the convolution, and the activation function can be a rectified linear unit function (relu) or relu6, or other activation functions; this embodiment does not limit the specific activation function.

[0069] In this embodiment of the disclosure, the backbone network of the deep learning model includes P network blocks-1 and Q network blocks-2, which can improve detection efficiency and detection accuracy.

[0070] Optionally, P = Q, and the P network blocks-1 and Q network blocks-2 are connected alternately.

[0071] In this configuration, P network blocks-1 and Q network blocks-2 are connected alternately, that is, one network block-1, one network block-2, then another network block-1, then another network block-2, ..., then another network block-1, then another network block-2 are connected alternately in sequence.

[0072] In this embodiment of the disclosure, P = Q, and the P network blocks-1 and Q network blocks-2 in the backbone network of the deep learning model are alternately connected, which can improve detection efficiency and detection accuracy.

[0073] In this embodiment of the disclosure, the loss function can be a region loss function, a mean square error (MSE) loss function, or other loss functions. This embodiment of the disclosure does not limit the loss function.

[0074] For example, P = Q = 2, such as Figure 4As shown, the backbone network of the deep learning model (the part within the dashed box) includes one input block and four network structure blocks. These four structure blocks consist of two network block-1s and two network block-2s (connected alternately: one network block-1, one network block-2, then another network block-1, and then another network block-2 in sequence). The activation function in each network block-1 and each network block-2 is ReLU. The loss function of the deep learning model is the region loss function.

[0075] Optionally, in this embodiment of the present disclosure, multiple training samples can be generated based on the target image and at least one frame image, and the deep learning localization model can be further trained based on the multiple training samples, so that the final deep learning localization model can be better applied to the detection and localization of handles in images acquired in different environments, thereby improving the detection accuracy and robustness of handle localization.

[0076] Optionally, embodiments of this disclosure may perform handle detection and localization on each frame of a video segment (real-time or non-real-time captured video segment) based on a deep learning localization model, or may perform handle detection and localization on the first frame (i.e., the target image) of a video segment based on a deep learning localization model to obtain the first region position information of the handle in the first frame image, and track and locate the handle in other frames of the video segment after the first frame image based on the first region position information and a tracking algorithm to obtain the second region position information of the handle in other frames.

[0077] It should be noted that the first frame image is the first frame image in the video clip that can be used to detect and locate the region of the complete handle based on a deep learning localization model.

[0078] For example, combined Figure 2 ,like Figure 5 As shown, after step 102 above, the handle positioning method provided in this embodiment may further include step 105 below.

[0079] 105. Based on the first region location information and the tracking algorithm, the handle in at least one frame of image is tracked and located to obtain at least one second region location information.

[0080] The location information of each second region is used to indicate the position of the handle in each frame of the image, and the target image and the first frame of the at least one frame of the image are two frames of images acquired consecutively.

[0081] Optionally, in this embodiment of the disclosure, the tracking algorithm may be a mean shift algorithm, a continuously adaptive mean-shift (CamShift) algorithm, or other tracking algorithms; this embodiment of the disclosure does not limit the specific algorithm used.

[0082] For example, taking the MeanShift algorithm as the tracking algorithm, after obtaining the position information of the first region of the handle in the target image, the position information of the first region is tracked and located in at least one frame of the image. The specific steps for tracking and locating the handle in at least one frame of the image based on the position information of the first region and the MeanShift algorithm are as follows:

[0083] Step 1: The handle rectangular region R detected in the current frame image (i.e., the position indicated by the handle's regional position information in the current frame image), obtain the histogram H of the rectangular region R, and perform normalization processing. Then, back-project the normalized H′ and denote it as P.

[0084] Step 2: Assume P has N×M pixels, and the coordinates are represented by Z. ij ={(x i ,y j )} i＝1...N,j＝1...M Z indicates ij The corresponding pixel value is P ij , (x c ,y c Let P be the coordinates of the center P of the region, and let vector P be the coordinates of the center P. and The MeanShift vector is then:

[0085]

[0086] Step 3: Calculate the coordinate offset of the center point: 's' is the frame number of the image. This means that the coordinates of the center point of the next frame are equal to the coordinates of the center point of the previous frame. The sum of.

[0087] Step 4: Repeat (2) and (3) until Where ε is an infinitesimal real number.

[0088] Repeat Steps 1 to 4 above to obtain the second region position information of the handle in each frame of the at least one frame image. It should be noted that when tracking and locating the handle in the k-th frame image of the at least one frame image, the current frame image in Step 1 above is the (k-1)-th frame image, where k is a positive integer, k = 1, 2, 3, ..., and when k = 1, the current frame image is the target image.

[0089] In this embodiment of the disclosure, based on the location information of the first region and the tracking algorithm, tracking and locating the handle in at least one frame of the image can improve the detection efficiency and accuracy of the handle, and can achieve real-time detection.

[0090] Optionally, combined Figure 5 ,like Figure 6 As shown, after step 102 above, the handle positioning method provided in this embodiment may further include step 106 below.

[0091] 106. After obtaining the second region location information of each frame image through tracking and positioning, the final region location information of each frame image is obtained according to the following step S1.

[0092] The final region location information is used to indicate the position of the handle in the corresponding frame image.

[0093] S1 includes: when s is less than N, determining the second region position information corresponding to the s-th frame image as the final region position information of the s-th frame image; when s is greater than or equal to N, smoothing the N second region position information corresponding to the consecutive N frame images to obtain the final region position information of the s-th frame image, and the last frame image of the consecutive N frame images is the s-th frame image; where s is 0, 1, 2, 3..., and N is a positive integer.

[0094] It can be understood that when s is less than N, the second region position information of the handle in the s-th frame image calculated based on the first region position information and the tracking algorithm is the final region position information of the s-th frame image; when s is greater than or equal to N, the N second region position information of the handle in the (s-N+1)-th to s-th frame images calculated based on the first region position information and the tracking algorithm are smoothed, and the region position information obtained after smoothing is the final region position information of the s-th frame image.

[0095] Optionally, in the embodiments of this application, the smoothing algorithm can be a mean smoothing algorithm, a state pool smoothing algorithm, or other smoothing algorithms. The specific algorithm can be determined according to the actual situation, and the embodiments of this application do not limit it.

[0096] For example, a state pool smoothing algorithm is adopted, and the specific process includes: S41, establishing a state pool (Buffer), continuously adding the obtained image region position information (first region position information and second region position information) to the state pool based on the recognition result of the handle positioning in the image; S42, smoothing the region position information in the state pool to obtain accurate and stable positioning results (final region position information). In this embodiment, the region where the handle of the image is located is a rectangular frame region, therefore the region position information is the rectangular frame position information. In step S42, the rectangular frame position information of consecutive frames is smoothed based on the state pool in S41, and the rectangular frame position information p of the current frame... cur The calculation is as follows:

[0097]

[0098] Where k represents the index number of the previous frame, and N represents the number of frames in the state pool that participate in the calculation. For example, N can be set to 10, which means smoothing the second region position information of the current frame image and the second region position information of the 9 frames before the current frame to obtain the final region position information of the current frame image.

[0099] In this embodiment, the final region location information is obtained by smoothing the second region location information obtained through tracking and positioning. This avoids jitter in the tracking and positioning results of the handle in consecutive frame images, making the positioning results of the handle in the image more stable and accurate.

[0100] Figure 7 This is a structural block diagram of a handle positioning device according to an embodiment of the present disclosure, such as... Figure 7 As shown, it includes: an acquisition module 701 and a positioning module 702; the acquisition module 701 is used to acquire a target image including a handle; the positioning module 702 is used to detect and locate the handle in the target image acquired by the acquisition module 701 based on a deep learning positioning model, to obtain first region position information, which is used to indicate the position of the handle in the target image; wherein, the deep learning positioning model is trained based on training sample data collected under different environments.

[0101] Optionally, the handle positioning device further includes: a training module; the acquisition module 701 is further configured to acquire training sample data before detecting and locating the handle in the target image based on the deep learning positioning model to obtain the first region location information; each training sample in the training sample data includes: a sample image including the handle and labeling information, the labeling information being used to indicate the position of the handle in the sample image; the training module is configured to train the deep learning model based on the training sample data acquired by the acquisition module 701 to obtain the deep learning positioning model.

[0102] Optionally, the deep learning model is a simplified version of the original lightweight network model MobileNet-v2.

[0103] Optionally, the backbone network of this deep learning model includes P network blocks-1 and Q network blocks-2, where P and Q are positive integers.

[0104] Optionally, P = Q, and the P network blocks-1 and Q network blocks-2 are connected alternately.

[0105] Optionally, the positioning module 702 is further configured to detect and locate the handle in the target image based on a deep learning positioning model to obtain first region position information, and then track and locate the handle in at least one frame of images based on the first region position information and a tracking algorithm to obtain at least one second region position information. Each second region position information is used to indicate the position of the handle in each frame of images. The target image and the first frame of the at least one frame of images are two consecutively acquired frames.

[0106] Optionally, the acquisition module 701 is further configured to, after detecting and locating the handle in the target image based on a deep learning model to obtain the first region position information, and after obtaining the second region position information of each frame image through tracking and positioning, obtain the final region position information of each frame image according to the following step S1; the final region position information is used to indicate the position of the handle in the corresponding frame image; wherein, S1 includes: when s is less than N, determining the second region position information corresponding to the s-th frame image as the final region position information of the s-th frame image; when s is greater than or equal to N, smoothing the N second region position information corresponding to the N consecutive frame images to obtain the final region position information of the s-th frame image, and the last frame image of the N consecutive frame images is the s-th frame image; wherein, s is 0, 1, 2, 3..., and N is a positive integer.

[0107] In this embodiment, each module can implement the handle positioning method provided in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0108] Figure 8 This is a schematic diagram of a wearable device provided in an embodiment of the present disclosure. It is used to illustrate the wearable device that implements any handle positioning method in the embodiments of the present disclosure and should not be construed as a specific limitation on the embodiments of the present disclosure.

[0109] like Figure 8 As shown, the wearable device 800 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the wearable device 800. The processor 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0110] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 808 including, for example, magnetic tape, hard disk, etc.; and communication devices 809. Communication device 809 allows wearable device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although a wearable device 800 with various devices is shown, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0111] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processor 801, it can perform the functions defined in any of the handle positioning methods provided in embodiments of this disclosure.

[0112] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0113] In some implementations, the client and server can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0114] The aforementioned computer-readable medium may be included in the wearable device or may exist independently without being assembled into the wearable device.

[0115] The aforementioned computer-readable medium carries one or more programs, which, when executed by the wearable device, cause the wearable device to: acquire a target image including a handle; detect and locate the handle in the target image based on a deep learning localization model to obtain first region location information, the first region location information being used to indicate the position of the handle in the target image; wherein the deep learning localization model is trained based on training sample data collected under different environments.

[0116] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0118] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0119] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0120] In the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0121] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0122] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0123] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A handle positioning method, characterized by, The method comprises: obtaining a target image comprising a handle; detecting and positioning the handle in the target image based on a deep learning positioning model to obtain first region position information, the first region position information being used to indicate the position of the handle in the target image; wherein the deep learning positioning model is obtained by training a deep learning model based on training sample data collected in different environments, the deep learning model is a model obtained by simplifying an original lightweight network model MobileNet-v2, the deep learning model comprises a backbone network and a loss function, the backbone network is a feature extraction part of the deep learning model, the loss function is a target position regression part of the deep learning model, and the backbone network comprises two network blocks-1 and two network blocks-2, and the two network blocks-1 and the two network blocks-2 are alternately connected.

2. The method of claim 1, wherein, Before the deep learning positioning model is used to detect and position the handle in the target image to obtain the first region position information, the method further comprises: obtaining the training sample data, each training sample in the training sample data comprising a sample image comprising the handle and label information, the label information being used to indicate the position of the handle in the sample image; training a deep learning model based on the training sample data to obtain the deep learning positioning model.

3. The method according to claim 1 or 2, characterized in that, After the deep learning positioning model is used to detect and position the handle in the target image to obtain the first region position information, the method further comprises: based on the first region position information and a tracking algorithm, tracking and positioning the handle in at least one image to obtain at least one second region position information, each second region position information being used to indicate the position of the handle in each image, the target image and a first image in the at least one image being two images continuously collected.

4. The method of claim 3, wherein, After the deep learning positioning model is used to detect and position the handle in the target image to obtain the first region position information, the method further comprises: after obtaining the second region position information of each image through tracking and positioning, obtaining the final region position information of each image according to the following step S1; the final region position information being used to indicate the position of the handle in the corresponding image; wherein S1 comprises: when s is less than N, determining the second region position information corresponding to the s-th image as the final region position information of the s-th image; when s is greater than or equal to N, performing smoothing processing on N second region position information corresponding to consecutive N images to obtain the final region position information of the s-th image, the last image of the consecutive N images being the s-th image; wherein s is 0, 1, 2, 3,..., and N is a positive integer.

5. A handle positioning device, characterized in that comprises: an obtaining module and a positioning module; the obtaining module is configured to obtain a target image comprising a handle; The positioning module is configured to detect and position the handle in the target image obtained by the acquisition module based on a deep learning positioning model to obtain first region position information, the first region position information being used to indicate a position of the handle in the target image. The deep learning positioning model is obtained by training a deep learning model based on training sample data collected in different environments, the deep learning model is a model obtained by simplifying an original lightweight network model MobileNet-v2, the deep learning model comprises a backbone network and a loss function, the backbone network is a feature extraction part of the deep learning model, the loss function is a target position regression part of the deep learning model, and the backbone network comprises two network blocks-1 and two network blocks-2, and the two network blocks-1 and the two network blocks-2 are alternately connected.

6. A wearable device, comprising: The computer program product comprises a computer program, and the computer program is executed by the processor to implement the handle positioning method in any one of claims 1 to 4. The computer program product comprises a computer program, and the computer program is executed by the processor to implement the handle positioning method in any one of claims 1 to 4.

7. A computer readable storage medium characterized in that, The computer program product comprises a computer program, and the computer program is executed by the processor to implement the handle positioning method in any one of claims 1 to 4.

8. A computer program product, characterised in that, ​

Images