Gesture pointing recognition method, mobile device control method, mobile device and medium

Abstract

The application relates to the technical field of mobile device control, and discloses a gesture pointing recognition method, a mobile device control method, a mobile device and a medium. The gesture pointing recognition method comprises the following steps: collecting a preset number of video frames that are continuous in time sequence to obtain a target hand image sequence; based on the target hand image sequence, initial coordinate information of each preset key point of the target hand is extracted; based on the initial coordinate information of each preset key point, geometric features representing the spatial pointing relationship of the hand are calculated; each frame of geometric features is processed through a single-frame geometric feature extraction network to output a single-frame feature vector with geometric semantics; a preset number of continuous single-frame feature vectors are combined into a time sequence input sequence, and the time sequence input sequence is analyzed across frames through a time sequence modeling network to output a stabilized time sequence fusion feature; based on the time sequence fusion feature, gesture action classification and spatial pointing direction calculation are synchronously performed to obtain a gesture pointing result.

Description

Technical Field

[0001] This application relates to the field of mobile device control technology, and in particular to a gesture pointing recognition method, a mobile device control method, a mobile device, and a medium. Background Technology

[0002] Existing mobile device control technologies generally suffer from unnatural interaction, inaccurate pointing, and untimely response. Traditional methods mainly rely on remote control handles or mobile applications, requiring users to hold additional devices or frequently operate graphical interfaces. This involves numerous steps, high learning costs, and makes it difficult to provide intuitive and real-time guidance on the device's movement direction and target location. Another approach uses conventional gesture recognition technology, which can identify basic actions such as clenching a fist, waving, and opening a palm. However, its recognition targets are only discrete action categories, lacking the ability to quantitatively perceive the specific pointing direction of gestures in three-dimensional space. It cannot output continuous parameters such as azimuth and pitch angles that can be used for spatial positioning, and therefore cannot support the device to move autonomously in the direction indicated by the user or perform space-related tasks. Summary of the Invention

[0003] In view of this, embodiments of this application provide a gesture pointing recognition method, a mobile device control method, a mobile device, and a computer-readable storage medium.

[0004] In a first aspect, embodiments of this application provide a gesture pointing recognition method, including: A predetermined number of consecutive video frames are acquired in time sequence to obtain a sequence of images of the target hand; Based on the target hand image sequence, the initial coordinate information of each preset key point of the target hand is extracted; The geometric features representing the spatial pointing relationship of the hand are calculated based on the initial coordinate information of each of the preset key points; The geometric features of each frame are processed by a single-frame geometric feature extraction network to output a single-frame feature vector with geometric semantics. The preset number of consecutive single-frame feature vectors are combined to form a temporal input sequence, and the temporal input sequence is analyzed across frames through a temporal modeling network to output stable temporal fusion features. Based on the aforementioned temporal fusion features, gesture action classification and spatial pointing direction calculation are performed simultaneously to obtain gesture pointing results; wherein, the gesture pointing results include gesture action type and gesture pointing direction.

[0005] Secondly, embodiments of this application provide a mobile device control method, including: Based on the gesture pointing direction obtained from any of the foregoing embodiments, the current pose of the mobile device in the world coordinate system it has initialized, and the effective action distance measured along the gesture pointing direction, the spatial coordinates of the target point in the world coordinate system are determined. A navigation path is generated based on the spatial coordinates of the target point, the current pose of the mobile device, and environmental obstacle information; Control the mobile device to move along the navigation path to the target point and perform a task corresponding to the gesture type.

[0006] Thirdly, embodiments of this application provide a mobile device, the mobile device including a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the above-described gesture pointing recognition method and / or the above-described mobile device control method.

[0007] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed on a processor, implements the gesture pointing recognition method and / or the mobile device control method described above.

[0008] The embodiments of this application have the following beneficial effects: This application acquires continuous video frames, extracts the initial coordinates of preset key points of the hand, and calculates the geometric features representing spatial pointing relationships; after processing by a single-frame feature extraction network and a temporal modeling network, it outputs stable temporal fusion features, and simultaneously performs gesture action classification and spatial pointing direction calculation to obtain a gesture pointing result containing gesture action type and gesture pointing direction. This method does not rely on image texture and background information, but only on modeling the spatial relationship between key points, improving the applicability under different lighting and distance conditions, and achieving consistent output of action intent and spatial pointing. Furthermore, based on the gesture pointing direction, the device's pose in the world coordinate system established by itself during initialization, and the effective action distance measured along the direction, the spatial coordinates of the target point in the coordinate system are determined; then, combined with the target point coordinates, the device's current pose, and obstacle information, a navigation path is generated, controlling the device to move to the target point and execute the corresponding task. The entire process directly maps the user's natural gestures into executable targets and actions in physical space, achieving contactless, low-latency, and high-precision spatial intent-driven intelligent control without the need for a remote control, APP interface, or preset instruction set. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This paper illustrates a first flowchart of a gesture pointing recognition method according to an embodiment of the present application. Figure 2 This paper illustrates a second flowchart of a gesture pointing recognition method according to an embodiment of this application. Figure 3 A third flowchart of the gesture pointing recognition method according to an embodiment of this application is shown; Figure 4 The fourth flowchart of the gesture pointing recognition method according to an embodiment of this application is shown; Figure 5 The fifth flowchart of the gesture pointing recognition method according to an embodiment of this application is shown; Figure 6 A first flowchart of a mobile device control method according to an embodiment of this application is shown; Figure 7 A second flowchart of a mobile device control method according to an embodiment of this application is shown. Detailed Implementation

[0011] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0012] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0013] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0014] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0015] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0016] Before describing the method of the embodiments of this application, the application object of the embodiments of this application will be described first. The method of this embodiment is mainly applied to mobile devices, which are intelligent devices with autonomous movement capabilities and equipped with a visual acquisition unit, a distance measurement sensor and a motion execution mechanism. During the initialization phase, the mobile device constructs its own world coordinate system and obtains its real-time pose in the world coordinate system through synchronous positioning and mapping technology or a global navigation satellite system. It should be noted that the mobile devices in this embodiment include, but are not limited to, sweeping robots, aerial photography drones, industrial inspection robots, etc. All types of mobile devices have the hardware foundation required to execute the method of the embodiments of this application, that is, they can acquire video frames, detect key points of the hand, calculate the direction of the gesture, determine the spatial coordinates of the target point, generate a navigation path and drive the body to move to the target point to perform the corresponding task.

[0017] The following examples illustrate the gesture recognition method.

[0018] Figure 1 A schematic flowchart of a gesture pointing recognition method according to an embodiment of this application is shown. Exemplarily, the gesture pointing recognition method includes steps S100-S600: Step S100: Acquire a preset number of consecutive video frames in time sequence to obtain a target hand image sequence.

[0019] In this step, the preset number of video frames refers to consecutive video frames that are adjacent in time and without intervals. For example, it can be data collected under the condition that the frame rate is not less than 30fps. The acquisition process can be performed by the visual acquisition unit on the mobile device. The visual acquisition unit and the subsequent hand key point detection unit form a data flow pipeline, thereby ensuring that the video frames are input to the subsequent processing module in the original temporal order.

[0020] It should be noted that the preset number of video frames is at least 3 frames to meet the basic requirements of cross-frame temporal modeling for dynamic feature change analysis. It can be understood that the preset number is not specifically limited in this embodiment; the limitation to at least 3 is mainly to enable the subsequent temporal modeling network to effectively compare the fluctuation trends of key point geometric features between adjacent frames and implement adaptive weighting based on the fluctuation amplitude, thereby outputting stable temporal fusion features. In other words, the preset number can be adjusted according to the actual application scenario, the gesture stability requirements, the computing power of the mobile device, and real-time constraints, including but not limited to 3, 4, or 5 frames.

[0021] Step S200: Based on the target hand image sequence, extract the initial coordinate information of each preset key point of the target hand.

[0022] In this embodiment, the preset key points are 21 locations that can stably represent the spatial pointing posture of the hand anatomically, including: wrist reference point, thumb tip point, thumb base point, thumb proximal joint point, thumb distal joint point, index finger tip point, index finger proximal joint point, index finger middle joint point, index finger distal joint point, middle finger tip point, middle finger proximal joint point, middle finger middle joint point, middle finger distal joint point, ring finger tip point, ring finger proximal joint point, ring finger middle joint point, ring finger distal joint point, little finger tip point, little finger proximal joint point, little finger middle joint point, little finger distal joint point; wherein, in this embodiment, the wrist reference point is set as the common reference origin for all subsequent geometric calculations, that is, the subsequent reference key point.

[0023] In this step, after obtaining the target hand image sequence, the initial coordinate information of each preset key point of the target hand can be obtained through the hand key point detection unit. For example, the extraction work can be performed based on the MediaPipe in the hand key point detection unit.

[0024] Furthermore, the initial coordinate information in this step can be two-dimensional pixel coordinates, and the initial coordinate set formed by its 21 key points can be represented as follows: .

[0025] Step S300: Calculate the geometric features representing the spatial pointing relationship of the hand based on the initial coordinate information of each preset key point.

[0026] Among them, geometric features are used to depict the pointing posture of the hand in three-dimensional space. Their composition does not depend on the overall texture or background information of the image, but is entirely derived from the spatial geometric relationship between preset key points. These geometric features include three categories: the first category is the Euclidean distance between any two preset key points (the relative distance between pairs of key points); the second category is the angle between vectors formed with the reference key point as the starting point and the other preset key points as the ending points (the angle between the pointing vectors of each finger); the third category is the degree of curvature of the joint chain formed by adjacent joints on a single finger (the curvature state of the joint chain of a single finger).

[0027] In some implementations, such as Figure 2 As shown, step S300 includes steps S310-S320: Step S310: Using the reference key point as the origin, perform relative coordinate normalization on the initial coordinate information of other preset key points to obtain the corresponding normalized coordinate information.

[0028] Since this embodiment uses the wrist key point as the reference key point, in this step, the wrist key point is used as the global reference origin. The coordinates of the wrist key point will be described later. As the standard, the rest , , representing the coordinates of the remaining 20 key points.

[0029] This step is the key point of the wrist. Using the global reference origin, a relative coordinate transformation is performed on the remaining 20 key points. The transformation formula is as follows: After this transformation, the coordinates of all 20 preset key points are expressed as offsets relative to the reference key points. This eliminates the scale differences in absolute pixel coordinates caused by changes in the distance between the user's hand and the visual acquisition unit, device translation, or imaging angle shift, thus ensuring spatial scale invariance for subsequent geometric feature calculations.

[0030] Step S320: Calculate the geometric features representing the spatial pointing relationship of the hand based on the normalized coordinate information.

[0031] For the relative distance between any two keypoints: this relative distance is the Euclidean length of the line segment formed by any two preset keypoints in the image plane; for the i-th preset keypoint and the j-th preset keypoint, where i and j are both integers between 1 and 20 and i is not equal to j, their relative distance is... It can be done through formula get. In this embodiment, the relative distances between the reference key point and the second preset key point, the relative distance between the reference key point and the sixth preset key point, the relative distance between the sixth preset key point and the seventh preset key point, the relative distance between the seventh preset key point and the eighth preset key point, and the relative distance between the eighth preset key point and the ninth preset key point can be calculated. Among them, the second preset key point is the tip of the thumb, the sixth preset key point is the tip of the index finger, the seventh preset key point is the proximal joint of the index finger, the eighth preset key point is the middle joint of the index finger, and the ninth preset key point is the distal joint of the index finger. The above distances collectively characterize the degree of finger extension and the spatial span pointing towards the main axis.

[0032] The angle between the pointing vectors of each finger is the smallest angle between two two-dimensional vectors formed with the reference key point as the common starting point and different preset key points as the ending points. The angle between the first pointing vector formed with the reference key point as the starting point and the i-th preset key point as the ending point, and the second pointing vector formed with the reference key point as the starting point and the j-th preset key point as the ending point, is obtained by the following calculation: First, calculate the horizontal component of the first pointing vector. With vertical component The horizontal component of the second pointing vector With vertical component Then calculate the dot product of the two vectors, its value is equal to Then calculate the magnitude of the first pointing vector, which is equal to the length of the first pointing vector. The square of and Take the square root of the sum of the squares of the first vector; then calculate the magnitude of the second pointing vector, which is equal to the sum of the squares of the first vectors. The square of and The sum of the squares is taken as the square root; finally, the cosine of the included angle is equal to the dot product divided by the product of the magnitudes of the first pointing vector and the second pointing vector.

[0033] In this embodiment, the angle between the pointing vector formed by the reference key point as the starting point and the sixth preset key point as the ending point and the pointing vector formed by the reference key point as the starting point and the seventh preset key point as the ending point can be calculated. This angle represents the degree of extension of the index finger joint chain and is the core basis for judging the stability of the pointing posture of a single finger.

[0034] For the bending state of a single finger joint chain: the bending state is the angle of the broken line formed by three consecutive preset key points on the same finger; for the k-th preset key point, the (k+1)-th preset key point, and the (k+2)-th preset key point, where k is an integer satisfying consecutive numbering, the bending state is obtained through the following calculation: first, determine the first line segment vector formed by starting from the k-th preset key point and ending at the (k+1)-th preset key point; then determine the second line segment vector formed by starting from the (k+1)-th preset key point and ending at the (k+2)-th preset key point; then calculate the angle between the first line segment vector and the second line segment vector; in this embodiment, for the index finger joint chain, the angle between the first line segment vector starting from the seventh preset key point and ending at the eighth preset key point, and the second line segment vector starting from the eighth preset key point and ending at the ninth preset key point can be calculated sequentially; this angle represents the degree of bending of the middle joint of the index finger, used to distinguish between fully extended pointing and semi-flexed gestures, and improve the discrimination accuracy of gesture action classification.

[0035] In step S400, the geometric features of each frame are processed by a single-frame geometric feature extraction network to output a single-frame feature vector with geometric semantics.

[0036] The single-frame geometric feature extraction network is a lightweight neural network structure (such as MobileNetV2), whose core contains at least one depthwise separable convolutional module with an inverse residual structure. This network does not receive the original image data, but only the set of single-frame geometric features calculated by step S320. The single-frame feature vector is a one-dimensional vector with a fixed dimension. Its different segments encode three types of geometric features: the relative position of key points, the distance between key points, and the angle between vectors. This ensures that the downstream temporal modeling unit can directly extract the corresponding semantic information based on the dimensional partition.

[0037] In some implementations, such as Figure 3 As shown, step S400 specifically includes steps S410-S430: Step S410: Organize and reshape the geometric features of a single frame into a two-dimensional feature map.

[0038] Among them, the single-frame geometric features are an ordered combination of the three types of geometric parameters output in step S320, including 20 sets of normalized coordinates ( ), the relative distance between several pairs of key points The parameters include the angles between several sets of pointing vectors and the bending states of several sets of single-finger joint chains. All parameters are concatenated into a one-dimensional numerical sequence in a preset order. The one-dimensional numerical sequence is then reshaped into a two-dimensional matrix with rows and columns. It should be noted that this reshaping operation does not change the numerical values ​​and relative order of the parameters, but only adjusts their spatial arrangement to adapt to the requirements of the subsequent convolution operation for the two-dimensional input format.

[0039] Step S420: Multi-level feature extraction is performed on the two-dimensional feature map through a depthwise separable convolution module to obtain a multi-channel feature map.

[0040] In the feature extraction process, the channel dimension is first expanded and then compressed, and the correlation between various geometric features in the single-frame feature vector is independently modeled in the expanded high-dimensional space.

[0041] Exemplarily, the depthwise separable convolutional module includes sequentially connected up-dimensional convolutional layers, depthwise convolutional layers, and down-dimensional convolutional layers; First, pointwise convolution operations are performed on the 2D feature map through dimensionality-increasing convolutional layers. 1×1 convolutions expand the number of input channels to a higher dimension, enabling the network to model the independent variations and coupling relationships within three types of geometric features: relative keypoint positions, keypoint distances, and vector angles, in a high-dimensional space. Then, spatial convolution operations (e.g., 3×3 depthwise convolutions) are performed on the dimensionality-increasing feature map through depthwise convolutional layers. This operation is performed independently within each channel, without mixing feature values ​​from different channels. This ensures that the relative keypoint positions, keypoint distances, and vector angles remain logically isolated during spatial association modeling, avoiding geometric semantic confusion. Finally, pointwise convolution operations are performed on the feature map output from the depthwise convolution through dimensionality-reducing convolutional layers, compressing the number of channels to the target dimension, completing feature dimensionality reduction and redundancy filtering. After multiple levels of repeated calls to the depthwise separable convolution module, a 2D feature map with multiple channels is output, where each channel carries an enhanced representation of a specific type of geometric feature after abstraction and refinement.

[0042] Step S430: Perform global average pooling on the multi-channel feature map to obtain a one-dimensional feature vector.

[0043] Among them, the different dimensions of the one-dimensional feature vector encode three types of geometric features: the relative position of key points, the distance between key points, and the angle between vectors, respectively, forming a single-frame feature vector with geometric semantics.

[0044] As an example, global average pooling performs a global spatial average operation on each channel of a multi-channel feature map, that is, it calculates the arithmetic mean of the values ​​at all spatial locations within each channel, thereby compressing the two-dimensional channel feature map into a one-dimensional channel feature vector, thus eliminating spatial size redundancy.

[0045] It should be noted that the dimension of this one-dimensional channel feature vector is equal to the number of channels in the multi-channel feature map; this one-dimensional channel feature vector is used as a single-frame feature vector; the first few dimensions of this single-frame feature vector encode geometric features such as the relative positions of key points, the middle few dimensions encode geometric features such as the distances between key points, and the last few dimensions encode geometric features such as the angles between vectors; this dimensional partitioning can be solidified during the training phase through geometric fidelity loss function constraints, ensuring that the single-frame feature vector prioritizes the retention of core geometric information that can be used for gesture pointing direction calculation during compression, rather than only serving the gesture action classification task.

[0046] This embodiment uses an end-to-end joint training method to enable the single-frame geometric feature extraction network, the temporal modeling network, and the dual-task output layer to be collaboratively optimized, ensuring that different dimensions of the single-frame feature vector encode three types of geometric features: the relative position of key points, the distance between key points, and the vector angle.

[0047] Construct a multi-task training dataset: Collect action videos of 100 users performing gestures at different distances (0.5–5 m) and under different lighting conditions; store the original coordinates of the hand key points of 3 consecutive frames in each sample, and label each frame sample with gesture action type label as well as the true azimuth and pitch angle values; finally form a multi-task dataset containing 100,000 samples, whose data structure is "3 consecutive frames of geometric features + action type + dual-angle ground truth", which is different from the existing technology datasets that only label the action type of a single frame.

[0048] Training parameters are set as follows: Batch Size is 32, initial learning rate is 0.001, and the optimizer is Adam.

[0049] The loss function can be defined as: ;in, The gesture classification loss is used to constrain the accuracy of gesture type recognition; The spatial pointing angle regression loss is used to constrain the accuracy of azimuth and pitch angle calculations. During training, the loss gradient is simultaneously backpropagated to the dual-task output layer, the temporal modeling network, and the single-frame geometric feature extraction network to uniformly constrain the parameter updates of each layer, so that the geometric features output by the network simultaneously meet the requirements of gesture action classification accuracy and spatial pointing angle calculation accuracy.

[0050] During training, a geometric fidelity loss constraint is introduced: an auxiliary geometric verification branch is added after the output of the 256-dimensional single-frame feature vector to supervise the update of network weights in reverse; this loss term is based on the 256-dimensional feature vector to reversely reconstruct the relative offset distance and joint angle of key points, and calculates the error between the reconstructed value and the original pre-calculated geometric parameters; during training, this geometric fidelity loss is minimized, which forces the network to prioritize the retention of relative position, offset distance, and spatial angle geometric features that can be used for spatial angle calculation during dimensionality reduction and compression, so as to avoid losing the core information for angle calculation due to dimensionality compression.

[0051] After training, the dimension of the single-frame feature vector is fixed to 256 dimensions, of which the first 80 dimensions encode geometric features such as the relative positions of key points, the middle 96 dimensions encode geometric features such as the distances between key points, and the last 80 dimensions encode geometric features such as the angles between vectors. This dimensional partitioning structure is forcibly formed by the geometric fidelity loss term during training, ensuring that downstream temporal modeling and dual-task outputs can directly extract the corresponding geometric semantics based on the dimensional position.

[0052] Step S500: The temporal input sequence is formed by combining a preset number of single-frame feature vectors, and the temporal input sequence is analyzed across frames through a temporal modeling network to output stable temporal fusion features.

[0053] The temporal input sequence is an ordered sequence composed of multiple consecutive single-frame feature vectors output from step S400 stacked in chronological order. The temporal modeling network is a stacked long short-term memory network, including a first-layer long short-term memory network (LSTM1) and a second-layer long short-term memory network (LSTM2), which form a serial cascaded structure. The temporal modeling network is connected to the upstream single-frame geometric feature extraction network and the downstream dual-task output layer through a low-latency data channel to ensure that the temporal features are transferred seamlessly between modules and that the geometric semantic partitioning is maintained.

[0054] In some implementations, such as Figure 4 As shown, step S500 specifically includes steps S510-S530: Step S510: Through the first layer of the Long Short-Term Memory network, the single-frame feature vector of each frame in the temporal input sequence is parsed frame by frame to generate the hidden state vector of the corresponding frame.

[0055] Among them, the different dimensions of the hidden state vector encode three types of geometric features: the relative position of key points, the distance between key points, and the angle between vectors.

[0056] In this step, the first-layer long short-term memory network receives the single-frame feature vector of each frame in the temporal input sequence as an independent input; different segments of the single-frame feature vector encode three types of geometric features: relative position of key points, distance between key points, and vector angle; the first-layer long short-term memory network performs gated loop calculation on each frame input, updates and outputs the hidden state vector corresponding to the current frame based on the single-frame feature vector of the current frame and its hidden state at the previous time step.

[0057] It should be noted that the hidden state vector is a one-dimensional vector with a fixed dimension (e.g., 256 dimensions). Its different segments maintain semantic consistency with the corresponding segments of the input single-frame feature vector. That is, the first few dimensions (e.g., 80 dimensions) encode geometric features such as the relative position of key points, the middle few dimensions (e.g., 96 dimensions) encode geometric features such as the distance between key points, and the last few dimensions (e.g., 80 dimensions) encode geometric features such as the angle between vectors. The hidden state vector not only carries the static representation of the three types of geometric features within a single frame, but also integrates the local dynamic information of the frame in the temporal context. Thus, it can provide a basic representation with temporal awareness capabilities for the second-layer long short-term memory network to perform cross-frame analysis.

[0058] Step S520: Through the second layer of the long short-term memory network, the hidden state vector sequence of the continuous frames output by the first layer of the long short-term memory network is longitudinally compared to calculate the numerical fluctuation amplitude of the same geometric features in each frame.

[0059] In this step, the second-layer Long Short-Term Memory network receives a sequence of hidden state vectors from multiple consecutive frames output in step S510. For the segment of the relative position of the encoded keypoint in the hidden state vector, all values ​​of the segment in multiple consecutive frames are extracted to form a one-dimensional numerical sequence. Similarly, all values ​​of the segment of the distance between encoded keypoints and the segment of the angle between encoded vectors in multiple consecutive frames are extracted to form corresponding one-dimensional numerical sequences. For the numerical sequence corresponding to each type of geometric feature, the numerical fluctuation amplitude between consecutive frames is calculated. The numerical fluctuation amplitude is the maximum absolute value of the difference between the corresponding position values ​​of two adjacent frames in the sequence, or the standard deviation of the sequence. This numerical fluctuation amplitude is used to characterize the stability of the corresponding geometric features over time during the hand's pointing action.

[0060] Specifically, when the fluctuation amplitude of the geometric features of the relative position of key points exceeds a preset first threshold, or the fluctuation amplitude of the geometric features of the distance between key points exceeds a preset second threshold, or the fluctuation amplitude of the geometric features of the vector angle exceeds a preset third threshold, it indicates that the current frame is affected by instantaneous hand shaking or image noise interference, and the confidence of such geometric features in the corresponding latent state vector decreases, which needs to be suppressed in subsequent fusion processing.

[0061] Step S530: Based on the numerical fluctuation amplitude of the same geometric features in each frame, combined with a preset fluctuation threshold, an adaptive weight is applied to the part of the hidden state vector in each frame that represents the same geometric features, and the weighted hidden state vectors of each frame are fused to output temporal fusion features.

[0062] For example, for each type of geometric feature, the numerical fluctuation amplitude of that geometric feature in multiple consecutive frames calculated in step S520 is compared with the corresponding preset fluctuation threshold to determine the corresponding weight. For instance, when the numerical fluctuation amplitude of the relative position of key points in a frame is greater than the preset first threshold, the weight of that frame in the relative position segment of key points is determined to be a positive number less than 1, and this weight decreases monotonically as the fluctuation amplitude increases. Similarly, when the numerical fluctuation amplitude of the distance between key points in a frame is greater than the preset second threshold, the weight of that frame in the distance between key points is determined to be a positive number less than 1. When the numerical fluctuation amplitude of the vector angle in a frame is greater than the preset third threshold, the weight of that frame in the vector angle segment is determined to be a positive number less than 1. For frames that do not exceed the corresponding preset fluctuation threshold, their weight in the corresponding segment is set to 1. It should be noted that the first, second, and third thresholds can all be set according to actual needs, i.e., the scene. In addition, the weight settings can also be set according to actual needs, and no specific limitation is made here.

[0063] Subsequently, weighted fusion is performed on each segment of the latent state vector across multiple consecutive frames: the values ​​of the relative positions of keypoints in each frame are multiplied by their corresponding weights and summed to obtain the weighted fusion result for that segment; the values ​​of the distances between keypoints in each frame are multiplied by their corresponding weights and summed to obtain the weighted fusion result for that segment; the values ​​of the vector angles in each frame are multiplied by their corresponding weights and summed to obtain the weighted fusion result for that segment; finally, the weighted fusion results of the above three types of segments are concatenated in the same order to form a one-dimensional vector of fixed dimension (its dimension can also be 256 dimensions); this one-dimensional vector is the temporal fusion feature; the temporal fusion feature fully preserves the semantic partitioning structure of the three geometric features of relative keypoint positions, distances between keypoints, and vector angles, and its values ​​are weighted across frames to suppress abrupt changes caused by instantaneous hand tremors or image noise, exhibiting stability superior to single-frame features, and can be synchronously used by downstream gesture action classification branches and spatial pointing direction calculation branches.

[0064] Step S600: Based on temporal fusion features, perform gesture action classification and spatial pointing direction calculation simultaneously to obtain gesture pointing results.

[0065] The gesture pointing result includes the gesture action type and the gesture pointing direction. The gesture action type is used to represent the control intention category corresponding to the gesture currently performed by the user. The gesture pointing direction is used to represent the orientation and pitch of the user's finger in three-dimensional space. In some implementations, such as Figure 5 As shown, step S600 specifically includes steps S610-S640: Step S610: The temporal fusion features are synchronously input into the gesture action classification branch and the spatial pointing direction calculation branch that are set in parallel.

[0066] It can be understood that the gesture action classification branch and the spatial pointing direction calculation branch are two computational paths that are structurally independent, functionally separate, but have the same data source; the two run in parallel in physical implementation and have no sequential dependency; the temporal fusion feature is used as the only input, is completely copied and sent to both branches at the same time, without any truncation, filtering or resampling. The gesture classification branch extracts only a subset of geometric features related to hand posture stability and morphological recognition from the temporal fusion features, and generates gesture types after performing nonlinear mapping. The spatial pointing direction calculation branch reuses three types of geometric features encoded in the temporal fusion features: relative position of key points, distance between key points, and vector angle. Combined with the device imaging system parameters, it calculates the pointing direction of the gesture in three-dimensional space. Since the two branches share the same upstream temporal modeling output, the generated gesture types and gesture pointing directions naturally have frame-level alignment and temporal coupling characteristics, avoiding instruction misalignment or intent drift caused by branch asynchrony.

[0067] Step S620: In the gesture action classification branch, extract the geometric feature subset representing the stability of hand posture from the temporal fusion features, and output the gesture action type and corresponding confidence after nonlinear mapping.

[0068] The geometric feature subset includes keypoint distance geometric features and vector angle geometric features encoded in the temporal fusion features. Keypoint distance geometric features reflect the spatial span changes between finger joints, while vector angle geometric features reflect the relative orientation relationship between finger pointing vectors. Together, they characterize the stable geometric configuration of the hand when performing different control intentions.

[0069] In this step, a subset of geometric features is input to a classification mapping unit (such as a Softmax multi-classifier). This classification mapping unit performs a non-linear mapping operation, mapping the input features to an element in a preset set of gesture action types. The set of gesture action types includes at least three mutually exclusive categories (such as single-finger pointing, fist clenching, two-finger switching, etc.), and each category corresponds to a gesture form with clear control semantics. The non-linear mapping process is completed based on the parameters learned during the training phase to ensure that the output result has the ability to discriminate changes in the input features.

[0070] Since the temporal fusion features themselves have suppressed instantaneous jitter through the cross-frame weighted fusion processing in step S500, the gesture action type naturally possesses temporal stability. Its output does not depend on single-frame judgment, but is based on the consistency of geometric features in multiple consecutive frames, which can effectively avoid misidentification of gesture categories caused by slight hand tremors or image noise.

[0071] In step S630, in the spatial pointing direction calculation branch, the three types of geometric features encoded in the temporal fusion features—relative position of key points, distance between key points, and vector angle—are reused and combined with the device imaging system parameters to calculate the hand pointing direction in three-dimensional space.

[0072] The equipment imaging system parameters include the camera intrinsic parameter matrix, which is pre-calibrated and used to establish a linear mapping relationship between the pixel coordinates of the image plane and the three-dimensional spatial coordinates in the camera coordinate system; the relative position geometric features of key points provide the relative coordinate offset between the reference key points and the target key points; the distance geometric features between key points and the vector angle geometric features are used to verify the spatial rationality of the selected target key points to ensure that the pointing axis is stable and reliable. In this step, firstly, based on the pixel coordinates of the normalized reference keypoints... pixel coordinates of target key points (such as the fingertip key point) Calculate the relative offset between the two pixels in the image plane to obtain the two-dimensional pixel pointing vector: .

[0073] Then, using the camera intrinsic parameter matrix, the two-dimensional pixel pointers are directed to vectors. Converted to a 3D spatial pointing vector in camera coordinate system This transformation process can be performed using matrix multiplication in homogeneous coordinates, resulting in a vector. It can represent the direction of a spatial ray extending from the camera's optical center along the direction the user's finger is pointing.

[0074] Furthermore, based on this three-dimensional spatial pointing vector The direction of the gesture in three-dimensional space is calculated; this direction can be determined by the pitch angle and azimuth angle. The pitch angle... It can be a vector Angle with the XY horizontal plane; azimuth angle For vectors The angle between the projection direction in the XY horizontal plane and the positive X-axis direction.

[0075] When calculating the pitch angle, first use the formula... Calculate the modulus of the projection in the horizontal plane, and then based on the formula... Calculate pitch angle ;in, The value range is [-90°, 90°]; when At 0 o'clock, A positive value indicates that the direction is upward; when At 0 o'clock, A negative value indicates that the direction is tilted downwards.

[0076] Additionally, azimuth angle It can be obtained through the four-quadrant arctangent function Calculations are performed to ensure that the results cover the entire circumference and are correct in the quadrants. ;in, The value range is [0°, ), with the positive X-axis as the zero-degree starting point, and measured counterclockwise to represent the absolute directional position of the gesture in the horizontal plane.

[0077] Step S640: Perform time-attenuation weighted fusion processing on the gesture pointing direction calculated from multiple consecutive frames, and output the target gesture pointing direction.

[0078] In this embodiment, the pitch angle components calculated from the three most recent consecutive frames can be used. , , With azimuth components , , Time decay weighted fusion processing is performed separately, where the subscript t represents the current frame, t-1 represents the previous frame, and t-2 represents the frame before that. It should be noted that the more recent the frame, the higher the weight is assigned, which can enhance the latest pose information and suppress historical noise. The weighted fusion of pitch angle components can be expressed as: The weighted fusion of azimuth components can be expressed as: .

[0079] It should be noted that this processing can be performed at the end of the spatial pointing direction solution branch, thus forming a second smoothing operation after the cross-frame temporal modeling in step S500, forming a dual temporal constraint mechanism of "LSTM pre-filtering and back-end weighted fusion", which can suppress single-frame recognition jitter and instantaneous false detection, so that the output target gesture pointing direction has high stability and high repeatability accuracy.

[0080] Step S650: When the confidence level of the gesture action type recognition is higher than the first threshold and the change amplitude of the gesture pointing direction output in multiple consecutive frames is lower than the second threshold, the gesture pointing result is obtained.

[0081] It is understood that, in this embodiment, the final gesture pointing result needs to be satisfied as follows: First stability condition: The recognition result of the gesture action type remains consistent across multiple consecutive frames, and the value of its corresponding discrimination criterion is greater than the preset first threshold. The second stability condition is that the variation amplitudes of the pitch and azimuth components calculated over multiple consecutive frames are both lower than a preset second threshold. Specifically, the variation amplitude of the pitch component is defined as the maximum absolute value of the difference between the pitch components of any two adjacent frames in those consecutive frames; the variation amplitude of the azimuth component is defined as the maximum absolute value of the difference between the azimuth components of any two adjacent frames in those consecutive frames. The larger of the two is used as the final determination criterion for the second stability condition. The system confirms the generation of a valid gesture pointing result only when both the first and second stability conditions are simultaneously satisfied.

[0082] Based on the above gesture pointing recognition method, the mobile device control method of this application embodiment will be described below.

[0083] Figure 6 A schematic flowchart of a mobile device control method according to an embodiment of this application is shown. Exemplarily, the mobile device control method includes steps S10-S30: Step S10: Based on the gesture pointing direction obtained by the gesture pointing recognition method, the current pose of the mobile device in the world coordinate system established by itself, and the effective action distance measured along the gesture pointing direction, determine the spatial coordinates of the target point in the world coordinate system.

[0084] In this step, the gesture pointing direction is given by the target gesture pointing direction output in step S600, including the azimuth and pitch angles; the current pose of the mobile device is the real-time coordinates of the device in the world coordinate system. ,in The X and Y axes represent the current position of the device; the effective range k is the straight-line distance between the device and the passable ground plane, measured along the ray pointing towards the gesture; the spatial coordinates (x, y) of the target point in the world coordinate system are given by the formula... The parameter k is determined as follows: the device synchronously emits a laser radar or ultrasonic ranging signal along the direction of the gesture pointing ray; obstacles and passable areas on the ground are detected sequentially from near to far along the ray; the ranging return values ​​of suspended objects and obstacles are filtered; the straight-line distance of the first intersection point between the ray and the passable grid plane on the ground is taken as k; in indoor SLAM scenarios, this k value is corrected by matching a pre-built grid map; in outdoor GPS scenarios, this k value is directly determined using the raw echo data from the ranging sensor.

[0085] In some implementations, such as Figure 7 As shown, determining the spatial coordinates of the target point in the world coordinate system includes steps S11-S14: Step S11: Construct a ray with the current pose of the mobile device as the starting point and the direction of the gesture as the direction.

[0086] This step uses the device's current pose coordinates. As the starting point of the ray, with azimuth angle and pitch angle Define the ray direction and construct a gesture-pointing spatial ray; this ray equation is used for subsequent spatial relationship calculations with environmental maps or geographic planes.

[0087] Step S12: In response to the mobile device being in a structured environment, based on a pre-built environment map, determine the first intersection point between the ray and the plane of the passable area, and use the coordinates of the first intersection point as the coordinates of the initial target point.

[0088] Among them, the structured environment refers to an indoor scene in which a raster map has been pre-built using Simultaneous Localization and Mapping (SLAM) technology; the environment map is a two-dimensional raster map generated based on the world coordinate system, and each raster cell has attribute labels, including but not limited to "accessible", "obstacles", "suspended" etc.

[0089] The device loads and updates the map in real time during operation. The process of determining the first intersection point is as follows: the gesture pointing ray constructed in step S11 is matched point by point with the SLAM grid map. The attributes of each grid are detected sequentially from near to far along the ray extension path. Obstacle grids and suspended grids are filtered out. The first grid intersection point with the attribute "passable" in the ray path is taken, and its center coordinates in the world coordinate system are used as the initial target point coordinates. .

[0090] Furthermore, after determining the coordinates of the first intersection point in step S12, this embodiment can further perform SLAM map error correction: First, extract the grid attributes corresponding to the coordinates and determine whether it is an obstacle, edge blind spot, or impassable area; if the determination result is yes, then use the direction of the gesture pointing ray as a constraint, finely adjust the coordinate point along the ray direction, and match the nearest valid passable grid; at the same time, combine the average historical positioning error of the device to perform deviation compensation for the coordinates; after the above two corrections, the target point positioning error is finally controlled within 10 cm.

[0091] Step S13: In response to the mobile device being in an unstructured environment, based on the acquired device location information of the mobile device, the gesture pointing direction is converted into a geographic plane offset and superimposed on the device location information to generate the geographic coordinates of the target point.

[0092] In this step, unstructured environment refers to open environment without a fixed SLAM grid map, such as an open outdoor field; device location information is the initial latitude and longitude coordinates obtained by the device through the GPS / BeiDou positioning system; the azimuth and elevation angles are converted into geospatial offsets, and based on the Earth plane approximation algorithm and the ground distance conversion factor corresponding to the current latitude, the angular offset is converted into a latitude and longitude offset difference; this latitude and longitude offset difference is superimposed on the device's initial latitude and longitude coordinates to generate the geographic coordinates of the target point.

[0093] Step S14: Convert the geographic coordinates of the target point into spatial coordinates in the world coordinate system.

[0094] In this step, the geographic coordinates of the target point generated in step S13 are converted into Cartesian coordinates in the world coordinate system of the device through a coordinate transformation method. This transformation ensures that the target point coordinates and the device pose and path planning modules are calculated in the same coordinate system, thus ensuring control accuracy and system consistency.

[0095] Step S20: Generate a navigation path based on the spatial coordinates of the target point, the current pose of the mobile device, and environmental obstacle information. In this step, the input data includes: the spatial coordinates of the target point determined by steps S10–S14, the real-time pose of the device in the world coordinate system, and dynamic obstacle information (including obstacle position, size, and motion state) perceived and updated in real time through LiDAR, ultrasonic sensors, or visual SLAM; the navigation path is a continuous executable trajectory that starts from the device's current position, avoids all obstacles, and finally arrives at the target point; the path generation is based on the A* algorithm, and three key optimizations are performed for the gesture control scenario: First, add gesture priority weights.

[0096] When the gesture action type identified in step S600 is an emergency pointing action, the heuristic function weight coefficient of the A* algorithm can be set to 1.5. This weight increase makes the A* algorithm more inclined to choose the path with the shortest straight distance during the path search process, so as to shorten the response time. It is suitable for high-time-efficiency task scenarios such as rapid avoidance of dangerous areas and emergency braking of equipment.

[0097] Second, dynamic replanning.

[0098] During the execution of the current navigation path, the device continuously monitors the output of the gesture pointing recognition module. When a new gesture command is received and the command is confirmed as a valid gesture pointing result by step S650, the system immediately interrupts the ongoing path planning process, clears the original path queue, and uses the target point corresponding to the new command as the endpoint to re-invoke the A* algorithm to generate a brand new navigation path. This mechanism ensures that the device response always follows the user's latest intention and avoids path deviation or task delay caused by command delay.

[0099] Third, path smoothing processing.

[0100] The original output of the A* algorithm is a polyline path composed of a series of discrete grid nodes. Abrupt changes in its direction can easily cause jerking in mobile devices. To address this, the system performs B-spline curve fitting on the polyline path output by A*: using each node of the polyline path as control points, a cubic B-spline curve is constructed to generate a continuous, differentiable, and bounded curvature smooth path. This smooth path is then sent to the execution control module as the final navigation command, significantly reducing the steering jitter of the sweeping robot, the attitude oscillation of the aircraft, and the impact on the wheel assembly of the inspection robot, thereby improving motion stability and user experience.

[0101] Step S30: Control the mobile device to move along the navigation path to the target point and perform the task corresponding to the gesture action type.

[0102] In this step, the navigation path generated in step S20 is input to the motion controller of the mobile device, driving the device to move along the path from its current position to the target point. During the movement, the device collects its own pose and environmental information in real time through the encoder, inertial measurement unit (IMU), and lidar, and continuously feeds this status data back to the Simultaneous Localization and Mapping (SLAM) module to achieve dynamic updates of the device's pose and ensure path tracking accuracy, forming a closed loop of position awareness. When the device moves to the target point and meets the positioning conditions, it triggers the execution of a task matching the gesture type. The positioning conditions are that the spatial distance between the device's current position and the target point falls within a preset tolerance range, and the device's orientation is basically consistent with the direction of the gesture. The content of the task can be determined by the gesture type. For example, if the gesture type is a single finger pointing, a cleaning task, an inspection task, or a drone photography task is started; if the gesture type is a fist clenching, an emergency stop is executed; if the gesture type is a two-finger switching, the device enters a standby state after completing the current task, ready to respond to the next gesture command, etc.

[0103] This application improves accuracy under different lighting and environments by using hand keypoint coordinate sequences as input, eliminating reliance on image texture and background. Furthermore, it employs benchmark keypoint normalization and collaborative modeling of three types of geometric features, allowing the network to focus on the inherent spatial orientation of the hand. A single-frame feature extraction network combined with an inverse residual structure ensures that different segments in the output vector stably encode relative position, distance, and angle information. Dual-layer LSTM temporal modeling effectively suppresses jitter interference and maintains geometric semantic consistency through fluctuation perception and adaptive weighting. Simultaneously executed gesture classification and angle calculation share the same temporal fusion features, supplemented by a dual judgment mechanism of confidence and angle change to ensure command reliability. Spatial mapping uniformly anchors the world coordinate system established during device initialization, adaptively selecting SLAM raster matching or geographic offset transformation based on environment type to achieve highly consistent target localization in both structured and unstructured scenes. Path planning incorporates gesture semantic guidance, dynamic replanning, and trajectory smoothing, balancing response speed and motion quality. The entire solution forms a closed-loop collaboration from pixel to execution, systematically solving problems such as inaccurate pointing, slow response, and weak scene adaptation in traditional gesture control.

[0104] This application also provides a mobile device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the gesture pointing recognition method and / or the mobile device control method described above.

[0105] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0106] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0107] This application also provides a computer-readable storage medium for storing the computer program used in the aforementioned mobile device. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0109] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0110] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A gesture pointing recognition method, characterized in that, include: A predetermined number of consecutive video frames are acquired in time sequence to obtain a sequence of images of the target hand; Based on the target hand image sequence, the initial coordinate information of each preset key point of the target hand is extracted; The geometric features representing the spatial pointing relationship of the hand are calculated based on the initial coordinate information of each of the preset key points; The geometric features of each frame are processed by a single-frame geometric feature extraction network to output a single-frame feature vector with geometric semantics. The preset number of consecutive single-frame feature vectors are combined to form a temporal input sequence, and the temporal input sequence is analyzed across frames through a temporal modeling network to output stable temporal fusion features. Based on the aforementioned temporal fusion features, gesture action classification and spatial pointing direction calculation are performed simultaneously to obtain gesture pointing results; wherein, the gesture pointing results include gesture action type and gesture pointing direction.

2. The gesture pointing recognition method according to claim 1, characterized in that, The calculation of geometric features representing the spatial pointing relationship of the hand based on the initial coordinate information of each of the preset key points includes: Using the reference key point as the origin, the initial coordinate information of the other preset key points is subjected to relative coordinate normalization processing to obtain the corresponding normalized coordinate information; Based on the normalized coordinate information, calculate the geometric features that characterize the spatial pointing relationship of the hand; the geometric features include the relative distance between each pair of key points, the angle between the pointing vectors of each finger, and the bending state of the single finger joint chain.

3. The gesture pointing recognition method according to claim 1, characterized in that, The single-frame geometric feature extraction network includes a depthwise separable convolutional module with an inverse residual structure; the different dimensions of the single-frame feature vector correspond to the relative positions of key points, the distance between key points, and the vector angle, respectively. The single-frame geometric feature extraction network processes the geometric features of each frame to output a single-frame feature vector with geometric semantics, including: The geometric features of the single frame are organized and reshaped into a two-dimensional feature map; The depthwise separable convolution module is used to perform multi-level feature extraction on the two-dimensional feature map to obtain a multi-channel feature map. During the feature extraction process, the channel dimension is first expanded and then compressed, and the correlation between various geometric features in the single-frame feature vector is independently modeled in the expanded high-dimensional space. Global average pooling is performed on the multi-channel feature map to obtain a one-dimensional feature vector; wherein, the different dimensions of the one-dimensional feature vector encode three types of geometric features: the relative position of the key points, the distance between the key points, and the angle between the vectors, respectively, thus forming the single-frame feature vector with geometric semantics.

4. The gesture pointing recognition method according to claim 1, characterized in that, The time-series modeling network is a stacked long short-term memory network, including a first layer of long short-term memory network and a second layer of long short-term memory network; The step of performing cross-frame analysis on the temporal input sequence using a temporal modeling network to output stable temporal fusion features includes: The first-layer long short-term memory network is used to parse the single-frame feature vector of each frame in the temporal input sequence frame by frame to generate the hidden state vector of the corresponding frame; wherein, the different dimensions of the hidden state vector encode three types of geometric features: the relative position of key points, the distance between key points, and the vector angle. By using the second layer of the long short-term memory network, the hidden state vector sequence of consecutive frames output by the first layer of the long short-term memory network is longitudinally compared to calculate the numerical fluctuation amplitude of the same geometric features in each frame. Based on the numerical fluctuation amplitude of similar geometric features in each frame, combined with a preset fluctuation threshold, adaptive weights are applied to the part of the latent state vector representing similar geometric features in each frame, and the weighted latent state vectors of each frame are fused to output the temporal fusion feature.

5. The gesture pointing recognition method according to claim 1, characterized in that, Based on the temporal fusion features, the gesture action classification and spatial pointing direction calculation are performed simultaneously to obtain the gesture pointing result, including: The temporal fusion features are synchronously input into the gesture action classification branch and the spatial pointing direction calculation branch, which are set in parallel. In the gesture classification branch, a subset of geometric features representing hand posture stability is extracted from the temporal fusion features, and the gesture type and corresponding confidence level are output after nonlinear mapping. In the spatial pointing direction calculation branch, the three types of geometric features encoded in the temporal fusion features—relative position of key points, distance between key points, and vector angle—are reused, and combined with the device imaging system parameters, the hand gesture pointing direction in three-dimensional space is calculated. Perform time-attenuation weighted fusion processing on the gesture pointing direction calculated from multiple consecutive frames, and output the target gesture pointing direction; The gesture pointing result is obtained when the confidence level of the gesture action type is higher than the first threshold and the change amplitude of the gesture pointing direction in multiple consecutive frames is lower than the second threshold.

6. A mobile device control method, characterized in that, include: Based on the gesture pointing direction obtained from any one of claims 1 to 5, the current pose of the mobile device in the world coordinate system it has initialized, and the effective action distance measured along the gesture pointing direction, the spatial coordinates of the target point in the world coordinate system are determined. A navigation path is generated based on the spatial coordinates of the target point, the current pose of the mobile device, and environmental obstacle information; Control the mobile device to move along the navigation path to the target point and perform a task corresponding to the gesture type.

7. The mobile device control method according to claim 6, characterized in that, Determining the spatial coordinates of the target point in the world coordinate system includes: Construct a ray with the current pose of the mobile device as the starting point and the direction of the gesture as the direction; In response to the mobile device being in a structured environment, based on a pre-built environment map, the first intersection point between the ray and the plane of the passable area is determined, and the coordinates of the first intersection point are used as the coordinates of the initial target point; In response to the mobile device being in an unstructured environment, based on the acquired device location information of the mobile device, the gesture pointing direction is converted into a geographic plane offset and superimposed on the device location information to generate the geographic coordinates of the target point; The geographic coordinates of the target point are converted into spatial coordinates in the world coordinate system.

8. A mobile device, characterized in that, The mobile device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the gesture pointing recognition method according to any one of claims 1-5 and / or the mobile device control method according to any one of claims 6-7.

9. The mobile device according to claim 8, characterized in that, The mobile device includes at least one of a robotic vacuum cleaner, a drone, and an industrial inspection robot.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed on a processor, implements the gesture pointing recognition method according to any one of claims 1-5 and / or the mobile device control method according to any one of claims 6-7.

Images