Air gesture interaction method, device and equipment, medium and vehicle

Abstract

The invention relates to an air gesture interaction method, device and equipment, a medium and a vehicle. The air gesture interaction method comprises the following steps: acquiring a preset hot area corresponding to each control group; when it is detected that a cursor, mapped to the control display interface, of the first air gesture of the user moves into the target preset hot area, displaying a target control group and a direction guiding gesture corresponding to the target preset hot area; and when it is detected that a second air gesture of the user meets a gesture condition corresponding to the guide gesture, moving the target control group according to a moving track of the second air gesture. According to the embodiment of the invention, independent movement control can be carried out on the target control group in the plurality of control groups, so that the use experience of a user is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of display devices, and in particular to an air gesture interaction method, device, equipment, medium and vehicle. Background Art

[0002] At present, with the development of technology, users can interact with terminal devices through air gestures. For example, the correspondence between air gestures and functions can be set. After that, the user makes an air gesture within the perception range of the terminal device, and the terminal device recognizes the air gesture, and then determines the function corresponding to the air gesture and adjusts it.

[0003] In the related art, a user can use air gestures to scroll the controls displayed on a terminal device. However, if multiple independent control groups are displayed on the terminal device, when performing a scrolling operation, the terminal device can only scroll the multiple control groups simultaneously according to the user's air gestures, and cannot scroll one of the multiple control groups alone, resulting in a poor user experience. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides an air gesture interaction method, device, equipment, medium and vehicle.

[0005] In a first aspect, the present disclosure provides an air gesture interaction method, comprising:

[0006] Displaying multiple control groups in the control display interface, and displaying the preset hot zone corresponding to each control group;

[0007] When it is detected that the cursor mapped by the user's first air gesture on the control display interface moves into the target preset hot zone, the target control group and direction guidance gesture corresponding to the target preset hot zone are displayed;

[0008] When it is detected that the user's second air gesture satisfies the gesture condition corresponding to the direction guiding gesture, the target control group is moved according to the moving track of the second air gesture.

[0009] In a second aspect, the present disclosure provides an air gesture interaction device, comprising:

[0010] A first display module, used to display multiple control groups in a control display interface, and to display a preset hot zone corresponding to each control group;

[0011] The second display module is used to display the target control group and direction guidance gesture corresponding to the target preset hot zone when detecting that the cursor mapped by the user's first air gesture on the control display interface moves into the target preset hot zone;

[0012] The control moving module is used to move the target control group according to the movement trajectory of the second air gesture when it is detected that the user's second air gesture meets the gesture condition corresponding to the direction guidance gesture.

[0013] In a third aspect, the present disclosure provides an air gesture interaction device, including:

[0014] processor;

[0015] A memory for storing executable instructions;

[0016] Among them, the processor is used to read executable instructions from the memory and execute the executable instructions to implement the air gesture interaction method of the first aspect.

[0017] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements the air gesture interaction method of the first aspect.

[0018] In a fifth aspect, the present disclosure provides a vehicle, comprising the above-mentioned air gesture interaction device.

[0019] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0020] The interactive method, apparatus, device, medium and vehicle for air gestures of the disclosed embodiments can obtain the preset hot zone corresponding to each control group, and then display the target control group and direction guidance gesture corresponding to the target preset hot zone when it is detected that the cursor mapped by the user's first air gesture on the control display interface moves into the target preset hot zone, and then move the target control group according to the moving trajectory of the second air gesture when it is detected that the user's second air gesture satisfies the gesture condition corresponding to the direction guidance gesture. Thus, when the cursor corresponding to the user's first air gesture moves into the target preset hot zone, the target control group and direction guidance gesture corresponding to the target preset hot zone are displayed, and then when the user's second air gesture satisfies the gesture condition corresponding to the direction guidance gesture, the target control group is moved according to the moving trajectory of the second air gesture, so that the target control groups among the multiple control groups can be individually moved and controlled, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0022] Figure 1A schematic diagram of a flow chart of an air gesture interaction method provided in an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of a control display interface provided by an embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of another control display interface provided by an embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of hand skeleton points provided by an embodiment of the present disclosure;

[0026] Figure 5 A flowchart of a mapping process provided by an embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of another control display interface provided by an embodiment of the present disclosure;

[0028] Figure 7 A schematic diagram of the structure of an air gesture interaction device provided by an embodiment of the present disclosure;

[0029] Figure 8 A schematic diagram of the structure of an air gesture interaction device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0031] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0033] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0035] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0036] In order to solve the above problems, the embodiments of the present disclosure provide a method, device, equipment, medium and vehicle for air gesture interaction. Figures 1 to 6 The air gesture interaction method provided by the embodiment of the present disclosure is described in detail.

[0037] Figure 1 A flow chart of an air gesture interaction method provided by an embodiment of the present disclosure is shown.

[0038] In the disclosed embodiment, the air gesture interaction method may be executed by a processor of an electronic device having an air gesture interaction function. Specifically, the electronic device may include but is not limited to mobile terminals such as mobile phones, vehicle-mounted devices, vehicle controllers, tablet computers, wearable devices, etc.

[0039] like Figure 1 As shown, the air gesture interaction method may include the following steps.

[0040] S110, obtaining a preset hot zone corresponding to each control group.

[0041] In an embodiment of the present disclosure, when a user uses an electronic device, the electronic device can obtain a preset hot zone corresponding to each control group and display it in a control display interface.

[0042] Optionally, the control display interface may be an interface for displaying multiple control groups.

[0043] Optionally, the control group may be a group including multiple controls. For example, the control group may be composed of a row of controls or a column of controls, and multiple control groups may be composed of multiple rows of controls or multiple columns of controls, which is not limited here.

[0044] Alternatively, a control may refer to an object that encapsulates data and methods, and with which a user may interact to input or manipulate data.

[0045] Optionally, the preset hot zone may be an interactive portion in a predefined control display interface, which may trigger the electronic device to perform a corresponding operation after interaction.

[0046] Specifically, when a user uses an electronic device, the electronic device can obtain a preset hot zone corresponding to each control group, and display multiple control groups in a control display interface, and display the preset hot zone corresponding to each control group.

[0047] Figure 2 An interface schematic diagram of a control display interface provided by an embodiment of the present disclosure is shown.

[0048] like Figure 2 As shown, in the control display interface 201 of the electronic device, two control groups can be displayed, that is, each control group can include 4 controls in the same row, and each control group can correspond to its own preset hot zone 202, wherein the preset hot zone 202 can be based on the movable direction of the control, for example, if the control can be moved to the left, the preset hot zone 202 can be located at the rightmost side of the control display interface 201; if the control can be moved to the right, the preset hot zone 202 can be located at the leftmost side of the control display interface 201; if the control group includes multiple controls in the same column, the control can be moved downward, the preset hot zone 202 can be located at the top of the control display interface 201; if the control can be moved upward, the preset hot zone 202 can be located at the bottom of the control display interface 201, etc., which is not limited here.

[0049] S120. When it is detected that the cursor mapped by the user's first air gesture on the control display interface moves into a target preset hot zone, a target control group and a direction guidance gesture corresponding to the target preset hot zone are displayed.

[0050] In an embodiment of the present disclosure, after displaying a preset hot zone, the electronic device can display a target control group and direction guidance gesture corresponding to the target preset hot zone when it detects that the cursor mapped by the user's first air gesture on the control display interface moves into the target preset hot zone.

[0051] Optionally, the first air gesture may be a gesture used by a user to interact with an electronic device. For example, the first air gesture may be a waving gesture, a sliding gesture, a hovering gesture, etc., which is not limited here.

[0052] Optionally, the cursor may be a mark indicating the current operation position in the control display interface.

[0053] Optionally, the target preset hot zone may be a preset hot zone selected by a cursor among multiple preset hot zones.

[0054] Optionally, the target control group may be a control group corresponding to a target preset hot zone.

[0055] Optionally, the direction guiding gesture may be used to guide the moving direction of the user's gesture.

[0056] Specifically, the user can operate the electronic device through a first air gesture, and the electronic device can detect the position of the cursor mapped by the first air gesture on the control display interface. For example, the electronic device can obtain the user's first air gesture through an image acquisition device such as a camera, and determine the position of the cursor mapped by the first air gesture on the control display interface. After the electronic device detects that the position of the cursor moves into the target preset hot zone, it can display the target control group and direction guidance gesture corresponding to the target preset hot zone.

[0057] Continue to see Figure 2 A cursor 203 is displayed in the control display interface 201. The cursor 203 can be the position of the user's first air gesture mapped on the control display interface, and the cursor 203 can move with the user's first air gesture.

[0058] Figure 3 An interface schematic diagram of another control display interface provided by an embodiment of the present disclosure is shown.

[0059] like Figure 3 As shown, when the user performs a first air gesture on the electronic device, the electronic device can detect the position of a cursor 302 corresponding to the first air gesture in the control display interface 301, and when it is detected that the cursor 302 moves into a target preset hot zone 303, the electronic device can display multiple controls included in a target control group 304 corresponding to the target preset hot zone 303 in the control display interface 301, and display a direction guidance gesture 305, such as a direction guidance gesture 305 that can be a left guidance gesture.

[0060] S130. When it is detected that the user's second air gesture satisfies the gesture condition corresponding to the direction guiding gesture, move the target control group according to the moving trajectory of the second air gesture.

[0061] In an embodiment of the present disclosure, after displaying the target control group and the direction guidance gesture, the electronic device can move the target control group according to the moving trajectory of the second air gesture if it detects that the user's second air gesture meets the gesture condition corresponding to the direction guidance gesture.

[0062] Optionally, the second air gesture may be similar to the first air gesture, which is not limited here.

[0063] Optionally, the gesture condition may be a pre-set condition.

[0064] Optionally, the movement trajectory may be a trajectory formed by the movement of the second air gesture.

[0065] Specifically, after displaying the target control group and the direction guidance gesture, the electronic device can obtain the user's second air gesture through an image acquisition device such as a camera, and when it is detected that the second air gesture meets the gesture condition corresponding to the direction guidance gesture, the target control group is moved according to the movement trajectory of the second air gesture. The specific judgment method for meeting the gesture conditions is described in detail below.

[0066] Therefore, in the embodiment of the present disclosure, the preset hot zone corresponding to each control group can be obtained, and then when it is detected that the cursor mapped by the user's first air gesture on the control display interface moves into the target preset hot zone, the target control group and the direction guidance gesture corresponding to the target preset hot zone are displayed, and then when it is detected that the user's second air gesture satisfies the gesture condition corresponding to the direction guidance gesture, the target control group is moved according to the moving trajectory of the second air gesture. Therefore, when the cursor corresponding to the user's first air gesture moves into the target preset hot zone, the target control group and the direction guidance gesture corresponding to the target preset hot zone are displayed, and then when the user's second air gesture satisfies the gesture condition corresponding to the direction guidance gesture, the target control group is moved according to the moving trajectory of the second air gesture, so that the target control groups among the multiple control groups can be individually moved and controlled, thereby improving the user experience.

[0067] Optionally, before S120, the air gesture interaction method may also include: obtaining a first air gesture image of the user, and extracting the coordinates of the hand skeleton points in the first air gesture image; mapping the coordinates of the first target skeleton points in the hand skeleton points to obtain the coordinates of the cursor of the first air gesture on the control display interface.

[0068] In an embodiment of the present disclosure, the electronic device may acquire a first air gesture image of the user, and extract the coordinates of the hand skeleton points in the first air gesture image.

[0069] Optionally, the first air gesture image may be an image including a first air gesture of the user.

[0070] Optionally, the hand skeleton point may be a skeleton point corresponding to the hand in the user's first air gesture.

[0071] Specifically, the electronic device can obtain the user's first air gesture image through an image acquisition device such as a camera, and extract the coordinates of the hand skeleton points in the first air gesture image.

[0072] Figure 4 A schematic diagram of hand skeleton points provided by an embodiment of the present disclosure is shown.

[0073] like Figure 4As shown, the electronic device can obtain the user's first air gesture image and obtain the hand part 401 in the first air gesture image, then extract the hand skeleton point 402 corresponding to the hand part 401, and label each hand skeleton point to obtain the corresponding labeled hand skeleton point 403, and then take the wrist joint 0 as the origin to calculate the coordinates of each hand skeleton point in the space rectangular coordinate system and the coordinates in the plane rectangular coordinate system in the first air gesture image.

[0074] The method for extracting hand skeleton points may be any known and achievable method, which is not limited here. For example, taking the extraction method through model training as an example, it may specifically include:

[0075] 1.1. Data labeling: A color (RGB) camera collects the first air gesture image, or uses motion capture technology to output a partial hand image through a rendering engine. The image is classified into left and right hands according to the actual size, and the plane coordinates of 10 skeletal key points on the image are labeled and the wrist joint is used as the three-dimensional (3D) coordinate, that is, the origin of the spatial rectangular coordinate. Accurate data labeling in the depth direction is usually obtained through motion capture.

[0076] 1.2. Model design: A convolutional neural network (CNN) model is used, which includes multiple convolutional layers, pooling layers, and residual connections. A fully connected layer is used to regress 3D coordinates, or a Gaussian heat map is used to regress two-dimensional (2D) and 3D coordinates.

[0077] 1.3. Model training: Divide the data into training set, test set, and validation set, use the model to infer the training set, define the mean square error between the inference value and the label of the 2D and 3D coordinates as the loss, use gradient descent for training, update the network model weights, and continue to iterate until the loss reaches the optimal level to obtain the pre-trained model.

[0078] 2.1. Image acquisition: Deploy the camera to collect the first air gesture image in real time, and standardize the image data for pre-trained neural network inference.

[0079] 2.2. Model reasoning: The model reads standardized image data, obtains a set of feature maps through multiple convolutions, pooling layers, and residual connections, and outputs the left and right hand categories, the coordinates of the 10 key points in the plane rectangular coordinate system, and the coordinates in the space rectangular coordinate system with the wrist joint as the origin through the full connection of the neural network or Gaussian heat map regression.

[0080] Furthermore, after acquiring the coordinates of the hand skeleton points, the electronic device can map the coordinates of the first target skeleton point among the hand skeleton points to obtain the coordinates of the cursor of the first air gesture on the control display interface.

[0081] Optionally, the mapping process may be a process of mapping the first target bone point to a cursor.

[0082] Optionally, the first target bone point may be one of the plurality of hand bone points. Figure 4 The first target bone point can be bone point 2, bone point 3, etc., which is not limited here.

[0083] Specifically, after obtaining the coordinates of the hand bone points, the electronic device can select a first target bone point from multiple hand bone points and perform mapping processing on the first target bone point to obtain the coordinates of the cursor of the first target bone point on the control display interface, that is, the coordinates of the cursor of the first air gesture on the control display interface.

[0084] Figure 5 A flowchart of a mapping process provided by an embodiment of the present disclosure is shown.

[0085] like Figure 5 As shown, the electronic device can obtain the coordinates (x0, y0) of the first target skeleton point 502, such as skeleton point 2, in the first air gesture image 501, and map the first target skeleton point 502 to obtain the coordinates (x1, y1) of the cursor 504 of the first target skeleton point 502 on the control display interface 503.

[0086] The electronic device may perform mapping processing using a mapping processing formula. The mapping processing formula may be:

[0087] x1=x0 / w0*w1

[0088] y1=y0 / h0*h1

[0089] Among them, x1 can be the horizontal coordinate of the cursor on the control display interface, x0 can be the horizontal coordinate of the first target bone point in the first air gesture image, w0 can be the width of the first air gesture image, w1 can be the width of the control display interface, y1 can be the vertical coordinate of the cursor on the control display interface, y0 can be the vertical coordinate of the first target bone point in the first air gesture image, h0 can be the height of the first air gesture image, and h1 can be the height of the control display interface.

[0090] Therefore, in the embodiment of the present disclosure, the electronic device can accurately obtain the coordinates of the cursor corresponding to the first air gesture on the control display interface, so as to accurately determine whether the cursor has moved into the target preset hot zone, reduce the judgment error rate, and improve the user experience.

[0091] Optionally, S120 may specifically include: responding to the user's first air gesture, moving the corresponding cursor on the control display interface according to the moving trajectory of the first air gesture; when it is detected that the coordinates of the cursor are within the coordinate range of the target preset hot zone, determining that the cursor has moved into the target preset hot zone.

[0092] In an embodiment of the present disclosure, the electronic device may respond to a first air gesture of the user and move a corresponding cursor on the control display interface according to a movement trajectory of the first air gesture.

[0093] Specifically, the electronic device can recognize and respond to the user's first air gesture, determine the movement trajectory of the first air gesture, and move a corresponding cursor on the control display interface according to the movement trajectory.

[0094] Furthermore, after the electronic device moves the cursor according to the movement trajectory, it can determine that the cursor has moved into the target preset hot zone when it is detected that the coordinates of the cursor are within the coordinate range of the target preset hot zone.

[0095] Specifically, after moving the cursor according to the moving track, the electronic device can detect the coordinates of the cursor in real time, and when it is detected that the coordinates of the cursor are within the coordinate range of the target preset hot zone, it can be determined that the cursor has moved into the target preset hot zone.

[0096] Therefore, in the embodiment of the present disclosure, whether the cursor has entered the target preset hot zone can be determined by the coordinates of the cursor, so that the detection result is more accurate.

[0097] Optionally, S120 may also specifically include: highlighting each control in the target control group corresponding to the target preset hot zone, and expanding the initial area range of the target preset hot zone to the preset area range, the preset area range is the area range covering a control; displaying a direction guidance gesture, the direction guidance gesture is used to guide the user's gesture movement direction.

[0098] In an embodiment of the present disclosure, after detecting that the cursor moves into a target preset hot zone, the electronic device can highlight, such as by performing operations such as zooming in and shaking, each control in the target control group corresponding to the target preset hot zone, and expand the initial area range of the target preset hot zone to the preset area range.

[0099] Optionally, the initial area range may represent the initial range size of the target preset thermal zone.

[0100] Optionally, the preset area range may represent a preset range size of the target preset hot zone, wherein the preset area range may be an area range covering a control.

[0101] Specifically, after detecting that the cursor moves into the target preset hot zone, the electronic device can highlight each control in the target control group corresponding to the target preset hot zone, and expand the initial area range of the target preset hot zone to the preset area range.

[0102] Continue to see Figure 3 After detecting that the cursor 302 moves into the target preset hot zone 303, the electronic device can highlight each control in the target control group 304 corresponding to the target preset hot zone 303, and expand the initial area range of the target preset hot zone 303 to the preset area range, which can be an area range covering a control.

[0103] Furthermore, the electronic device may display direction guidance gestures.

[0104] Optionally, the direction guiding gesture may be a gesture movement direction for guiding the user.

[0105] Continue to see Figure 3 The electronic device may display a direction guiding gesture 305, which may be used to indicate the direction of the user's gesture movement, such as a leftward arrow indicating that the user's gesture movement direction is to the left.

[0106] Optionally, before S130, the interaction method of the air gesture may specifically include: acquiring multiple frames of second air gesture images of the user in real time, and extracting the coordinates of the second target bone point in each frame of the second air gesture image; calculating the offset speed data of the second target bone point in the spatial rectangular coordinate system according to the coordinates of the second target bone point in the multiple frames of the second air gesture image, the offset speed data including component speed data and merged speed data corresponding to each coordinate axis.

[0107] In an embodiment of the present disclosure, the electronic device can acquire multiple frames of the user's second air gesture images in real time, and extract the coordinates of the second target bone point in each frame of the second air gesture image.

[0108] Optionally, the second air gesture image may be an image including the second air gesture.

[0109] Optionally, the second target bone point may be a bone point in the hand bone point. Figure 4 The second target bone point can be bone point 7, bone point 8, bone point 9, bone point 10, etc., which is not limited here.

[0110] Specifically, the electronic device can acquire multiple frames of the user's second air gesture images in real time, and extract the coordinates of the second target bone point in each frame of the second air gesture image.

[0111] Furthermore, the electronic device may calculate the offset speed data of the second target skeleton point in the spatial rectangular coordinate system based on the coordinates of the second target skeleton point in multiple frames of the second air gesture image.

[0112] Optionally, the offset velocity data may include component velocity data corresponding to each coordinate axis and combined velocity data.

[0113] For example, taking the second target skeleton point as skeleton point 7, the electronic device can obtain the coordinates of skeleton point 7 in the spatial rectangular coordinate system in two consecutive frames of the second air gesture image, and calculate the difference between the coordinates, so as to obtain the offset dx of skeleton point 7 on the x-axis, the offset dy on the y-axis, and the offset dz on the z-axis, and then calculate the component velocity data vx of skeleton point 7 on the x-axis, the component velocity data vy on the y-axis, and the component velocity data vz on the z-axis according to the interval time Δt between two consecutive frames of the second air gesture image.

[0114] Optionally, the above calculation formulas may be:

[0115] FPS=1 / Δt

[0116] Among them, FPS can represent frame rate, and Δt can represent interval time.

[0117] dx=x t -x t-1

[0118] Among them, dx can represent the offset on the x-axis, x t It can be expressed as the horizontal coordinate at time t, x t-1 The horizontal coordinate at time t-1 can be expressed, and the offset dy on the y-axis and the offset dz on the z-axis can be calculated similarly.

[0119] vx=dx / Δt

[0120] Among them, vx can represent the component velocity data on the x-axis. Similarly, the component velocity data vy on the y-axis and the component velocity data vz on the z-axis of the skeleton point 7 are calculated. dx can represent the offset on the x-axis, and Δt can represent the interval time.

[0121] v=(vx^2+vy^2+vz^2)^0.5

[0122] Among them, v may represent the combined velocity data, vx may represent the component velocity data on the x-axis, vy may represent the component velocity data on the y-axis, and vz may represent the component velocity data on the z-axis.

[0123] Optionally, the electronic device can use skeleton point 7 as the second target skeleton point to calculate the offset velocity data, or can use skeleton point 7, skeleton point 8, skeleton point 9, and skeleton point 10 as the second target skeleton points and calculate the average value as the second target skeleton point, which is not limited here.

[0124] Therefore, in the embodiment of the present disclosure, the offset speed data can be calculated, so that it can be accurately determined whether the gesture conditions are met in the subsequent process, and the target control group among the multiple control groups can be individually moved and controlled, thereby improving the user experience.

[0125] Optionally, S130 may specifically include: when it is detected that the moving direction of the moving trajectory of the second air gesture is the same as the guiding direction corresponding to the direction guiding gesture, and the offset speed data of the second target bone point in the spatial rectangular coordinate system meets the preset speed data, determining that the second air gesture meets the gesture condition corresponding to the direction guiding gesture.

[0126] In an embodiment of the present disclosure, the electronic device can determine that the second air gesture satisfies the gesture condition corresponding to the direction guiding gesture when it detects that the moving direction corresponding to the moving trajectory of the second air gesture is the same as the guiding direction corresponding to the direction guiding gesture, and the offset speed data of the second target skeleton point in the spatial rectangular coordinate system satisfies the preset speed data.

[0127] Optionally, the preset speed data may be predefined speed data.

[0128] For example, taking the second target skeleton point as skeleton point 7, the electronic device first determines the moving direction corresponding to the moving trajectory of the user's second air gesture. For example, if the direction corresponding to the moving trajectory is moving to the left, which is the same as the guiding direction corresponding to the direction guiding gesture, and the offset speed data of the second target skeleton point in the spatial rectangular coordinate system satisfies the preset speed data. For example, if the guiding direction corresponding to the direction guiding gesture is to the left, the component speed data vx of the skeleton point 7 on the x-axis is less than 0, and the ratio of vx to the merged speed data v is greater than 0.6, and the merged speed data v is greater than 1, then it can be determined that the second air gesture satisfies the gesture condition corresponding to the direction guiding gesture; similarly, if the guiding direction corresponding to the direction guiding gesture is to the right, the component speed data vx of the skeleton point 7 on the x-axis is greater than 0, and the ratio of vx to the merged speed data v is greater than 1. If the ratio of the merged speed data v is greater than 0.6, and the merged speed data v is greater than 1, it can be determined that the second air gesture meets the gesture conditions corresponding to the direction guidance gesture; if the guidance direction corresponding to the direction guidance gesture is upward, the component speed data vy of the skeleton point 7 on the y-axis is less than 0, and the ratio of vx to the merged speed data v is greater than 0.8, and the merged speed data v is greater than 1, it can be determined that the second air gesture meets the gesture conditions corresponding to the direction guidance gesture; if the guidance direction corresponding to the direction guidance gesture is up and down, the component speed data vy of the skeleton point 7 on the y-axis is greater than 0, and the ratio of vx to the merged speed data v is greater than 0.8, and the merged speed data v is greater than 1, it can be determined that the second air gesture meets the gesture conditions corresponding to the direction guidance gesture, etc., which is not limited here.

[0129] Therefore, the electronic device can determine whether the second air gesture meets the corresponding gesture conditions based on different direction-guided gestures, so that when the gesture conditions are met, the target control group in the multiple control groups is moved and controlled separately, thereby improving the user experience.

[0130] Figure 6 An interface schematic diagram of another control display interface provided by an embodiment of the present disclosure is shown.

[0131] like Figure 6 As shown, the electronic device can move the control group in the first row according to the movement trajectory of the user's second air gesture, such as a movement trajectory to the left, until the user stops moving or the row cannot display more controls.

[0132] Optionally, after S130, the air gesture interaction method may further include: when detecting that the cursor on the control display interface moves out of the target preset hot zone, canceling the display of the target control group and direction guidance gesture corresponding to the target preset hot zone, and continuing to display multiple control groups and the preset hot zone corresponding to each control group.

[0133] In an embodiment of the present disclosure, when the electronic device detects that the cursor on the display interface moves out of the target preset hot zone, it cancels display of the target control group and direction guidance gesture corresponding to the target preset hot zone, and continues to display multiple control groups and the preset hot zone corresponding to each control group.

[0134] Specifically, after detecting that the cursor on the display interface moves out of the target preset hot zone, the electronic device can cancel the target control group and direction guidance gesture corresponding to the target preset hot zone, and continue to display the target control group and direction guidance gesture corresponding to the target preset hot zone. Figure 2 Multiple control groups are shown, and a preset hot zone 202 corresponding to each control group.

[0135] Thus, when the user does not need to slide the control, the target control group and the direction guidance gesture can be stopped from being displayed, and multiple control groups can continue to be displayed, thereby meeting the user's needs.

[0136] Figure 7 A schematic structural diagram of an air gesture interaction device provided by an embodiment of the present disclosure is shown.

[0137] In some embodiments of the present disclosure, Figure 7 The interactive device of the air gesture shown can be set in an electronic device. Specifically, the electronic device can include but is not limited to mobile terminals such as mobile phones, vehicle-mounted equipment, vehicle controllers, tablet computers, wearable devices, etc.

[0138] like Figure 7 As shown, the air gesture interaction device 700 may include a first display module 710 , a second display module 720 and a control movement module 730 .

[0139] The first display module 710 can be used to obtain a preset hot zone corresponding to each control group.

[0140] The second display module 720 can be used to display the target control group and direction guidance gesture corresponding to the target preset hot zone when it is detected that the cursor mapped by the user's first air gesture on the control display interface moves into the target preset hot zone.

[0141] The control moving module 730 can be used to move the target control group according to the movement trajectory of the second air gesture when it is detected that the user's second air gesture meets the gesture condition corresponding to the direction guidance gesture.

[0142] Therefore, in the embodiment of the present disclosure, the preset hot zone corresponding to each control group can be obtained, and then when it is detected that the cursor mapped by the user's first air gesture on the control display interface moves into the target preset hot zone, the target control group and the direction guidance gesture corresponding to the target preset hot zone are displayed, and then when it is detected that the user's second air gesture satisfies the gesture condition corresponding to the direction guidance gesture, the target control group is moved according to the moving trajectory of the second air gesture. Therefore, when the cursor corresponding to the user's first air gesture moves into the target preset hot zone, the target control group and the direction guidance gesture corresponding to the target preset hot zone are displayed, and then when the user's second air gesture satisfies the gesture condition corresponding to the direction guidance gesture, the target control group is moved according to the moving trajectory of the second air gesture, so that the target control groups among the multiple control groups can be individually moved and controlled, thereby improving the user experience.

[0143] In some embodiments of the present disclosure, the air gesture interaction device 700 may further include a first acquisition module and a first processing module.

[0144] The first acquisition module can be used to acquire the user's first air gesture image and extract the coordinates of the hand bone points in the first air gesture image before displaying the target control group and guide gesture corresponding to the target preset hot zone when it is detected that the cursor mapped by the user's first air gesture on the control display interface moves into the target preset hot zone.

[0145] The first processing module can be used to map the coordinates of the first target bone point in the hand bone points to obtain the coordinates of the cursor of the first air gesture on the control display interface.

[0146] In some embodiments of the present disclosure, the second display module 720 may specifically include a cursor moving unit and a first determining unit.

[0147] The cursor moving unit can be used to respond to the user's first air gesture and move the corresponding cursor on the control display interface according to the movement trajectory of the first air gesture.

[0148] The first determination unit may be configured to determine that the cursor has moved into the target preset hot zone when it is detected that the coordinates of the cursor are within the coordinate range of the target preset hot zone.

[0149] In some embodiments of the present disclosure, the second display module 720 may further specifically include a first display unit and a second display unit.

[0150] The first display unit can be used to highlight each control in the target control group corresponding to the target preset hot zone, and expand the initial area range of the target preset hot zone to the preset area range, which is the area range covering one control.

[0151] The second display unit can be used to display a direction guiding gesture, where the direction guiding gesture is used to guide the user's gesture movement direction.

[0152] In some embodiments of the present disclosure, the air gesture interaction device 700 may further include a second acquisition module and a second processing module.

[0153] The second acquisition module can be used to acquire multiple frames of the user's second air gesture images in real time and extract the coordinates of the second target bone point in each frame of the second air gesture image before moving the target control group according to the moving trajectory of the second air gesture when it is detected that the user's second air gesture meets the gesture condition corresponding to the direction guiding gesture.

[0154] The second processing module can be used to calculate the offset speed data of the second target bone point in the spatial rectangular coordinate system according to the coordinates of the second target bone point in multiple frames of second air gesture images, and the offset speed data includes component speed data and merged speed data corresponding to each coordinate axis.

[0155] In some embodiments of the present disclosure, the control moving module 730 may specifically include a second determining unit.

[0156] The second determination unit can be used to determine that the second air gesture meets the gesture condition corresponding to the direction guidance gesture when it is detected that the moving direction of the moving trajectory of the second air gesture is the same as the guidance direction corresponding to the direction guidance gesture, and the offset speed data of the second target bone point in the spatial rectangular coordinate system meets the preset speed data.

[0157] In some embodiments of the present disclosure, the air gesture interaction device 700 may further include a third display module.

[0158] The third display module can be used to cancel the display of the target control group and direction guidance gesture corresponding to the target preset hot zone after moving the target control group according to the moving trajectory of the second air gesture, and continue to display multiple control groups and the preset hot zone corresponding to each control group when detecting that the cursor on the control display interface removes the target preset hot zone.

[0159] It should be noted that Figure 7 The interactive device 700 for performing air gestures can be executed Figures 1 to 6 The various steps in the method embodiment shown in the figure are implemented Figures 1 to 6 The various processes and effects in the method embodiment shown are not described in detail here.

[0160] Figure 8 A schematic diagram of the structure of an air gesture interaction device provided by an embodiment of the present disclosure is shown.

[0161] In some embodiments of the present disclosure, Figure 8 The interactive device for air gestures shown may be an intelligent electronic device with which the user wants to interact with air gestures. The intelligent electronic screen device may include a first device and a second device having an interactive function for air gestures. Specifically, the electronic device may include but is not limited to mobile terminals such as mobile phones, vehicle-mounted devices, vehicle controllers, tablet computers, wearable devices, smart home devices, and the like.

[0162] like Figure 8 As shown, the air gesture interaction device may include a processor 801 and a memory 802 storing computer program instructions.

[0163] Specifically, the processor 801 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0164] The memory 802 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 802 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 802 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 802 is a non-volatile solid-state memory. In a particular embodiment, the memory 802 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.

[0165] The processor 801 reads and executes the computer program instructions stored in the memory 802 to perform the steps of the air gesture interaction method provided in the embodiment of the present disclosure.

[0166] In one example, the air gesture interaction device may further include a transceiver 803 and a bus 804. Figure 8 As shown, the processor 801, the memory 802 and the transceiver 803 are connected via a bus 804 and communicate with each other.

[0167] The bus 804 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 804 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0168] The embodiments of the present disclosure also provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the air gesture interaction method provided by the embodiments of the present disclosure.

[0169] The above-mentioned storage medium may, for example, include a memory 802 of computer program instructions, and the above-mentioned instructions may be executed by a processor 801 of the air gesture interaction device to complete the air gesture interaction method provided by the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (CompactDisc ROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0170] The embodiment of the present disclosure also provides a vehicle, which includes the above-mentioned air gesture interaction device. It is understandable that the vehicle may also include: a processor, a memory, and a computer program. The computer program is stored in the memory and is configured to be executed by the processor to implement the air gesture interaction method provided by the embodiment of the present disclosure. The processor and the memory are already in the Figure 3 The parts described in the illustrated embodiment will not be repeated here.

[0171] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0172] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for air gesture interaction, characterized in that: include: Get the preset hot zone corresponding to each control group; When it is detected that the cursor mapped by the user's first air gesture on the control display interface moves into a target preset hot zone, a target control group and a direction guidance gesture corresponding to the target preset hot zone are displayed; When it is detected that the user's second air gesture satisfies the gesture condition corresponding to the direction guiding gesture, the target control group is moved according to the moving track of the second air gesture.

2. The method according to claim 1, characterized in that In the case where it is detected that the cursor mapped on the control display interface of the user's first air gesture moves into the target preset hot zone, before displaying the target control group and the guide gesture corresponding to the target preset hot zone, the method also includes: Acquire a first air gesture image of the user, and extract coordinates of hand skeleton points in the first air gesture image; The coordinates of the first target bone point among the hand bone points are mapped to obtain the coordinates of the cursor of the first air gesture on the control display interface.

3. The method according to claim 2, characterized in that The detecting that the cursor mapped on the control display interface by the user's first air gesture moves into the target preset hot zone includes: In response to the first air gesture of the user, moving a corresponding cursor on the control display interface according to a moving track of the first air gesture; When it is detected that the coordinates of the cursor are within the coordinate range of the target preset hot zone, it is determined that the cursor moves into the target preset hot zone.

4. The method according to claim 1, characterized in that: The displaying of the target control group and direction guiding gesture corresponding to the target preset hot zone includes: Highlight each control in the target control group corresponding to the target preset hot zone, and expand the initial area range of the target preset hot zone to a preset area range, where the preset area range is an area range covering one control; The direction guiding gesture is displayed, where the direction guiding gesture is used to guide the user's gesture moving direction.

5. The method according to claim 1, characterized in that In the case where it is detected that the second air gesture of the user satisfies the gesture condition corresponding to the direction guiding gesture, before moving the target control group according to the moving track of the second air gesture, the method further includes: Acquire multiple frames of the user's second air gesture image in real time, and extract the coordinates of the second target bone point in each frame of the second air gesture image; According to the coordinates of the second target bone point in multiple frames of the second air gesture image, the offset speed data of the second target bone point in the spatial rectangular coordinate system is calculated, and the offset speed data includes component speed data and combined speed data corresponding to each coordinate axis.

6. The method according to claim 5, characterized in that The detecting that the second air gesture of the user satisfies the gesture condition corresponding to the direction guiding gesture includes: When it is detected that the moving direction of the moving trajectory of the second air gesture is the same as the guiding direction corresponding to the direction guiding gesture, and the offset speed data of the second target bone point in the spatial rectangular coordinate system meets the preset speed data, it is determined that the second air gesture meets the gesture condition corresponding to the direction guiding gesture.

7. The method according to claim 1, characterized in that After moving the target control group according to the moving track of the second air gesture, the method further includes: When it is detected that the cursor on the control display interface removes the target preset hot zone, the target control group and direction guidance gesture corresponding to the target preset hot zone are canceled, and the multiple control groups and the preset hot zone corresponding to each control group continue to be displayed.

8. An interactive device for air gestures, characterized in that: include: A first display module is used to obtain a preset hot zone corresponding to each control group; A second display module is used to display a target control group and a direction guidance gesture corresponding to the target preset hot zone when it is detected that the cursor mapped by the user's first air gesture on the control display interface moves into the target preset hot zone; A control moving module is used to move the target control group according to the movement trajectory of the user's second air gesture when it is detected that the user's second air gesture meets the gesture condition corresponding to the direction guiding gesture.

9. An interactive device for air gestures, characterized in that: include: processor; A memory for storing executable instructions; Wherein, the processor is used to read the executable instructions from the memory and execute the executable instructions to implement the air gesture interaction method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the processor implements the interactive method of air gestures described in any one of claims 1 to 7.

11. A vehicle, characterized in that: It includes the interactive device for air gestures as described in claim 8 or the interactive device for air gestures as described in claim 9 or the computer-readable storage medium as described in claim 10.

Images