Method and device for controlling electronic device based on gesture, storage medium and electronic device

Through camera acquisition and gesture recognition technology, the driver can control the electronic equipment in the car without diverting his attention, solving the safety hazards caused by the driver's attention transfer during human-computer interaction in the cabin, and improving driving safety .

CN113448429BActive Publication Date: 2024-11-01BEIJING HORIZON INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010215857.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-03-25
Publication Date
2024-11-01
Estimated Expiration
2040-03-25

AI Technical Summary

Technical Problem

When human-computer interaction is performed in the cabin, the driver needs to control the operation through the touch central control screen, which leads to a shift in attention and increases driving safety risks.

Method used

The driver's hand image sequence is collected through the camera, the hand area is determined, the feature map is obtained, and the gesture is recognized using the pre-established gesture recognition model, and the control command is generated to control the electronic device without diverting attention.

Benefits of technology

It improves the safety of the driving process, allows drivers to control electronic devices through gestures, reduce their dependence on the central control screen, and reduces the safety risks brought by attention divergence.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113448429B_ABST
    Figure CN113448429B_ABST
Patent Text Reader

Abstract

Disclosed are a method and device for controlling an electronic device based on gestures, a storage medium, and an electronic device, wherein the method comprises: determining the hand area of ​​the driver in each image from a captured image sequence; acquiring a feature map corresponding to the hand area in each image, and obtaining at least one feature map corresponding to the image sequence; determining the gesture recognition result in the at least one feature map through a pre-established gesture recognition model; determining a control instruction to be executed based on the gesture recognition result; and controlling the electronic device based on the control instruction. The present disclosure achieves the purpose of controlling an electronic device through a driver's gestures without the driver having to divert his attention, and improving the safety of the driving process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of human-computer interaction, and in particular to a method and device for controlling an electronic device based on gestures, a storage medium, and an electronic device. Background Art

[0002] In the related art, when human-computer interaction is performed in the vehicle cabin, it is necessary to perform the required control operations through the touch-screen central control screen. Summary of the invention

[0003] In the related technology, when the driver's attention is diverted to the central control screen, it will cause driving safety hazards.

[0004] In order to solve the above technical problems, the present disclosure is proposed. Embodiments of the present disclosure provide a method and device for controlling an electronic device based on gestures, a storage medium, and an electronic device.

[0005] According to one aspect of the present disclosure, a method for controlling an electronic device based on gestures is provided, comprising:

[0006] Determine the driver's hand region in each image from the acquired image sequence;

[0007] Acquire a feature map corresponding to a hand region in each of the images to obtain at least one feature map corresponding to the image sequence;

[0008] Determining a gesture recognition result in the at least one feature graph by using a pre-established gesture recognition model;

[0009] Determining a control instruction to be executed according to the gesture recognition result;

[0010] Based on the control instruction, the electronic device is controlled.

[0011] According to another aspect of the present disclosure, a device for controlling an electronic device based on gestures is provided, comprising:

[0012] A first determination module is used to determine the driver's hand area in each image from the acquired image sequence;

[0013] An acquisition module, used to acquire a feature map corresponding to the hand region in each image, and obtain at least one feature map corresponding to the image sequence;

[0014] a second determination module, configured to determine a gesture recognition result in the at least one feature map by using a pre-established gesture recognition model;

[0015] A third determination module, used to determine the control instruction to be executed according to the gesture recognition result;

[0016] The device control module is used to control the electronic device based on the control instruction.

[0017] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute any of the above-mentioned methods for controlling an electronic device based on gestures.

[0018] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0019] processor;

[0020] a memory for storing instructions executable by the processor;

[0021] The processor is used to read the executable instructions from the memory and execute the executable instructions to implement any of the above-mentioned methods for controlling an electronic device based on gestures.

[0022] The present disclosure may include the following benefits:

[0023] The driver's hand area in each image is determined from the collected image sequence. According to the feature map corresponding to the hand area in each image, at least one feature map corresponding to the image sequence can be obtained. The gesture recognition result in at least one feature map is determined by a pre-established hand recognition model, and then the control instruction to be executed is determined. Based on the control instruction, the electronic device is controlled. The purpose of controlling the electronic device through the driver's gesture is achieved without the driver's attention being diverted, and the safety of the driving process is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1 It is a scene graph for controlling an electronic device based on gestures to which the present disclosure is applicable.

[0026] Figure 2 It is a flowchart of a method for controlling an electronic device based on gestures provided by an exemplary embodiment of the present disclosure.

[0027] Figure 3 It is a flowchart of a method for controlling an electronic device based on gestures provided by another exemplary embodiment of the present disclosure.

[0028] Figure 4A is a schematic diagram of a depth feature map provided by another exemplary embodiment of the present disclosure.

[0029] Figure 4B It is a schematic diagram of an intensity characteristic diagram provided by another exemplary embodiment of the present disclosure.

[0030] Figure 5 It is a flowchart of a method for controlling an electronic device based on gestures provided by another exemplary embodiment of the present disclosure.

[0031] Figure 6 It is a flowchart of a method for controlling an electronic device based on gestures provided by another exemplary embodiment of the present disclosure.

[0032] Figure 7 It is a flowchart of a method for controlling an electronic device based on gestures provided by another exemplary embodiment of the present disclosure.

[0033] Figure 8 It is a flowchart of a method for controlling an electronic device based on gestures provided by another exemplary embodiment of the present disclosure.

[0034] Fig. 9 It is a flowchart of a method for controlling an electronic device based on gestures provided by another exemplary embodiment of the present disclosure.

[0035] Fig.10 It is a schematic diagram of a network structure of an electronic device controlled based on gestures provided by another exemplary embodiment of the present disclosure.

[0036] Fig.11 It is a schematic diagram of determining a target gesture and a target control instruction according to a possibility probability value provided by another exemplary embodiment of the present disclosure.

[0037] Fig.12 is a structural diagram of a gesture recognition device provided by an exemplary embodiment of the present disclosure.

[0038] Fig.13 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0040] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0041] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0042] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0043] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0044] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0045] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0046] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0047] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0048] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0049] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0050] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0051] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0052] SUMMARY OF THE DISCLOSURE

[0053] In the present disclosure, the driver's hand area in each image is determined from the collected image sequence, and at least one feature map corresponding to the image sequence can be obtained according to the feature map corresponding to the hand area in each image. The gesture recognition result in at least one feature map is determined by a pre-established hand recognition model, and then the control instruction to be executed is determined, and the electronic device is controlled based on the control instruction. The purpose of controlling the electronic device through the driver's gesture is achieved without the driver's attention being diverted, and the safety of the driving process is improved.

[0054] In one example, reference Figure 1 To describe the scenario of controlling an electronic device based on gestures in an embodiment of the present disclosure.

[0055] Cameras can be set at the vehicle's central control, rearview mirror or ceiling light. In the present disclosure, as long as the camera is positioned to capture the driver's hand area, it can be used. In addition, in order to save costs, the camera can be a low-resolution camera, such as a low-resolution TOF (Time of Flight) camera. The high-frequency light emitted by the modulated light transmitter will be reflected back after hitting an object. The receiver will capture the time of the round trip, and the distance of the object can be obtained by calculation. The light propagation time is different in places with different depths of field. The time difference is used to form a high-precision three-dimensional stereogram, and the recognition purpose is completed by comparison.

[0056] The camera 101 disposed inside the vehicle 100 can collect an image sequence including the hand area of ​​the driver 102. By determining the hand area of ​​the driver 102 in each image from the image sequence, at least one feature map corresponding to the image sequence can be obtained. Furthermore, the driver's gesture can be recognized based on the at least one feature map, that is, a gesture recognition result is obtained, so as to control the electronic device based on the control instruction corresponding to the gesture recognition result.

[0057] The electronic devices include but are not limited to the electronic devices inside the vehicle, such as the electric sunroof, central control equipment, audio playback equipment, Bluetooth communication equipment, other terminals, etc. Other terminals may include terminals carried by the driver or other people in the vehicle. After the Bluetooth communication device inside the vehicle is connected to the designated terminal by Bluetooth, the driver can control the Bluetooth communication device to send control instructions to the terminal through gestures, thereby controlling the terminal.

[0058] Exemplary Methods

[0059] Figure 2 FIG. 1 is a flow chart of a method for controlling an electronic device based on gestures provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to vehicles, such as Figure 2 As shown, the following steps are included:

[0060] Step 201: determine the driver's hand area in each image from the acquired image sequence.

[0061] In the present disclosure, an image sequence may be captured by a camera pre-installed on a vehicle. Furthermore, for the image sequence, a human body key point recognition method may be used to identify the hand area of ​​the driver in each image.

[0062] Step 202: Acquire a feature map corresponding to the hand region in each image to obtain at least one feature map corresponding to the image sequence.

[0063] In the present disclosure, the feature map corresponding to the hand area in each image may include but is not limited to a depth feature map and an intensity feature map, and at least one feature map corresponding to the image sequence is determined based on the depth feature map and / or the intensity feature map. The depth feature map is used to characterize the distance of each pixel point in the hand area in each image from the camera, and the intensity feature map is used to characterize the brightness of each pixel point in the hand area.

[0064] Step 203: Determine a gesture recognition result in the at least one feature map by using a pre-established gesture recognition model.

[0065] In the present disclosure, the gesture recognition model may use a resnet (residual) network or a MobileNet (a neural network built on a mobile terminal) backbone network. At least one feature map is input into a pre-established gesture recognition model, and a gesture recognition result in at least one feature map output by the gesture recognition model may be determined.

[0066] Step 204: Determine the control instruction to be executed according to the gesture recognition result.

[0067] In the present disclosure, the driver can pre-set the direct correspondence between the gesture and the control instruction, and after determining the gesture recognition result, the control instruction to be executed can be determined according to the gesture recognition result. In the present disclosure, the control instruction can be an instruction for controlling an electronic device, such as opening or closing a sunroof, starting or stopping an audio player, adjusting the volume of an audio player, communicating with a car Bluetooth, starting or stopping a car air conditioner, etc.

[0068] Step 205: Control the electronic device based on the control instruction.

[0069] In the present disclosure, after determining the control instruction to be executed, the corresponding electronic device can be controlled based on the control instruction. For example, if the control instruction is to open or close the sunroof, then the electronic device is the sunroof, and the sunroof can be controlled to open or close according to the control instruction. For another example, if the control instruction is to adjust the volume of an audio playback device, then the electronic device is the audio playback device, and the volume of the car audio can be adjusted for the car audio. Of course, other electronic devices can also be controlled in the same way, which will not be described one by one here.

[0070] In the above embodiment, the hand area of ​​the driver in each image is determined from the collected image sequence, and at least one feature map corresponding to the image sequence can be obtained according to the feature map corresponding to the hand area in each image. The gesture recognition result in at least one feature map is determined by a pre-established hand recognition model, and then the control instruction to be executed is determined, and the electronic device is controlled based on the control instruction. The purpose of controlling the electronic device through the driver's gesture is achieved without the driver's attention being diverted, and the safety of the driving process is improved.

[0071] In one example, if Figure 3 As shown in the above Figure 2 Based on the illustrated embodiment, the process of obtaining the feature map corresponding to the hand region in each image in step 202 may include the following steps:

[0072] Step 2021, obtaining a depth feature map corresponding to the hand area in each image.

[0073] In the present disclosure, the camera can determine the depth feature map corresponding to each pixel in each image in the image sequence according to the time and speed of light of the emitted high-frequency light returning, for example Figure 4A After extracting the driver's hand area in each image, the depth feature corresponding to the hand area can be extracted. The depth feature map is used to characterize the distance between each pixel point of the hand area in each image and the image acquisition device.

[0074] Step 2022, obtaining an intensity feature map corresponding to the hand area in each image.

[0075] In the present disclosure, when the camera collects the image sequence, it can also determine the brightness corresponding to each pixel in each image in the image sequence to obtain an intensity feature map corresponding to each image, such as Figure 4B After determining the hand region of the driver in each image of the image sequence, the intensity feature map corresponding to the hand region in each image can be directly extracted. The intensity feature map is used to characterize the brightness of each pixel in the hand region.

[0076] In the above embodiment, the depth feature map and intensity feature map corresponding to the hand area in each image can be extracted through a pre-installed camera, so as to subsequently determine at least one feature map corresponding to the image sequence. Since the depth feature map can represent the distance of each pixel point in the hand area in the image from the camera, and the intensity feature map can represent the brightness of each pixel point in the hand area in the image, the characteristics of the hand area can be accurately determined by combining the depth feature map and the intensity feature map. The above feature maps can accurately reflect the driver's intentions.

[0077] In one example, the process of obtaining at least one feature map corresponding to the image sequence in step 202 may include the following steps:

[0078] Step 2023: pre-process the depth feature map and / or intensity feature map corresponding to the hand area in each image to obtain at least one feature map corresponding to the image sequence.

[0079] In the present disclosure, since the camera uses a low-resolution camera, there is a lot of noise, and it is necessary to preprocess the depth feature map and / or intensity feature map corresponding to the hand area in each image to reduce the impact of noise and obtain at least one feature map corresponding to the image sequence.

[0080] In the above embodiment, the depth feature map and / or intensity feature map corresponding to the hand area in each image can be preprocessed to obtain at least one feature map corresponding to the image sequence, thereby reducing the impact of noise and making the subsequently determined gesture recognition result more accurate.

[0081] In one example, if Figure 5 As shown, based on the above-mentioned embodiment, in step 2023, preprocessing the depth feature map corresponding to the hand area in each image to obtain at least one feature map corresponding to the image sequence may include the following steps:

[0082] Step 301: determine a first target pixel point that needs to be retained on a depth feature map corresponding to a hand region in each image.

[0083] In the present disclosure, the first target pixel point may be a pixel point whose depth value on the depth feature map is within a preset area range. For example, the preset area range is 20 cm to 70 cm, and all pixels whose depth values ​​on the depth feature map satisfy the preset area range of [20, 70] belong to the first target pixel point.

[0084] In step 302, all pixels other than the first target pixel are removed from the depth feature map corresponding to the hand area in each image to obtain at least one depth feature map corresponding to the image sequence.

[0085] In the present disclosure, it is necessary to remove all pixels other than the first target pixel, so as to obtain at least one depth feature map corresponding to the image sequence.

[0086] In the above embodiment, on the depth feature map corresponding to the hand area in each image, all pixels whose depth values ​​are outside the preset area range are removed, and only the first target pixel whose depth value is within the preset area range is retained to obtain at least one depth feature map corresponding to the image sequence. This reduces the influence of noise and improves the accuracy of the subsequent gesture recognition result.

[0087] In one example, if Figure 6 As shown in the above Figure 5 Based on the illustrated embodiment, preprocessing the intensity feature map corresponding to the hand region in each image in step 2023 to obtain at least one feature map corresponding to the image sequence may include the following steps:

[0088] Step 401, determining a second target pixel point on the intensity feature map corresponding to the hand area in each image.

[0089] Among them, the position of each second target pixel point on the intensity feature map corresponding to the hand area in each image corresponds one-to-one to the position of each first target pixel point on the depth feature map corresponding to the hand area in each image.

[0090] The second target pixel point on the intensity feature map corresponds to the first target pixel point on the depth feature map whose depth value is within the preset area.

[0091] Step 402: remove all pixels other than the second target pixel in the intensity feature map corresponding to the hand area in each image to obtain at least one intensity feature map corresponding to the image sequence.

[0092] In the present disclosure, all pixels except the second target pixel can be removed from the intensity feature map corresponding to the hand area in each image, so as to obtain at least one intensity feature map corresponding to the image sequence.

[0093] In the above embodiment, on the intensity feature map corresponding to the hand area in each image, the second target pixel points corresponding to the first target pixel points whose depth values ​​are within the preset area on the depth feature map are retained to obtain at least one intensity feature map corresponding to the image sequence. This also reduces the influence of noise and improves the accuracy of the subsequent gesture recognition results.

[0094] In one example, if Figure 7 As shown in the above Figure 2 Based on the illustrated embodiment, before executing step 203, the method may further include the following steps:

[0095] In step 206, the image size of the at least one feature map is adjusted to a preset size; and / or the dimension of the at least one feature map is adjusted to a preset dimension corresponding to the gesture recognition model.

[0096] In the present disclosure, in order to facilitate better gesture recognition in the subsequent process, after obtaining at least one feature map corresponding to the image sequence, the image size of at least one feature map can be adjusted to a preset size, and / or the dimension of the at least one feature map can be adjusted to a preset dimension corresponding to the gesture recognition model.

[0097] For example, the feature map may be upsampled or downsampled according to the preset size, so as to adjust the image size to the preset size. If the preset dimension corresponding to the gesture recognition model is 224×224, the dimension of at least one feature map may be adjusted to 224×224.

[0098] In one example, if Figure 8 As shown in the above Figure 2 Based on the illustrated embodiment, step 203 may include the following steps:

[0099] Step 2031: Determine key point information corresponding to the hand area in the at least one feature map through a pre-established gesture recognition model.

[0100] The gesture recognition model has been trained based on the backbone network of ResNet or Mobile Net. By inputting at least one feature map into the gesture recognition model, key point information corresponding to the hand area in at least one feature map output by the gesture recognition model can be obtained. The key point information may include but is not limited to information on various joints and fingertips of the hand.

[0101] Step 2032: Obtain, based on the key point information, a probability value of the possibility that the gesture in the at least one feature image matches each preset gesture.

[0102] In the present disclosure, based on these key point information, the possibility probability value of the gesture in at least one feature map matching each preset gesture can be obtained.

[0103] For example, there are 9 preset gestures. Based on the key point information, the possibility probability values ​​of at least one gesture in the feature map matching the 9 gestures can be determined. For example, the possibility probability value of matching preset gesture 1 is P1, the possibility probability value of matching preset gesture 2 is P2, and so on.

[0104] In the above embodiment, the key point information corresponding to the hand area in at least one feature map can be determined by a pre-established gesture recognition model, and then the probability value of the possibility that the gesture in at least one feature map matches each preset gesture can be obtained based on the key point information. This allows the control instruction to be determined based on the gesture recognition result, which has high usability.

[0105] In one example, if Fig. 9 As shown in the above Figure 8 Based on the illustrated embodiment, step 204 may include the following steps:

[0106] Step 2041, determining the maximum value among the possibility probability values.

[0107] In the present disclosure, a maximum value may be determined among the possibility probability values ​​matching each preset gesture.

[0108] Step 2042: If the maximum value is greater than or equal to a preset threshold, the preset gesture corresponding to the maximum value is used as a target gesture.

[0109] In the present disclosure, in order to avoid misjudgment, if the above maximum value is greater than or equal to a preset threshold, it can be considered that the driver has made a valid gesture, and the preset gesture corresponding to the maximum value is used as the target gesture.

[0110] Step 2043: Determine a target control instruction corresponding to the target gesture according to a preset correspondence relationship between gestures and control instructions.

[0111] In the present disclosure, a correspondence relationship between gestures and control instructions is preset, and according to this correspondence relationship, a target control instruction corresponding to the target gesture can be determined.

[0112] For example, the correspondence includes "ok gesture" corresponding to "confirmation instruction", "palm five fingers open gesture" corresponding to "rejection instruction", "thumbs up gesture" corresponding to "good comment instruction", "V-sign gesture" corresponding to "open or close sunroof instruction", etc. If the target gesture is the ok gesture, it can be determined that the target control instruction is the confirmation instruction.

[0113] Step 2044: If the maximum value is less than the preset threshold, it is determined that the target gesture is an invalid gesture and the target control instruction is empty.

[0114] In the present disclosure, if the maximum value of the probability value of the possibility of matching each preset gesture determined previously is less than a preset threshold, then the target gesture is considered to be an invalid gesture and the target control instruction is empty.

[0115] In the above embodiment, the maximum value of the possibility probability value can be determined based on the possibility probability value matching each preset gesture, and whether the target gesture is a valid gesture can be judged based on the maximum value and the preset threshold. On the premise that the target gesture is a valid gesture, the corresponding target control instruction can be determined based on the target gesture, thereby avoiding misjudgment of some gesture actions of the driver and improving the accuracy of controlling electronic devices through gestures.

[0116] In one example, Fig.10: is a schematic diagram of a network structure of an electronic device based on gesture control provided by an exemplary embodiment of the present disclosure, including:

[0117] After determining the hand area of ​​the driver in each image in the acquired image sequence, a depth feature map and an intensity feature map corresponding to the hand area in each image are obtained, and after preprocessing the depth feature map and the intensity feature map, at least one feature map corresponding to the image sequence is obtained.

[0118] At least one feature map is input into the gesture recognition model. In the present disclosure, the gesture recognition model can be input into the gesture recognition model according to two channels, depth and intensity. The gesture recognition model uses MobileNetV2 as the backbone network.

[0119] Since the preset dimension corresponding to the MobileNetV2 network is 32 channels, at least one feature map can be input into the gesture recognition model through 2 channels, and then increased to 12 channels using 1×1 convolution, and then increased from 12 channels to 32 channels through the convolution layer.

[0120] The gesture recognition model can determine the key point information corresponding to the hand area in at least one feature map, and then output the probability value of the possibility that the gesture in at least one feature map matches each preset gesture based on the key point information. For example, the preset gestures include 9 gestures, and the output probability value is as follows: Fig.11 shown.

[0121] exist Fig.11 The maximum value is determined among the 9 possible probability values ​​shown, and then it is determined whether the maximum value is greater than or equal to a preset threshold. If it is greater than or equal to the preset threshold, the target control instruction corresponding to the target gesture corresponding to the maximum value can be determined based on the correspondence between the gesture and the control instruction. Otherwise, it can be determined that the target gesture is an invalid gesture and the target control instruction is empty.

[0122] In the above embodiment, the above network architecture can quickly and accurately identify the driver's target gesture and determine whether the target gesture is a valid gesture. When the target gesture is a valid gesture, the corresponding target control instruction is determined, so as to control the electronic device to execute the target control instruction, thereby achieving the purpose of controlling the electronic device through the driver's gesture without the driver's attention being diverted, and improving the safety of the driving process.

[0123] In the present disclosure, when training a gesture recognition model, a sample feature map including a gesture area can be used to train a neural network. By adjusting the network parameters of the neural network, the gesture recognition result output by the neural network is matched with the gesture recognition result pre-marked in the sample feature map or is within the tolerance range, that is, the loss function is minimized, thereby obtaining a gesture recognition model.

[0124] In the present disclosure, during the training of the gesture recognition model, the loss function may include a first loss function and a second loss function, wherein the first loss function can be used to improve the recognition performance between gestures of different categories, and the second loss function can be used to learn gesture categories with a smaller proportion.

[0125] For example, the gesture of raising the index finger to represent 1 is very similar to the gesture of bending the index finger to represent 9. If the driver makes these two gestures, it is very likely to cause misjudgment. Therefore, the network function can be adjusted to minimize the first loss function, so that the gesture recognition model can better recognize similar gestures.

[0126] For another example, the neural network may learn more about how to identify gesture categories that account for a large proportion during the training process, such as the gestures commonly used by drivers, such as the "OK gesture" and "V gesture". The neural network can accurately determine the commonly used gestures by adjusting the network parameters, but the gestures that drivers rarely make, such as "raising three fingers to represent the gesture of 3", may not be easy for the neural network to recognize these gestures during the training process, and the misjudgment rate is high. The present disclosure can use the second loss function to allow the neural network to better learn the gesture categories that account for a small proportion, thereby improving the accuracy of the gesture recognition model.

[0127] In the above embodiment, the gesture recognition model can be trained by using the first loss function and the second loss function to improve the accuracy of gesture recognition by the gesture recognition model, and the usability is high.

[0128] Exemplary Devices

[0129] Fig.12 FIG. 6 is a structural diagram of an apparatus 6 for controlling an electronic device based on gestures provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to vehicles, such as Fig.12 As shown, including:

[0130] A first determination module 610 is used to determine the driver's hand area in each image from the acquired image sequence;

[0131] An acquisition module 620 is used to acquire a feature map corresponding to the hand region in each image, and obtain at least one feature map corresponding to the image sequence;

[0132] A second determination module 630, configured to determine a gesture recognition result in the at least one feature map by using a pre-established gesture recognition model;

[0133] A third determination module 640 is used to determine the control instruction to be executed according to the gesture recognition result;

[0134] The device control module 650 is used to control the electronic device based on the control instruction.

[0135] In the present disclosure, the acquisition module 620 includes:

[0136] A first acquisition submodule is used to acquire a depth feature map corresponding to the hand area in each image, wherein the depth feature map is used to represent the distance between each pixel point of the hand area in each image and the image acquisition device;

[0137] The second acquisition submodule is used to acquire an intensity feature map corresponding to the hand area in each image, wherein the intensity feature map is used to characterize the brightness of each pixel in the hand area.

[0138] In the present disclosure, the acquisition module 620 further includes:

[0139] The preprocessing submodule is used to preprocess the depth feature map and / or intensity feature map corresponding to the hand area in each image to obtain at least one feature map corresponding to the image sequence.

[0140] In the present disclosure, the preprocessing submodule includes:

[0141] Determining a first target pixel point to be retained on the depth feature map corresponding to the hand area in each image, wherein the depth value of the first target pixel point is within a preset interval;

[0142] All pixels other than the first target pixel are removed from the depth feature map corresponding to the hand area in each of the images to obtain at least one depth feature map corresponding to the image sequence.

[0143] In the present disclosure, the preprocessing submodule includes:

[0144] Determine a second target pixel point on the intensity feature map corresponding to the hand region in each of the images, wherein the position of each of the second target pixel points on the intensity feature map corresponding to the hand region in each of the images corresponds one-to-one to the position of each of the first target pixel points on the depth feature map corresponding to the hand region in each of the images;

[0145] All pixel points other than the second target pixel point are removed from the intensity feature map corresponding to the hand area in each of the images to obtain at least one intensity feature map corresponding to the image sequence.

[0146] In the present disclosure, the device further comprises:

[0147] a size adjustment module, configured to adjust the image size of the at least one feature map to a preset size; and / or

[0148] A dimension adjustment module is used to adjust the dimension of the at least one feature map to a preset dimension corresponding to the gesture recognition model.

[0149] In the present disclosure, the second determining module includes:

[0150] A first determination submodule, configured to determine key point information corresponding to the hand region in the at least one feature map by using a pre-established gesture recognition model;

[0151] The second determination submodule is used to obtain a probability value of the possibility that the gesture in the at least one feature map matches each preset gesture according to the key point information.

[0152] In the present disclosure, the third determination module includes:

[0153] A third determination submodule is used to determine a maximum value among the possibility probability values;

[0154] a fourth determination submodule, configured to use the preset gesture corresponding to the maximum value as a target gesture if the maximum value is greater than or equal to a preset threshold;

[0155] a fifth determination submodule, configured to determine a target control instruction corresponding to the target gesture according to a preset correspondence relationship between the gesture and the control instruction;

[0156] The sixth determination submodule is configured to determine that the target gesture is an invalid gesture and the target control instruction is empty if the maximum value is less than the preset threshold.

[0157] Exemplary Electronic Devices

[0158] refer to Fig.13 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them. Fig.13 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0159] like Fig.13As shown, the electronic device 7 includes a processor 711 and a memory 712 connected to the processor 711. The memory 712 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 711 may run the program instructions to implement the method for controlling an electronic device based on gestures of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0160] In one example, the electronic device 7 may further include: an input device 713 and an output device 714 connected to the processor 711 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0161] For example, when the electronic device is the first device or the second device, the input device 713 is the microphone or microphone array described above, which is used to capture the input signal of the sound source. When the electronic device is a stand-alone device, the input device 713 can be a communication network connector, which is used to receive the collected input signal from the first device and the second device.

[0162] In addition, the input device 713 may also include, for example, a keyboard, a mouse, and the like.

[0163] The output device 714 can output various information to the outside, including the determined distance information, direction information, etc. The output device 714 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0164] Of course, to simplify, Fig.13 Only some of the components related to the present disclosure in the electronic device 7 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 7 may also include any other appropriate components.

[0165] Exemplary computer program products and computer-readable storage media

[0166] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the sound source localization method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0167] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0168] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the processor executes the steps of the sound source localization method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0169] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0170] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0171] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0172] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0173] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0174] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for controlling an electronic device based on gestures, comprising: Determine the driver's hand region in each image from the acquired image sequence; Acquire a feature map corresponding to a hand region in each of the images to obtain at least one feature map corresponding to the image sequence, wherein the feature map corresponding to the hand region in each of the images includes a depth feature map; Determining a gesture recognition result in the at least one feature graph by using a pre-established gesture recognition model; Determining a control instruction to be executed according to the gesture recognition result; Based on the control instruction, control the electronic device; The obtaining of at least one feature map corresponding to the image sequence includes: determining a first target pixel point that needs to be retained on the depth feature map corresponding to the hand area in each image, wherein the depth value of the first target pixel point is within a preset interval; and removing all pixels other than the first target pixel point on the depth feature map corresponding to the hand area in each image to obtain at least one depth feature map corresponding to the image sequence.

2. The method according to claim 1, wherein: The obtaining of a feature map corresponding to a hand region in each image includes: Acquire a depth feature map corresponding to the hand region in each image, wherein the depth feature map is used to represent the distance between each pixel point of the hand region in each image and the image acquisition device; An intensity feature map corresponding to the hand region in each of the images is obtained, wherein the intensity feature map is used to characterize the brightness of each pixel in the hand region.

3. The method according to claim 2, wherein: The obtaining of at least one feature map corresponding to the image sequence comprises: Preprocess the depth feature map and / or the intensity feature map corresponding to the hand area in each image to obtain at least one feature map corresponding to the image sequence.

4. The method according to claim 3, wherein: The preprocessing of the intensity feature map corresponding to the hand region in each image to obtain at least one feature map corresponding to the image sequence includes: Determine a second target pixel point on the intensity feature map corresponding to the hand region in each of the images, wherein the position of each of the second target pixel points on the intensity feature map corresponding to the hand region in each of the images corresponds one-to-one to the position of each of the first target pixel points on the depth feature map corresponding to the hand region in each of the images; All pixel points other than the second target pixel point are removed from the intensity feature map corresponding to the hand area in each of the images to obtain at least one intensity feature map corresponding to the image sequence.

5. The method according to claim 1, wherein: Before determining the gesture recognition result in the at least one feature map by using the pre-established gesture recognition model, the method further includes: Adjusting the image size of the at least one feature map to a preset size; and / or The dimension of the at least one feature map is adjusted to a preset dimension corresponding to the gesture recognition model.

6. The method according to claim 1, wherein: The determining, by using a pre-established gesture recognition model, a gesture recognition result in the at least one feature graph includes: Determining key point information corresponding to the hand area in the at least one feature map by using a pre-established gesture recognition model; A probability value of the possibility that a gesture in the at least one feature map matches each preset gesture is obtained according to the key point information.

7. A device for controlling an electronic device based on gestures, comprising: A first determination module is used to determine the driver's hand area in each image from the acquired image sequence; an acquisition module, configured to acquire a feature map corresponding to the hand region in each image, and obtain at least one feature map corresponding to the image sequence, wherein the feature map corresponding to the hand region in each image includes a depth feature map; a second determination module, configured to determine a gesture recognition result in the at least one feature map by using a pre-established gesture recognition model; A third determination module, used to determine the control instruction to be executed according to the gesture recognition result; A device control module, used to control the electronic device based on the control instruction; Among them, the acquisition module includes a preprocessing submodule, which is used to determine the first target pixel point that needs to be retained on the depth feature map corresponding to the hand area in each image, wherein the depth value of the first target pixel point is within a preset interval; on the depth feature map corresponding to the hand area in each image, all pixels other than the first target pixel point are removed to obtain at least one depth feature map corresponding to the image sequence.

8. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method for controlling an electronic device based on gestures as described in any one of claims 1 to 6.

9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the method for controlling an electronic device based on gestures as described in any one of claims 1 to 6.