Gesture-based vehicle interaction method, electronic equipment and vehicle
By identifying and analyzing the driver's gesture images, the control of on-board equipment and applications is achieved, and the problem of in-board equipment and application control in the prior art is solved, and interaction efficiency and driving safety are improved.
Patent Information
- Application Number
- CN202310988674.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-07
- Publication Date
- 2025-02-14
AI Technical Summary
The control of existing vehicle-mounted equipment and vehicle-mounted applications mainly relies on touch display screens or voice operations, which leads to difficulty in performing complex operations quickly and accurately and affects driving safety.
By identifying and parsing the driver's gesture images, determining the target gesture operation, and generating control instructions for corresponding vehicle-mounted devices or applications, it is possible to control the vehicle-mounted devices and applications without eye-catching the display screen or cumbersome voice interaction.
It improves the interaction efficiency between users and vehicles, ensures driving safety during the interaction between drivers and vehicles, and provides a more natural and intuitive vehicle control method.
Smart Images
Figure CN119440237A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of intelligent automobile technology, and in particular to a vehicle interaction method based on gestures, an electronic device, and a vehicle. Background Art
[0002] As the level of automobile intelligence increases, the number of in-vehicle devices and applications that are equipped with them is also increasing. The existing control of in-vehicle devices and applications is mostly achieved through touch-screen in-vehicle display or voice operation. However, voice control is difficult to quickly and accurately control in-vehicle devices to perform more complex operations, and touch-screen in-vehicle display requires the driver to keep his eyes fixed on the screen, which affects driving safety. Summary of the invention
[0003] The present disclosure provides a vehicle interaction method based on gestures, an electronic device and a vehicle.
[0004] According to a first aspect of the present disclosure, a gesture-based vehicle interaction method is provided, comprising:
[0005] Determine a target gesture operation according to at least one gesture image;
[0006] Determine a target vehicle-mounted device or a target vehicle-mounted application corresponding to the target gesture operation;
[0007] Generate control instructions for the target vehicle device or target vehicle application based on the target gesture operation.
[0008] According to a second aspect of the present disclosure, there is provided an electronic device, including:
[0009] at least one processor; and
[0010] a memory communicatively connected to the at least one processor; wherein,
[0011] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method in any embodiment of the present disclosure.
[0012] According to a third aspect of the present disclosure, a vehicle is provided, comprising the electronic device according to the second aspect of the present disclosure.
[0013] According to the technology disclosed in the present invention, gestures can be directly used to control vehicle-mounted devices or vehicle-mounted applications, thereby improving the interaction efficiency between users and vehicles and ensuring driving safety during the interaction between drivers and vehicles.
[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0016] Figure 1 is a flow chart of a vehicle interaction method based on gestures according to an embodiment of the present disclosure;
[0017] Figure 2 is a schematic diagram of determining a gesture rotation angle according to an embodiment of the present disclosure;
[0018] Figure 3 is a schematic diagram of gesture control for a vehicle-mounted display screen according to an embodiment of the present disclosure;
[0019] Figure 4 is a flowchart of gesture control wake-up according to an embodiment of the present disclosure;
[0020] Figure 5 is a logical architecture diagram of a vehicle interaction method based on gestures according to an embodiment of the present disclosure;
[0021] Figure 6 The block diagram is a block diagram of an electronic device used to implement the vehicle interaction method based on gestures according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0023] Figure 1 : is a flow chart of a vehicle interaction method based on gestures according to an embodiment of the present disclosure, comprising:
[0024] Step S110: determining a target gesture operation according to at least one gesture image;
[0025] Step S120: determining a target vehicle-mounted device or a target vehicle-mounted application corresponding to the target gesture operation;
[0026] Step S130: Generate a control instruction for a target vehicle-mounted device or a target vehicle-mounted application according to the target gesture operation.
[0027] Exemplarily, in step S110, at least one gesture image can be obtained by detecting a preset gesture control area by at least one vehicle-mounted camera to obtain and identify the user's gesture operation in the gesture control area. The vehicle-mounted camera can be a monocular camera or a multi-camera. At least one gesture image can also be detected by an OMS (Occupancy Monitoring System) camera. The OMS carried in a smart vehicle can provide functions such as monitoring of the occupants in the vehicle, monitoring of vital signs, and positioning through visual sensors such as cameras and radar sensors, etc., to improve driving safety. Therefore, without setting up additional cameras and gesture control areas, the OMS can be locked to a user with gesture control intentions in the vehicle by voice wake-up or gesture wake-up, and the gesture operation of the corresponding user can be obtained based on the OMS camera.
[0028] In addition, other detection devices such as vehicle-mounted ultrasonic radars may also be used to detect and identify user gesture operations. At least one gesture image may be an image detected by a single detection device, such as an image detected by a vehicle-mounted camera; at least one gesture image may also be a collection of images detected by multiple detection devices, such as an ultrasonic image detected by a vehicle-mounted ultrasonic radar and an image detected by a vehicle-mounted camera, thereby integrating multiple detection devices to improve the accuracy of gesture operation recognition.
[0029] In step S120, the user's gesture operation may be the control of a certain vehicle-mounted device, such as the rotation of the vehicle-mounted display screen, the lifting and lowering of the window, etc. The vehicle-mounted display screen may be any display screen set on the vehicle, such as the central control screen, the co-pilot screen, the head-up display, and the rear screen, etc.; it may also be the control of a certain vehicle-mounted application, such as the vehicle-mounted navigation APP, the vehicle-mounted music player, etc. After the target gesture operation is identified, it is necessary to determine the target vehicle-mounted device or target vehicle-mounted application corresponding to the target gesture operation, so as to perform control interaction based on the gesture.
[0030] According to the method of this embodiment, a vehicle interaction means other than voice operation and touch operation is provided. There is no need for cumbersome multiple rounds of voice interaction with the vehicle, nor is there a need to look at the vehicle-mounted display screen. The vehicle-mounted equipment or vehicle-mounted applications can be directly controlled by gestures, thereby improving the interaction efficiency between the user and the vehicle and ensuring the driving safety of the driver during the interaction with the vehicle.
[0031] In one implementation, determining the target vehicle-mounted device corresponding to the target gesture operation in step S120 includes: determining the target vehicle-mounted device according to a pre-stored mapping relationship between the gesture operation and the vehicle-mounted device.
[0032] The mapping relationship between the pre-stored gesture operations and the vehicle-mounted devices may be configured at the factory or may be customized by the user. For example, the gesture operation of "yeah" may correspond to the vehicle-mounted camera, specifically to control the vehicle-mounted camera to take pictures; the gesture operations of clenching a fist and spreading a palm may correspond to the vehicle-mounted air conditioner, specifically to turn on and off the air conditioner, etc. The mapping relationship between the above gesture operations and the vehicle-mounted devices is only an example, and the present application is not limited thereto.
[0033] In one implementation, determining the target in-vehicle application corresponding to the target gesture operation in step S120 includes: determining the current in-vehicle application on the in-vehicle display screen as the target in-vehicle application.
[0034] Among them, the current in-vehicle application can be understood as the in-vehicle application at the forefront of the interface on the in-vehicle display screen. The same gesture operation can correspond to different control methods in different in-vehicle applications. For example, when the current in-vehicle application is a side door window control application, the gesture operations of sliding the palm up and down can correspond to the opening and closing of the window respectively; when the current in-vehicle application is a in-vehicle music player, the gesture operations of sliding the palm up and down can correspond to the song switching operations of "previous song" and "next song" respectively.
[0035] It can be understood that whether determining the target vehicle-mounted device based on the pre-stored mapping relationship between gesture operations and vehicle-mounted devices, or determining the target vehicle-mounted application based on the current vehicle-mounted application on the vehicle-mounted display screen, is to accurately determine the control object of the target gesture operation. In addition to these two determination methods, other control methods such as voice control can also be combined with gesture control to achieve control of vehicle-mounted devices or vehicle-mounted applications. For example, the target vehicle-mounted device corresponding to the target gesture operation can be determined as the vehicle-mounted display screen based on the recognized user voice "Turn on vehicle-mounted display screen gesture control", thereby achieving control of the vehicle-mounted display screen based on the specific operation content of the target gesture operation.
[0036] According to the method of this embodiment, the control object of the target gesture operation can be accurately determined based on the pre-stored mapping relationship or the current vehicle-mounted application in the vehicle-mounted display screen, thereby avoiding the target gesture operation from erroneously controlling other devices or applications, and achieving efficient and precise control of vehicle-mounted devices and vehicle-mounted applications.
[0037] In one implementation, determining the target gesture operation according to at least one gesture image in step S110 includes:
[0038] The gesture image is detected to obtain at least one hand key point in the gesture image; and a target gesture operation is determined according to the at least one hand key point.
[0039] Among them, the hand key points can be the joints of the hand, or they can be customized points that can accurately identify the hand posture. After obtaining at least one gesture image, the gesture image can be processed using a trained hand key point recognition model to obtain at least one hand key point in the gesture image, so as to determine the target gesture operation based on the identified hand key points.
[0040] In one implementation, the at least one gesture image includes gesture images collected at different times, and determining the target gesture operation according to the at least one gesture image in step S110 includes:
[0041] Based on at least one gesture image, gesture change information within a preset time period is determined; wherein the gesture change information includes hand posture changes and / or gesture rotation angles; and based on the gesture change information, a target gesture operation is determined.
[0042] It is understandable that the target gesture operation can be a static gesture operation, such as making a "yeah" sign or making a fist, etc.; the target gesture operation can also be a dynamic gesture operation, such as a change in the gesture posture of the hand from making a fist to spreading the palm, or a clockwise or counterclockwise rotation of the palm, etc. The preset duration can be the maximum duration of the dynamic gesture operation specified in advance. Therefore, it is necessary to identify the gesture images collected at different times, so as to determine the gesture change information within the preset duration, and further determine the target gesture operation based on the gesture change information.
[0043] Specifically, determining gesture change information within a preset time period according to at least one gesture image includes: respectively acquiring coordinates of at least two first hand key points in a first gesture image and coordinates of at least two second hand key points in a second gesture image; wherein the time interval between the first gesture image and the second gesture image is the preset time period, and positions of the first hand key points and the second hand key points on the hand correspond one to one;
[0044] The gesture change information is determined according to the coordinates of at least two first hand key points and the coordinates of at least two second hand key points.
[0045] For example, if the first gesture image and the second gesture image are two gesture images acquired at a time interval of 2S, the coordinates of the two first hand key points in the first gesture image are key point A (x1, y1, z1) and key point B (x2, y2, z2), and the coordinates of the two second hand key points in the second gesture image are key point C (x3, y3, z3) and key point D (x4, y4, z4), and key point A and key point C both correspond to the center point of the palm, and key point B and key point D both correspond to the top joint point of the index finger, then the position change information of the center point of the palm and the top joint point of the index finger within 2S can be determined according to the coordinates of key points A, B, C, and D, thereby determining the gesture change information. If more complex gesture change information needs to be obtained, more gesture images acquired at different times within a preset time length can be obtained, and more point coordinates of hand key points that correspond one-to-one to hand key points in other gesture images can be obtained in each gesture image, thereby calculating the gesture change information.
[0046] Figure 2 It is a schematic diagram of determining the rotation angle of a gesture according to an embodiment of the present disclosure, wherein key point A in the first gesture image corresponds to the center point of the palm, key point B corresponds to the top joint point of the index finger, and the straight line segment AB can be determined based on key points A and B. Key point C in the second gesture image corresponds to the center point of the palm, key point D corresponds to the top joint point of the index finger, and the straight line segment CD can be determined based on key points C and D. The angle between the two direct segments AB and CD is also the gesture rotation angle within the time interval (preset duration) between obtaining the first gesture image and the second gesture image. If the angle θ between the direct segments AB and CD is 60°, and the first gesture image is obtained earlier than the second gesture image, it can be determined that the gesture change information is a 60° clockwise rotation of the hand.
[0047] Preferably, during vehicle driving, factors such as ground bumps may cause the hands to passively shake and generate gesture change information. Therefore, an angle threshold may be preset. If the detected gesture rotation angle is less than the preset angle threshold, it is considered that the gesture has not changed, thereby avoiding erroneous operations.
[0048] It can be understood that in the embodiments of the present application, the first gesture image and the second gesture image can be obtained at different times by a multi-eye camera (such as a binocular camera or a tri-eye camera); multiple first gesture images and multiple second gesture images can be obtained at different times in a combined form by setting multiple monocular cameras in different directions, and the gesture images obtained by multiple monocular cameras in different directions at the same time can be combined to obtain the three-dimensional coordinates of the hand key points; it can also be in the form of a combination of a monocular camera and a multi-eye camera, for example, a binocular camera is set in the direction of commonly used gestures, and a monocular camera is set in other directions to assist in obtaining gesture images; for gesture images obtained by different combinations or different types of cameras, it is only necessary to pre-train a matching hand key point recognition model to obtain the three-dimensional coordinates of each hand key point.
[0049] According to the method of this embodiment, static or dynamic target gesture operations are identified based on the key points of the hand, so that richer gesture operations can be used to control vehicle-mounted applications or vehicle-mounted devices, expanding the control scope that can be achieved by gesture operations.
[0050] Preferably, gesture images can be acquired in a combined form by setting up multiple monocular cameras in different directions, so that the user's 3D control gestures can be recognized by a low-cost 2D camera, and the vehicle-mounted application or vehicle-mounted equipment can be controlled accordingly based on the recognized gesture actions.
[0051] In one embodiment, if the camera for detecting user gesture operations is only a monocular camera, such as a monocular OMS camera, the OMS camera can be used to obtain two-dimensional gesture images of the target user at different times, and identify the two-dimensional coordinates of the key points of the hand in each two-dimensional gesture image. Further combined with the relative position relationship between the user's hand and the OMS camera at different times, the three-dimensional coordinates of the key points of the hand in each two-dimensional gesture image in the car are determined, and based on the changes in the three-dimensional coordinates of the key points of the hand at different times, the hand posture changes and / or gesture rotation angles, that is, 3D control gestures, are obtained, and the 3D control gestures are identified to form corresponding control of the target vehicle-mounted application or target vehicle-mounted equipment.
[0052] Specifically, an in-vehicle coordinate system can be established with the center point of the vehicle as the coordinate origin. The radar sensor provided by the OMS, such as a millimeter-wave radar, can be used to detect the relative position relationship between the user's hand and the OMS camera at different times. The coordinates of the key points of the hand in the two-dimensional gesture image at the corresponding time are adjusted according to the relative position relationship, and the three-dimensional coordinates of the key points of the hand at the corresponding time in the in-vehicle coordinate system are obtained, thereby determining the 3D control gesture based on the changes in the three-dimensional coordinates of the key points of the hand over a period of time.
[0053] For example, if the vehicle display screen is located directly in front of the user and the user wants to increase the distance between the vehicle display screen and himself, he can make a gesture of pushing the palm forward parallel to the plane where the vehicle display screen is located. At this time, based on the three-dimensional coordinate changes of the user's hand center point in the in-vehicle coordinate system within 2S detected and processed by the OMS camera and radar sensor, the 3D control gesture is determined to make the vehicle display screen move away from the user in a direction perpendicular to the plane where it is located.
[0054] By adopting the method of this embodiment, the existing OMS camera in the smart vehicle can be directly used to recognize 3D control gestures, and it is suitable for the case where the OMS camera is a single monocular camera. There is no need to additionally set up a gesture control area and a matching vehicle-mounted camera for detecting the gesture control area. It still supports gestures to control vehicle-mounted devices or vehicle-mounted applications while saving costs, thereby improving the interaction efficiency between users and vehicles.
[0055] In one embodiment, the target vehicle-mounted device includes a vehicle-mounted display screen with a mechanical arm. In step S130, a control instruction for the target vehicle-mounted device or the target vehicle-mounted application is generated according to the target gesture operation, including:
[0056] When the target gesture operation is to move in a first direction, generating a control instruction to control the vehicle-mounted display screen to move in the first direction through the mechanical arm;
[0057] When the target gesture operation is to rotate the target angle in the second direction, a control instruction is generated to control the vehicle-mounted display screen to rotate the target angle in the second direction through the mechanical arm.
[0058] Among them, the robotic arm can adopt a multi-degree-of-freedom vehicle-mounted digital robot to drive the vehicle-mounted display screen to complete actions in multiple degrees of freedom. The first direction can be any direction in the plane where the vehicle-mounted display screen is located, and the vehicle-mounted display screen can be controlled to move arbitrarily up, down, left, and right through target gesture operations. The first direction can also be a direction perpendicular to the plane where the vehicle-mounted display screen is located, for example, controlling the robotic arm to retract or extend to control the vehicle-mounted display screen to move in the forward and backward directions.
[0059] The second direction can be any direction in the plane where the vehicle display screen is located and a direction perpendicular to the plane where the vehicle display screen is located. The rotation direction can be clockwise or counterclockwise. The vehicle display screen can be rotated clockwise or counterclockwise according to the recognized gesture rotation angle to meet the user's viewing needs. For example, a robotic arm is used to control the vehicle display screen to flip up and down or left and right in a direction perpendicular to the plane where the display screen is located, or a robotic arm is used to control the vehicle display screen to flip clockwise or counterclockwise in the plane where it is located.
[0060] Figure 3is a schematic diagram of gesture control for a vehicle display screen according to an embodiment of the present disclosure, such as Figure 3 As shown, at least one gesture image captured by the vehicle camera can be used to identify each hand key point in each gesture image through a trained hand key point recognition model, and further combined with the coordinate information of the hand key points in the gesture images obtained at different times, the movement information of the hand in various directions within a preset time length can be determined. The xy plane can be the plane where the vehicle display screen is located, and the z-axis direction can be the depth direction of the robotic arm, so that the vehicle display screen can be moved in multiple different directions through the robotic arm. In addition, the hand rotation angle is calculated based on the palm rotation judgment algorithm, so that the vehicle display screen can be rotated clockwise or counterclockwise in the xy plane through the robotic arm.
[0061] According to the method of this embodiment, hand images at different times can be obtained through the vehicle-mounted camera to realize the control of movement or rotation in various directions of the multi-degree-of-freedom vehicle-mounted display screen without manually moving the vehicle-mounted display screen. It provides diversified control of the vehicle-mounted display screen based on gestures, enriching the means of interaction between users and vehicle-mounted devices.
[0062] In one embodiment, the gesture-based vehicle interaction method of the present disclosure further includes:
[0063] Based on a first vehicle-mounted camera among at least one vehicle-mounted camera, a gesture control wake-up operation is detected; when the gesture control wake-up operation is detected, other vehicle-mounted cameras among at least one vehicle-mounted camera are controlled to detect gestures to obtain at least one gesture image.
[0064] It is understandable that in order to more accurately determine the gesture operation, the vehicle may use multiple on-board cameras at different positions to detect the preset area, such as multiple monocular cameras set in different directions, so as to use the acquired multiple gesture images to comprehensively determine the target gesture operation. Therefore, a gesture control wake-up operation that can be recognized by a single on-board camera can be set, so that under normal conditions, only the first on-board camera is turned on to detect the gesture control wake-up operation in the preset area. When the gesture control wake-up operation is detected, other on-board cameras are controlled to detect the preset area to identify more abundant gesture control operations.
[0065] Preferably, a gesture control termination operation can also be set to control other cameras to sleep to save energy when there is no gesture control demand. The standby time can also be set. When no gesture image is detected in the preset area for longer than the standby time, other cameras are automatically put into sleep mode.
[0066] Preferably, the first vehicle-mounted camera that detects gesture control wake-up operations can be set as a monocular camera to identify simple gesture control wake-up operations. When a gesture control wake-up operation is detected, other vehicle-mounted cameras (which can be monocular or multi-cameras) are controlled to work together to detect gesture control conditions. In this way, when no user gesture is detected, only a single monocular camera with lower power consumption can be maintained in standby mode, thereby minimizing energy consumption of the vehicle-mounted gesture interaction function.
[0067] Figure 4 is a flowchart of gesture control wake-up according to an embodiment of the present disclosure, such as Figure 4 As shown, if "the hand posture within 2S is from clenched fist to open palm" is set as the gesture control wake-up operation, and "the hand posture within 2S is from open palm to clenched fist" is set as the gesture control termination operation, the first vehicle-mounted camera is used to detect the preset area, and the detected gesture image is processed by the trained hand key point recognition model. If the corresponding gesture control wake-up operation or gesture control termination operation is detected, a wake-up signal or a termination signal is generated to control other cameras to work or sleep.
[0068] According to the method of this embodiment, the gesture control mode is managed by presetting the gesture control wake-up operation, which sets the conditions for turning on the gesture control, avoids misrecognition and misoperation, and brings a better vehicle interaction experience. At the same time, the vehicle-mounted cameras other than those that recognize the gesture control wake-up operation can be put into sleep mode during the non-gesture control period, saving the vehicle's energy consumption.
[0069] Figure 5 is a logical architecture diagram of a vehicle interaction method based on gestures according to an embodiment of the present disclosure, such as Figure 5 As shown, the gesture image acquired by the vehicle camera is processed by the 3D hand key point monitoring algorithm to determine the hand key points in the gesture image, and further processed by the gesture recognition algorithm to determine the target gesture operation corresponding to one or more gesture images, and then the target vehicle application or target vehicle device corresponding to the identified target gesture operation is determined by the pre-stored gesture recognition mapper, and finally the target vehicle application or target vehicle device is controlled based on the operation content corresponding to the target gesture operation in the target vehicle application or target vehicle device. Among them, the mapping relationship between the gesture operation and the vehicle application and vehicle device in the gesture recognition mapper can be customized by the user, and the operation content corresponding to the specific gesture operation in each vehicle application and vehicle device can also be customized by the user.
[0070] It should be noted that the embodiments of the present application do not specifically limit the electrical performance, mechanical structure, etc. of the robotic arm, as long as the robotic arm can drive the movement of the vehicle-mounted screen.
[0071] The above describes the specific settings and implementation methods of the embodiments of the present application from different angles. Using the method provided in the embodiments of the present application, in addition to voice operation and touch operation, a vehicle interaction method through gesture operation can be provided, without the need for cumbersome multiple rounds of voice interaction with the vehicle, and without the need to look at the vehicle display screen, thereby improving the interaction efficiency between the user and the vehicle and ensuring the driving safety of the driver during the interaction with the vehicle; at the same time, according to the recognition of static or dynamic target gesture operations based on the key points of the hand, more abundant gesture operations can be used to control vehicle applications or vehicle devices, expanding the control scope that can be achieved by gesture operations.
[0072] As an implementation of the above methods, the embodiment of the present disclosure further provides a gesture-based vehicle interaction device, which can be applied to a vehicle and may include:
[0073] A first determination module, configured to determine a target gesture operation according to at least one gesture image;
[0074] A second determination module is used to determine a target vehicle-mounted device or a target vehicle-mounted application corresponding to a target gesture operation;
[0075] The instruction generation module is used to generate control instructions for a target vehicle-mounted device or a target vehicle-mounted application according to a target gesture operation.
[0076] In one embodiment, the first determination module is used to: detect the gesture image to obtain at least one hand key point in the gesture image;
[0077] Determine the target gesture operation based on at least one hand key point.
[0078] In one embodiment, at least one gesture image includes gesture images collected at different times, and the first determination module is used to: determine gesture change information within a preset time period based on at least one gesture image; wherein the gesture change information includes hand posture changes and / or gesture rotation angles; and determine the target gesture operation based on the gesture change information.
[0079] More preferably, the first determining module is further used for:
[0080] The coordinates of at least two first hand key points in the first gesture image and the coordinates of at least two second hand key points in the second gesture image are respectively obtained; wherein the time interval between the first gesture image and the second gesture image is a preset time length, and the positions of each first hand key point and each second hand key point on the hand correspond one-to-one; gesture change information is determined based on the coordinates of at least two first hand key points and the coordinates of at least two second hand key points.
[0081] In one embodiment, the target vehicle-mounted device includes a vehicle-mounted display screen with a mechanical arm, and the instruction generation module is used to: when the target gesture operation is to move in a first direction, generate a control instruction to control the vehicle-mounted display screen to move in the first direction through the mechanical arm;
[0082] When the target gesture operation is to rotate the target angle in the second direction, a control instruction is generated to control the vehicle-mounted display screen to rotate the target angle in the second direction through the mechanical arm.
[0083] In one implementation, the second determination module is used to determine the target vehicle-mounted device according to a pre-stored mapping relationship between the gesture operation and the vehicle-mounted device.
[0084] In one implementation, the second determination module is used to determine a current in-vehicle application on the in-vehicle display screen as a target in-vehicle application.
[0085] In one embodiment, at least one gesture image is obtained by detecting a preset gesture control area by at least one vehicle-mounted camera; or, at least one gesture image is obtained by detecting an in-vehicle personnel monitoring system OMS camera.
[0086] In one embodiment, the device further comprises:
[0087] A detection module, configured to detect a gesture control wake-up operation based on a first vehicle-mounted camera of at least one vehicle-mounted camera;
[0088] The acquisition module is used to control other vehicle-mounted cameras in at least one vehicle-mounted camera to detect gestures when a gesture control wake-up operation is detected, so as to acquire at least one gesture image.
[0089] The functions of each unit, module or sub-module in each device of the embodiments of the present disclosure can refer to the corresponding description in the above method embodiments, and have corresponding beneficial effects, which will not be repeated here.
[0090] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0091] Figure 6 FIG. 1 is a block diagram showing a structure of an electronic device according to an embodiment of the present application. Figure 6As shown, the electronic device includes: a memory 610 and a processor 620, and the memory 610 stores instructions that can be run on the processor 620. When the processor 620 executes the instruction, the method in the above embodiment is implemented. The number of the memory 610 and the processor 620 can be one or more. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.
[0092] The electronic device may also include a communication interface 630 for communicating with external devices and performing data exchange transmission. The various devices are interconnected using different buses and may be installed on a common motherboard or in other ways as needed. The processor 620 may process instructions executed within the electronic device, including instructions stored in or on a memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors and / or multiple buses may be used together with multiple memories and multiple memories. Similarly, multiple electronic devices may be connected, each device providing part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0093] Optionally, in a specific implementation, if the memory 610, the processor 620 and the communication interface 630 are integrated on a chip, the memory 610, the processor 620 and the communication interface 630 can communicate with each other through an internal interface.
[0094] It should be understood that the above processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It is worth noting that the processor can be a processor that supports the Advanced RISC Machine (ARM) architecture.
[0095] An embodiment of the present application provides a computer-readable storage medium (such as the memory 610 mentioned above), which stores computer instructions. When the program is executed by a processor, the method provided in the embodiment of the present application is implemented.
[0096] Optionally, the memory 610 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the electronic device for identifying lane edges, etc. In addition, the memory 610 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 610 may optionally include a memory generated remotely relative to the processor 620, and these remote memories may be connected to the electronic device for identifying lane edges via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0097] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other physical types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage media or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory media such as modulated data signals and carrier waves.
[0098] This embodiment also provides a vehicle, including a controller, which can be used to execute the method of this embodiment, or the controller can include any device of this embodiment, or the controller can be any electronic device of this embodiment.
[0099] Exemplarily, the processor in the controller or electronic device may include at least one of an autonomous driving domain control module, a vehicle body domain control module, and an audio and video entertainment domain control module.
[0100] For example, the vehicle in this embodiment can be a fuel vehicle, an electric vehicle, a solar vehicle, or any other powered vehicle. For example, the vehicle in this embodiment can be an autonomous driving vehicle.
[0101] Other components of the vehicle of this embodiment, such as the specific structure of the frame and wheels and the connecting and fastening components, can adopt various technical solutions known to ordinary technicians in this field now and in the future, and will not be described in detail here.
[0102] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0103] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A vehicle interaction method based on gesture, characterized in that: include: Determine a target gesture operation according to at least one gesture image; Determining a target vehicle-mounted device or a target vehicle-mounted application corresponding to the target gesture operation; A control instruction for the target vehicle-mounted device or the target vehicle-mounted application is generated according to the target gesture operation.
2. The method according to claim 1, characterized in that Determine a target gesture operation according to at least one gesture image, including: Detecting the gesture image to obtain at least one hand key point in the gesture image; The target gesture operation is determined according to the at least one hand key point.
3. The method according to claim 1, characterized in that The at least one gesture image includes gesture images collected at different times, and determining a target gesture operation according to the at least one gesture image includes: Determine gesture change information within a preset time period according to the at least one gesture image; wherein the gesture change information includes hand posture change and / or gesture rotation angle; The target gesture operation is determined according to the gesture change information.
4. The method according to claim 3, characterized in that Determining gesture change information within a preset time period according to the at least one gesture image includes: Respectively obtaining coordinates of at least two first hand key points in the first gesture image and coordinates of at least two second hand key points in the second gesture image; wherein the time interval between the first gesture image and the second gesture image is the preset time length, and the positions of the first hand key points and the second hand key points on the hand correspond one to one; The gesture change information is determined according to the coordinates of the at least two first hand key points and the coordinates of the at least two second hand key points.
5. The method according to claim 1, characterized in that The target vehicle-mounted device includes a vehicle-mounted display screen with a mechanical arm, and generating a control instruction for the target vehicle-mounted device or the target vehicle-mounted application according to the target gesture operation, including: When the target gesture operation is moving in a first direction, a control instruction is generated to control the vehicle-mounted display screen to move in the first direction through the mechanical arm.
6. The method according to claim 5, characterized in that Generating a control instruction for the target vehicle-mounted device or the target vehicle-mounted application according to the target gesture operation, further comprising: When the target gesture operation is to rotate the target angle in the second direction, a control instruction is generated to control the vehicle-mounted display screen to rotate the target angle in the second direction through the mechanical arm.
7. The method according to claim 1, characterized in that Determining a target vehicle-mounted device corresponding to the target gesture operation includes: The target vehicle-mounted device is determined according to a pre-stored mapping relationship between the gesture operation and the vehicle-mounted device.
8. The method according to claim 1, characterized in that Determining a target in-vehicle application corresponding to the target gesture operation includes: The current in-vehicle application in the in-vehicle display screen is determined as the target in-vehicle application.
9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Detecting a gesture control wake-up operation based on a first vehicle-mounted camera of at least one vehicle-mounted camera; When the gesture control wake-up operation is detected, other vehicle-mounted cameras in the at least one vehicle-mounted camera are controlled to detect gestures to obtain the at least one gesture image.
10. The method according to any one of claims 1 to 8, characterized in that The at least one gesture image is obtained by detecting a preset gesture control area by at least one vehicle-mounted camera; or, the at least one gesture image is obtained by detecting an OMS camera of an in-vehicle personnel monitoring system.
11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.
12. A vehicle, characterized in that: The electronic device comprising claim 11.