Multi-mode-based vehicle control method and device, storage medium, equipment and vehicle

Abstract

The invention discloses a multi-mode-based vehicle control method and device, a storage medium, equipment and a vehicle, and the method comprises the steps: obtaining a human body image if a vehicle key signal is detected within a preset range threshold value, and the human body image comprises a human face image, a static gesture image and a dynamic gesture image; if the static gesture in the static gesture image is a preset wake-up gesture and the face in the face image is a face corresponding to a preset face ID, determining a dynamic gesture corresponding to the dynamic gesture image according to a preset dynamic gesture recognition model; and controlling the vehicle according to the dynamic gesture. According to the invention, when the vehicle door of the parking space cannot be opened and a vehicle owner cannot enter the vehicle conveniently to control the vehicle, interaction with the vehicle can be realized through the human body image, so that the vehicle can be controlled more conveniently, interaction with the vehicle only depending on a mobile phone APP is avoided, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a multi-modal vehicle control method, device, storage medium, equipment and vehicle. Background Art

[0002] In order to improve the comfort of vehicle riding space, the size of vehicles is also increasing. When the vehicle is parked in a parking space, the space on both sides of the door is too small due to the vehicle being too large or the parking space being too small, and the door cannot be opened, making it difficult for the driver to enter the vehicle. At present, the existing vehicle control method is to interact with the vehicle through a mobile phone application (Application, APP), control the vehicle to move forward a short distance in the mobile phone APP, and then the owner opens the door to enter the vehicle after the door can be opened.

[0003] However, the inventors discovered during the implementation of the invention that the existing vehicle control method relies on a mobile phone APP to interact with the vehicle, and cannot be used when the APP is not installed, the mobile phone is out of power, or the connection between the mobile phone and the vehicle fails, causing inconvenience to users. Summary of the invention

[0004] The purpose of this application is to overcome the shortcomings of the prior art that vehicles can only interact through mobile phone APPs, which easily brings inconvenience to users, and to provide a multi-modal vehicle control method, device, storage medium, equipment and vehicle.

[0005] The technical solution of the present application provides a multi-modal vehicle control method, including:

[0006] If the car key signal is detected within the preset range threshold, a human body image is obtained, wherein the human body image includes a face image, a static gesture image, and a dynamic gesture image;

[0007] If the static gesture in the static gesture image is a preset wake-up gesture, and the face in the face image is a face corresponding to a preset face ID, determining a dynamic gesture corresponding to the dynamic gesture image according to a preset dynamic gesture recognition model, wherein the dynamic gesture includes a follow gesture and a pause gesture;

[0008] The vehicle is controlled according to the dynamic gesture.

[0009] Further, the dynamic gesture includes a follow-up gesture, and controlling the vehicle according to the dynamic gesture includes:

[0010] If the dynamic gesture is the following gesture, the vehicle is controlled to move following the human body.

[0011] Furthermore, the human body image further includes a body image, and if the dynamic gesture is the following gesture, controlling the vehicle to follow the movement of the human body includes:

[0012] If the dynamic gesture is the follow-up gesture, acquiring the gesture three-dimensional coordinates of the dynamic gesture image and the body three-dimensional coordinates of the body image;

[0013] If the difference between the three-dimensional coordinates of the gesture and the three-dimensional coordinates of the body meets a preset difference threshold, the vehicle is controlled to follow the movement of the human body.

[0014] Furthermore, if the difference between the three-dimensional coordinates of the gesture and the three-dimensional coordinates of the body meets a preset difference threshold, controlling the vehicle to follow the movement of the human body includes:

[0015] Acquiring vehicle operation status information, the vehicle operation status information including vehicle speed, obstacle distance between the vehicle and an obstacle, and human distance between the vehicle and a human body;

[0016] If the vehicle speed meets the preset vehicle speed threshold, the distance between the obstacle and the vehicle meets the preset first vehicle distance threshold, and the distance between the human body and the vehicle meets the preset second vehicle distance threshold, the vehicle is controlled to follow the movement of the human body.

[0017] Furthermore, if the dynamic gesture is the following gesture, controlling the vehicle to follow the movement of the human body includes:

[0018] If the dynamic gesture is the follow gesture and the human body is located on the left side of the dynamic gesture image, control the vehicle to move left;

[0019] If the dynamic gesture is the follow gesture and the human body is located in the middle of the dynamic gesture image, controlling the vehicle to move in a straight line;

[0020] If the dynamic gesture is the follow gesture and the human body is located on the right side of the dynamic gesture image, the vehicle is controlled to move right.

[0021] Further, the dynamic gesture includes a pause gesture, and the controlling the vehicle according to the dynamic gesture includes:

[0022] If the dynamic gesture is the pause gesture, the vehicle is controlled to pause.

[0023] Further, controlling the vehicle according to the dynamic gesture includes:

[0024] If it is detected that the distance between the vehicle and the human body exceeds a preset second vehicle distance threshold or the human face image is not detected, the vehicle is controlled to exit following.

[0025] Further, controlling the vehicle according to the dynamic gesture includes:

[0026] If it is detected that the vehicle key signal exceeds the preset range threshold, the vehicle is controlled to be shut down.

[0027] The technical solution of the present application also provides a multi-modal vehicle control device, comprising:

[0028] An acquisition unit, configured to acquire a human image if a car key signal is detected within a preset range threshold, wherein the human image includes a face image, a static gesture image, and a dynamic gesture image;

[0029] a dynamic gesture determining unit, configured to determine, if the static gesture in the static gesture image is a preset wake-up gesture and the face in the face image is a face corresponding to a preset face ID, a dynamic gesture corresponding to the dynamic gesture image according to a preset dynamic gesture recognition model;

[0030] A control unit is used to control the vehicle according to the dynamic gesture.

[0031] The technical solution of the present application also provides a storage medium, which stores computer instructions. When a computer executes the computer instructions, it is used to execute all the steps of the multi-modal vehicle control method as described above.

[0032] The technical solution of the present application also provides an electronic device, including:

[0033] at least one processor; and,

[0034] a memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-modal vehicle control method as described above.

[0036] The technical solution of the present application also provides a vehicle, comprising: the multi-modal vehicle control device as described above; or the storage medium as described above; or the electronic device as described above.

[0037] After adopting the above technical solution, it has the following beneficial effects: by real-time detection of car key signals, if the car key signal is detected within a preset range threshold, a human body image is obtained, and the human body image includes a face image, a static gesture image and a dynamic gesture image. If the static gesture in the static gesture image is a preset wake-up gesture, and the face in the face image is the face corresponding to the preset face ID, the dynamic gesture corresponding to the dynamic gesture image is determined according to the preset dynamic gesture recognition model, and the vehicle is controlled according to the dynamic gesture, so that when the parking door cannot be opened and it is inconvenient for the owner to enter the car to control the vehicle, the human body image can be used to interact with the vehicle, thereby controlling the vehicle more conveniently, avoiding relying on the mobile phone APP to interact with the vehicle, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The disclosure of this application will become easier to understand with reference to the accompanying drawings. It should be understood that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of this application. In the drawings:

[0039] Figure 1 It is a workflow diagram of a multi-modal vehicle control method provided in one embodiment of the present application;

[0040] Figure 2 It is a work flow chart of a multi-modal vehicle control method provided by the best embodiment of the present application;

[0041] Figure 3 It is a structural schematic diagram of a multi-modal vehicle control device provided in one embodiment of the present application;

[0042] Figure 4 It is a schematic diagram of the hardware structure of an electronic device for multimodal vehicle control provided in one embodiment of the present application. DETAILED DESCRIPTION

[0043] The specific implementation of the present application is further described below with reference to the accompanying drawings.

[0044] It is easy to understand that according to the technical solution of the present application, without changing the essential spirit of the present application, a variety of structural modes and implementation modes that can be replaced by those skilled in the art can be replaced with each other. Therefore, the following specific implementation modes and drawings are only exemplary descriptions of the technical solution of the present application, and should not be regarded as the entirety of the present application or as a limitation or restriction on the technical solution of the application.

[0045] The directional terms such as up, down, left, right, front, back, front, back, top, bottom, etc. mentioned or may be mentioned in this specification are defined relative to the structures shown in the drawings. They are relative concepts and may change accordingly according to their different positions and different usage conditions. Therefore, these or other directional terms should not be interpreted as restrictive terms. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0046] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above-mentioned in this application can be understood according to the specific circumstances.

[0047] like Figure 1 As shown, Figure 1 This is a workflow diagram of a multi-modal vehicle control method provided by an embodiment of the present application, including:

[0048] Step S101: if a car key signal is detected within a preset range threshold, a human body image is acquired, the human body image includes a face image, a static gesture image and a dynamic gesture image;

[0049] Step S102: if the static gesture in the static gesture image is a preset wake-up gesture, and the face in the face image is a face corresponding to a preset face ID, determining the dynamic gesture corresponding to the dynamic gesture image according to a preset dynamic gesture recognition model;

[0050] Step S103: Control the vehicle according to the dynamic gesture.

[0051] Specifically, when the vehicle is parked in a parking space, the controller monitors the car key signal in real time. If the car key signal is detected within a preset range threshold, the controller executes step S101 to collect a human image through a camera outside the vehicle to obtain a human image, which includes a face image, a static gesture image, and a dynamic gesture image; then, step S102 is executed to input the static gesture image into a preset static gesture recognition model to identify whether the static gesture in the static gesture image is a preset wake-up gesture. A static gesture refers to gesture recognition through a single-frame image, such as a palm, fist, etc. A wake-up gesture refers to an activation gesture. Wake-up means activation, such as password unlocking of a mobile terminal. The specified password will be recognized to unlock and enter the homepage. A gesture can be specified as an activation gesture, such as setting both hands raised as an activation gesture. The static gesture determines that both palms are raised at the same height for 3 consecutive seconds. If the hand gesture is raised to a certain degree, it is considered to have entered the activated state, otherwise it will continue to detect; and determine whether the face in the acquired face image is the face corresponding to the preset face ID, that is, determine whether the face in the face image is the face corresponding to the face ID registered in the vehicle system. If the acquired dynamic gesture image is input into the preset dynamic gesture recognition model, the dynamic gesture corresponding to the dynamic gesture image is identified. The dynamic gesture includes a follow gesture and a pause gesture. At this time, the user can also be prompted to enter the follow state by flashing the car lights. Finally, the controller executes step S103 to control the vehicle according to the dynamic gesture. For example, when the dynamic gesture is a follow gesture, the controller controls the vehicle to move with the human body; when the dynamic gesture is a pause gesture, the controller controls the vehicle to pause, so that when the parking door cannot be opened and the owner is inconvenient to enter the car to control the vehicle, the vehicle can be controlled by interacting with the vehicle through the human body image.

[0052] Among them, the preset range threshold can be set according to user needs. For example, if the car key is connected to the vehicle via Bluetooth, the preset range threshold is the maximum communication range of Bluetooth.

[0053] The vehicle key signal refers to a key signal located outside the vehicle. When the vehicle key distinguishes between an in-vehicle key and an out-vehicle key, the vehicle key signal is an out-vehicle key signal.

[0054] Among them, the static gesture recognition model and the dynamic gesture recognition model are both existing technologies and do not belong to the improvement points of the present application. Their working principles are not described in detail here.

[0055] In this embodiment, by real-time detection of car key signals, if the car key signal is detected within a preset range threshold, a human body image is obtained, and the human body image includes a face image, a static gesture image and a dynamic gesture image. If the static gesture in the static gesture image is a preset wake-up gesture, and the face in the face image is a face corresponding to a preset face ID, the dynamic gesture corresponding to the dynamic gesture image is determined according to a preset dynamic gesture recognition model, and the vehicle is controlled according to the dynamic gesture, so that when the parking door cannot be opened and it is inconvenient for the owner to enter the car to control the vehicle, the human body image can be used to interact with the vehicle, thereby controlling the vehicle more conveniently, avoiding relying on the mobile phone APP to interact with the vehicle, and improving the user experience.

[0056] like Figure 2 As shown, Figure 2 It is a workflow diagram of a multi-modal vehicle control method provided by another embodiment of the present application, including:

[0057] Step S201: if a car key signal is detected within a preset range threshold, a human body image is acquired;

[0058] Step S202: determining whether the static gesture in the static gesture image is a preset wake-up gesture;

[0059] Step S203: determining whether the face in the face image is the face corresponding to the preset face ID;

[0060] Step S204: determining a dynamic gesture according to a preset dynamic gesture recognition model;

[0061] Step S205: determining whether the dynamic gesture is a follow-up gesture;

[0062] Step S206: acquiring the three-dimensional coordinates of the gesture of the dynamic gesture image and the three-dimensional coordinates of the body of the body image;

[0063] Step S207: determining whether the difference between the three-dimensional coordinates of the gesture and the three-dimensional coordinates of the body meets a preset difference threshold;

[0064] Step S208: Acquire vehicle operation status information, the vehicle operation status information including vehicle speed, obstacle distance between the vehicle and an obstacle, and human distance between the vehicle and a human body;

[0065] Step S209: determining whether the vehicle speed meets a preset vehicle speed threshold, whether the distance to the obstacle meets a preset first distance threshold, and whether the distance to the human body meets a preset second distance threshold;

[0066] Step S210: sending a first reminder signal;

[0067] Step S211: determining whether the human body is located on the left side of the dynamic gesture image;

[0068] Step S212: Control the vehicle to move left;

[0069] Step S213: determining whether the human body is located in the middle of the dynamic gesture image;

[0070] Step S214: Control the vehicle to move in a straight line;

[0071] Step S215: determining whether the human body is located on the right side of the dynamic gesture image;

[0072] Step S216: Control the vehicle to move right;

[0073] Step S217: determining whether the dynamic gesture is a pause gesture;

[0074] Step S218: issuing a second reminder signal and controlling the vehicle to pause;

[0075] Step S219: determining whether the distance between the vehicle and the human body exceeds a preset second vehicle distance threshold or whether no face image is detected;

[0076] Step S220: issuing a third reminder signal and controlling the vehicle to exit following;

[0077] Step S221: determining whether the vehicle key signal exceeds a preset range threshold;

[0078] Step S222: Control the vehicle to shut down.

[0079] Specifically, when the vehicle is parked in a parking space, the controller monitors the car key signal in real time. If the car key signal is detected within a preset range threshold, first, the controller executes step S201 to collect a human image through a camera outside the vehicle, thereby obtaining a human image;

[0080] Secondly, executing step S202 to input the static gesture image into a preset static gesture recognition model, and determining whether the static gesture in the static gesture image is a preset wake-up gesture, if yes, executing step S203, otherwise executing step S201;

[0081] In step S203, it is determined whether the face in the acquired face image is the face corresponding to the preset face ID. If yes, step S204 is executed to input the dynamic gesture image into the preset dynamic gesture recognition model to recognize the dynamic gesture corresponding to the dynamic gesture image. At this time, the user can be prompted to enter the following state by flashing the car lights, etc. Otherwise, step S201 is executed;

[0082] Next, the controller executes step S205 to determine whether the dynamic gesture is a follow-up gesture, and if so, executes step S206 , otherwise, executes step S217 ;

[0083] In step S206, the three-dimensional coordinates of the gesture of the dynamic gesture image and the three-dimensional coordinates of the body of the body image may be obtained by a time of flight (TOF) camera or a binocular camera;

[0084] Next, the controller executes step S207 to determine whether the difference between the three-dimensional coordinates of the gesture and the three-dimensional coordinates of the body meets the preset difference threshold. If yes, it means that the dynamic gesture is a valid dynamic gesture to avoid misrecognition. For example, when the follow-up gesture is set to beckoning, the dynamic gesture recognition model may misrecognize actions such as touching the hair as beckoning. By comparing the difference between the three-dimensional coordinates of the gesture and the three-dimensional coordinates of the body with the preset difference threshold, such misrecognition can be filtered out to improve the accuracy, and step S208 is executed. Otherwise, step S205 is executed.

[0085] In step S208, the vehicle operation status information may be obtained by radar or other means, and the vehicle operation status information includes vehicle speed, obstacle distance between the vehicle and an obstacle, and human distance between the vehicle and a human body;

[0086] Next, execute step S209 to determine whether the vehicle speed meets the preset vehicle speed threshold, whether the distance between the obstacle and the vehicle meets the preset first distance threshold, and whether the distance between the human body and the vehicle meets the preset second distance threshold, if yes, execute step S210, otherwise execute step S205;

[0087] In step S210, the vehicle lights may flash and emit a beeping sound to remind the user;

[0088] Next, execute step S211 to determine whether the human body is located on the left side of the dynamic gesture image, if yes, execute step S212 and step S219, otherwise, execute step S213;

[0089] In step S213, it is determined whether the human body is located in the middle of the dynamic gesture image, if yes, step S214 and step S219 are executed, otherwise, step S215 is executed;

[0090] In step S25, it is determined whether the human body is located on the right side of the dynamic gesture image, if yes, step S216 and step S219 are executed, otherwise, step S211 is executed;

[0091] In step S217, determine whether the dynamic gesture is a pause gesture, if so, execute steps S218 and S219, otherwise execute step S205;

[0092] In step S218, when the dynamic gesture is determined to be a pause gesture, the vehicle stops but is still in an activated state and does not need a wake-up gesture. It can continue to determine whether it is a follow gesture, which facilitates the rapid start of the vehicle and can remind the user by flashing the lights and turning off the beeping sound.

[0093] In step S219, it is determined whether the distance between the vehicle and the human body exceeds a preset second vehicle distance threshold or whether no face image is detected. If yes, step S220 is executed, otherwise step S201 is executed;

[0094] In step S220, the flashing of the vehicle lights may be turned off, and the vehicle may be controlled to exit following;

[0095] Finally, step S221 is executed to determine whether the vehicle key signal exceeds the preset range threshold. If so, step S222 is executed to control the vehicle to shut down and lock the vehicle. Otherwise, step S201 is executed.

[0096] Among them, the preset difference threshold is preferably 50cm.

[0097] Among them, the vehicle speed threshold is preferably 5km / h.

[0098] The preset first vehicle distance threshold and the preset second vehicle distance threshold may be set to the same value or different values. The preset first vehicle distance threshold is preferably 2 m, and the preset second vehicle distance threshold is preferably 5 m.

[0099] Among them, the first reminder signal, the second reminder signal and the third reminder signal can be set according to user needs. The first reminder signal, the second reminder signal and the third reminder signal can be set to the same reminder signal or to different reminder signals. In order to facilitate the user to understand entering different states, the first reminder signal, the second reminder signal and the third reminder signal are set to different reminder signals. The first reminder signal is preferably the flashing of the car lights and a beeping sound, the second reminder signal is preferably the flashing of the car lights and the turning off of the beeping sound, and the third reminder signal is preferably the turning off of the flashing of the car lights.

[0100] Among them, the order between step S205-step S216 and step S217-step S218 can be interchanged or performed simultaneously. Those skilled in the art can easily understand that as long as the controller detects that the dynamic gesture is a follow gesture or a pause gesture, the corresponding steps can be executed without affecting the technical effect of the present application.

[0101] In this embodiment, by real-time detection of the car key signal, if the car key signal is detected within a preset range threshold, a human body image is obtained, the human body image includes a face image, a static gesture image and a dynamic gesture image, if the static gesture in the static gesture image is a preset wake-up gesture, and the face in the face image is a face corresponding to a preset face ID, the dynamic gesture corresponding to the dynamic gesture image is determined according to the preset dynamic gesture recognition model, and the dynamic gesture includes a follow gesture and a pause gesture, and the vehicle is controlled according to the dynamic gesture, so that when the parking door cannot be opened and the owner is inconvenient to enter the car to control the vehicle, the human body image is used to interact with the vehicle, thereby controlling the vehicle more conveniently, avoiding relying on the mobile phone APP to interact with the vehicle, and improving the user experience. At the same time, by judging whether the difference between the three-dimensional coordinates of the gesture and the three-dimensional coordinates of the body meets the preset difference threshold, misidentification is prevented and the accuracy is improved, and the user is reminded respectively through the first reminder signal, the second reminder signal and the third reminder signal, so that the user can understand the state of the vehicle, avoid panic, and improve safety.

[0102] like Figure 3 As shown, Figure 3 : is a schematic diagram of a structure of a multi-modal vehicle control device provided by an embodiment of the present application, comprising:

[0103] The acquisition unit 301 is used to acquire a human image if a car key signal is detected within a preset range threshold, where the human image includes a face image, a static gesture image, and a dynamic gesture image;

[0104] The dynamic gesture determining unit 302 is used to determine the dynamic gesture corresponding to the dynamic gesture image according to a preset dynamic gesture recognition model if the static gesture in the static gesture image is a preset wake-up gesture and the face in the face image is a face corresponding to a preset face ID;

[0105] The control unit 303 is used to control the vehicle according to dynamic gestures.

[0106] Figure 3 The multimodal vehicle control device shown can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0107] Optionally, the dynamic gesture includes a follow-up gesture, and the control unit 303 is further configured to:

[0108] If the dynamic gesture is a following gesture, the vehicle is controlled to follow the movement of the human body.

[0109] Optionally, the human body image also includes a body image, and the acquisition unit 301 is further used for:

[0110] If the dynamic gesture is a follow-up gesture, obtain the gesture three-dimensional coordinates of the dynamic gesture image and the body three-dimensional coordinates of the body image;

[0111] The control unit 303 is also used for:

[0112] If the difference between the three-dimensional coordinates of the gesture and the three-dimensional coordinates of the body meets the preset difference threshold, the vehicle is controlled to move along with the human body.

[0113] Optionally, the acquiring unit 301 is further configured to:

[0114] Acquiring vehicle operation status information, the vehicle operation status information including vehicle speed, obstacle distance between the vehicle and an obstacle, and human distance between the vehicle and a human body;

[0115] The control unit 303 is also used for:

[0116] If the vehicle speed meets the preset vehicle speed threshold, the distance to the obstacle meets the preset first vehicle distance threshold, and the distance to the human body meets the preset second vehicle distance threshold, the vehicle is controlled to follow the movement of the human body.

[0117] Optionally, the control unit 303 is further configured to:

[0118] Send out the first reminder signal.

[0119] Optionally, the control unit 303 is further configured to:

[0120] If the dynamic gesture is a follow gesture and the human body is on the left side of the dynamic gesture image, the vehicle is controlled to move left;

[0121] If the dynamic gesture is a follow gesture and the human body is in the middle of the dynamic gesture image, the vehicle is controlled to move in a straight line;

[0122] If the dynamic gesture is a follow gesture and the human body is located on the right side of the dynamic gesture image, the vehicle is controlled to move right.

[0123] Optionally, the dynamic gesture includes a pause gesture, and the control unit 303 is further configured to:

[0124] If the dynamic gesture is a pause gesture, the vehicle is controlled to pause.

[0125] Optionally, the control unit 303 is further configured to:

[0126] A second reminder signal is issued.

[0127] Optionally, the control unit 303 is further configured to:

[0128] If it is detected that the distance between the vehicle and the human body exceeds the preset second vehicle distance or the face image is not detected, the vehicle is controlled to exit following.

[0129] Optionally, the control unit 303 is further configured to:

[0130] A third reminder signal is issued.

[0131] Optionally, the control unit 303 is further configured to:

[0132] If it is detected that the vehicle key signal exceeds the preset range threshold, the vehicle is controlled to shut down.

[0133] The technical solution of the present application also provides a storage medium, which stores computer instructions. When a computer executes the computer instructions, it is used to execute all steps of the multi-modal vehicle control method in any of the aforementioned embodiments.

[0134] like Figure 4 As shown, Figure 4 : is a hardware structure diagram of an electronic device for a multi-modal vehicle control method provided by an embodiment of the present application, including:

[0135] at least one processor 401; and,

[0136] A memory 402 is communicatively connected to the at least one processor 401; wherein,

[0137] The memory 402 stores instructions that can be executed by the at least one processor 401, and the instructions are executed by the at least one processor 401 so that the at least one processor 401 can perform all steps of the multi-modal vehicle control method in any of the aforementioned method embodiments.

[0138] The electronic device is preferably an on-board electronic control unit (ECU), and further a microcontroller unit (MCU) in the on-board electronic control unit.

[0139] Figure 4 Take a processor 401 as an example:

[0140] The electronic device may further include: an input device 403 and an output device 404 .

[0141] The processor 401, the memory 402, the input device 403 and the output device 404 may be connected via a bus or other means, and the figure takes the connection via a bus as an example.

[0142] The memory 402 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the multi-modal vehicle control method in the embodiment of the present application, for example, Figure 1The processor 401 executes various functional applications and data processing by running the non-volatile software programs, instructions and modules stored in the memory 402, that is, the multi-modal vehicle control method in the above embodiment is implemented.

[0143] The memory 402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of a multimodal vehicle control method, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 402 may optionally include a memory remotely arranged relative to the processor 401, and these remote memories may be connected to a device for executing a multimodal vehicle control method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The input device 403 may receive a user click input, and generate a signal input related to user settings and function control based on a multi-modal vehicle control method. The output device 404 may include a display device such as a display screen.

[0145] The one or more modules are stored in the memory 402 , and when executed by the one or more processors 401 , the multi-modal vehicle control method in any of the above method embodiments is executed.

[0146] The technical solution of the present application also provides a vehicle, comprising: the multi-modal vehicle control device as described above; or the storage medium as described above; or the electronic device as described above.

[0147] The above description is only the principle and preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this field, the implementation methods obtained by appropriately combining the technical solutions disclosed in different embodiments are also included in the technical scope of the present invention. On the basis of the principle of the present application, several other variations can be made, which should also be regarded as the protection scope of the present application.

Claims

1. A multi-modal vehicle control method, characterized in that: include: If the car key signal is detected within the preset range threshold, a human body image is obtained, wherein the human body image includes a face image, a static gesture image, and a dynamic gesture image; If the static gesture in the static gesture image is a preset wake-up gesture, and the face in the face image is a face corresponding to a preset face ID, determining the dynamic gesture corresponding to the dynamic gesture image according to a preset dynamic gesture recognition model; The vehicle is controlled according to the dynamic gesture.

2. The multi-modal vehicle control method according to claim 1, characterized in that: The dynamic gesture includes a follow-up gesture, and controlling the vehicle according to the dynamic gesture includes: If the dynamic gesture is the following gesture, the vehicle is controlled to move following the human body.

3. The multi-modal vehicle control method according to claim 2, characterized in that: The human body image also includes a body image, and if the dynamic gesture is the following gesture, controlling the vehicle to follow the human body to move includes: If the dynamic gesture is the follow-up gesture, acquiring the gesture three-dimensional coordinates of the dynamic gesture image and the body three-dimensional coordinates of the body image; If the difference between the three-dimensional coordinates of the gesture and the three-dimensional coordinates of the body meets a preset difference threshold, the vehicle is controlled to follow the movement of the human body.

4. The multi-modal vehicle control method according to claim 3, characterized in that: If the difference between the three-dimensional coordinates of the gesture and the three-dimensional coordinates of the body meets a preset difference threshold, controlling the vehicle to follow the movement of the human body includes: Acquiring vehicle operation status information, the vehicle operation status information including vehicle speed, obstacle distance between the vehicle and an obstacle, and human distance between the vehicle and a human body; If the vehicle speed meets the preset vehicle speed threshold, the distance between the obstacle and the vehicle meets the preset first vehicle distance threshold, and the distance between the human body and the vehicle meets the preset second vehicle distance threshold, the vehicle is controlled to follow the movement of the human body.

5. The multi-modal vehicle control method according to any one of claims 2 to 4, characterized in that: If the dynamic gesture is the following gesture, controlling the vehicle to follow the movement of the human body includes: If the dynamic gesture is the follow gesture and the human body is located on the left side of the dynamic gesture image, control the vehicle to move left; If the dynamic gesture is the follow gesture and the human body is located in the middle of the dynamic gesture image, controlling the vehicle to move in a straight line; If the dynamic gesture is the follow gesture and the human body is located on the right side of the dynamic gesture image, the vehicle is controlled to move right.

6. The multi-modal vehicle control method according to claim 1, characterized in that: The dynamic gesture includes a pause gesture, and controlling the vehicle according to the dynamic gesture includes: If the dynamic gesture is the pause gesture, the vehicle is controlled to pause.

7. The multi-modal vehicle control method according to claim 1, characterized in that: The controlling the vehicle according to the dynamic gesture comprises: If it is detected that the distance between the vehicle and the human body exceeds a preset second vehicle distance threshold or the human face image is not detected, the vehicle is controlled to exit following.

8. The multi-modal vehicle control method according to claim 1, characterized in that: The controlling the vehicle according to the dynamic gesture comprises: If it is detected that the vehicle key signal exceeds the preset range threshold, the vehicle is controlled to be shut down.

9. A multi-modal vehicle control device, characterized in that: include: An acquisition unit, configured to acquire a human image if a car key signal is detected within a preset range threshold, wherein the human image includes a face image, a static gesture image, and a dynamic gesture image; a dynamic gesture determining unit, configured to determine, if the static gesture in the static gesture image is a preset wake-up gesture and the face in the face image is a face corresponding to a preset face ID, a dynamic gesture corresponding to the dynamic gesture image according to a preset dynamic gesture recognition model; A control unit is used to control the vehicle according to the dynamic gesture.

10. A storage medium, characterized in that: The storage medium stores computer instructions, and when a computer executes the computer instructions, it is used to execute all steps of the multi-modal vehicle control method as described in any one of claims 1-8.

11. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the multi-modal vehicle control method as described in any one of claims 1-8.

12. A vehicle comprising: The multi-modal vehicle control device according to claim 9; or the storage medium as claimed in claim 10; Or the electronic device as claimed in claim 11.

Images