Vehicle control method and device, new energy vehicle and storage medium

Abstract

The application relates to a vehicle control method and device, a new energy vehicle and a storage medium, and relates to the technical field of new energy vehicles. The method comprises the following steps: in the case that a target control mode of a vehicle is started, current scene information of the vehicle and an image of an object in the vehicle are acquired, and the image is analyzed to determine current action information of the object in the vehicle; based on the acquired current action information of the object in the vehicle, the current scene information of the vehicle and a pre-defined rule base, a target control instruction which matches the current scene and the current action is adaptively determined, and the vehicle is controlled to make a corresponding response based on the target control instruction. Based on the vehicle control method, a user in the vehicle does not need to perform physical button operation and touch button operation, and the convenience of vehicle control in the driving process is improved; meanwhile, since the target control instruction is determined based on the combination of the current scene of the vehicle and the current action of the object in the vehicle, the driving safety of the vehicle can also be ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of new energy vehicles, in particular to a vehicle control method and device, a new energy vehicle, a storage medium and a computer program product. BACKGROUND

[0002] In the current vehicle control system field, there are mainly several control technologies:

[0003] 1. Physical button control: Various physical buttons are set on the instrument panel, steering wheel or center console of the vehicle to realize the operation of vehicle functions. The layout of these physical buttons is fixed, and the driver needs to be familiar with the location and function of each button.

[0004] 2. Touch screen control technology: More and more vehicles are equipped with touch screen control systems. The driver can operate vehicle functions by touching icons or menus on the screen. This system usually has a graphical interface and can display more information and function options, but during vehicle driving, due to factors such as jolting and shaking, the driver may make mistakes when touching the screen.

[0005] 3. Voice control technology: Using the microphone and voice recognition system in the vehicle, the driver can control vehicle functions by speaking specific instructions. However, the current voice control system has limited natural language understanding ability, and the driver must use specific, system-preset instruction formats and vocabulary, otherwise the system may not accurately understand.

[0006] 4. Simple driver monitoring system (DMS) combined with limited control functions: Some vehicles are equipped with DMS, which controls through blinking, head movements, etc., but the recognition accuracy is poor and is easily disturbed by normal physiological movements of the driver (such as normal blinking, slight head shaking, etc.), leading to misjudgment.

[0007] In summary, the current vehicle control system cannot meet the growing demand for intelligent driving. SUMMARY

[0008] Therefore, it is necessary to provide a vehicle control method, device, new energy vehicle, computer-readable storage medium and computer program product to solve the above technical problems.

[0009] In a first aspect, the present application provides a vehicle control method, comprising:

[0010] In the case that the target control mode of the vehicle is turned on, the current scene information of the vehicle and the image of the object in the vehicle are obtained; the current scene information includes at least one of the current time, the current position, the current vehicle state parameter and the current environment information;

[0011] determine current action information of the in-vehicle object based on the image;

[0012] determine a target control instruction based on the current action information, the current scene information, and a pre-defined rule base; the rule base stores a plurality of sets of control conditions and control instructions corresponding to each set of control conditions, and each set of control conditions is a combination of preset action and scene conditions; the information combination of the current action information and the current scene information matches the control condition corresponding to the target control instruction;

[0013] control the vehicle to make a corresponding response based on the target control instruction.

[0014] In one of the embodiments, when the target control mode of the vehicle is turned on, the current scene information of the vehicle and the image of the in-vehicle object are obtained, including:

[0015] When the target control mode of the vehicle is turned on, the current scene information of the vehicle is obtained.

[0016] determine the frame rate and / or resolution of the driver monitoring system of the vehicle based on the current scene information;

[0017] control the driver monitoring system to obtain the image of the in-vehicle object based on the frame rate and / or resolution; the image is an image containing hand information of the in-vehicle object.

[0018] In one of the embodiments, the determination of the frame rate and / or resolution of the driver monitoring system based on the current scene information includes:

[0019] determine the scene complexity corresponding to the current scene information based on a pre-set scene complexity distinguishing standard;

[0020] obtain the frame rate and / or resolution corresponding to the scene complexity as the frame rate and / or resolution of the driver monitoring system based on a pre-set mapping table; in the mapping table, the scene complexity and the frame rate and / or resolution are in a positive correlation.

[0021] In one of the embodiments, the image is an image containing hand information of the in-vehicle object;

[0022] The determination of the current action information of the in-vehicle object based on the image includes:

[0023] perform grayscale processing on the image to obtain a grayscale image, aiming to reduce the amount of image data and retain the hand information therein;

[0024] perform background reduction processing on the grayscale image to obtain a background-processed image, taking the area where the hand information is located as the foreground.

[0025] identify hand key points based on the processed images of the at least two continuous frames, and determine current hand action information of the in-vehicle object based on the hand key points; the hand key points include finger key points, palm key points, and wrist key points.

[0026] In one of the embodiments, each set of control conditions in the rule base is a combination condition of a preset gesture action and a scene condition.

[0027] The determination of the target control instruction based on the current action information, the current scene information, and the predefined rule base includes:

[0028] matching the current scene information with the scene conditions included in each set of control conditions in the rule base to determine candidate scene conditions matched with the current scene information;

[0029] comparing the current hand action information with a preset gesture action corresponding to the candidate scene conditions;

[0030] In the case where the current hand action information belongs to the preset gesture action corresponding to the candidate scene conditions, a control instruction corresponding to a control condition combined by the candidate scene conditions and the preset gesture action is obtained as the target control instruction.

[0031] In one of the embodiments, the control of the vehicle to make a corresponding response based on the target control instruction includes:

[0032] determining a constraint condition corresponding to the target control instruction; the constraint condition includes at least one of a vehicle speed constraint condition and an interval duration constraint of the last execution of the same type of control instruction;

[0033] If there is no corresponding situation of any of the constraint conditions at present, the vehicle is controlled to make a corresponding response based on the target control instruction; otherwise, the target control instruction is not executed.

[0034] In one of the embodiments, the method further includes:

[0035] Within a set time after the control of the vehicle to make a corresponding response based on the target control instruction, if an adjustment operation of the in-vehicle object to the response is obtained, the number of adjustments corresponding to the target control instruction is recorded.

[0036] In the case where the number of adjustments is greater than a set number threshold, update information corresponding to the target control instruction is determined based on the target control instruction and the adjustment operation.

[0037] The information corresponding to the target control instruction in the rule base is updated based on the update information.

[0038] In a second aspect, the present application provides a vehicle control device, comprising:

[0039] an information acquisition module configured to acquire current scene information of the vehicle and an image of an object in the vehicle when a target control mode of the vehicle is turned on, wherein the current scene information comprises at least one of current time, current position, current vehicle state parameter and current environment information;

[0040] an image recognition module configured to determine current action information of the object in the vehicle based on the image;

[0041] an instruction determination module configured to determine a target control instruction based on the current action information, the current scene information and a pre-defined rule base, wherein the rule base stores a plurality of sets of control conditions and control instructions corresponding to each set of control conditions, each set of the control conditions is a combination of action condition and scene condition, and an information combination of the current action information and the current scene information matches the control condition corresponding to the target control instruction;

[0042] an instruction execution module configured to control the vehicle to make a corresponding response based on the target control instruction.

[0043] In a third aspect, the present application provides a new energy vehicle, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method according to any one of the above aspects.

[0044] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method according to any one of the above aspects.

[0045] In a fifth aspect, the present application provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of the method according to any one of the above aspects.

[0046] The vehicle control method, device, new energy vehicle, storage medium and computer program product improve the convenience of vehicle control in the driving process, without the need for physical button operation and touch button operation by the user in the vehicle. Meanwhile, the target control instruction is determined based on the combination of the current scene of the vehicle and the current action of the object in the vehicle, and is a control condition already existing in the predefined rule base, thereby improving the convenience and intelligent degree of operation, and also helping to ensure the driving safety of the vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0048] Figure 1 An application environment diagram of the vehicle control method in an embodiment;

[0049] Figure 2 A flowchart of the vehicle control method in an embodiment;

[0050] Figure 3 A flowchart of obtaining the current scene information of the vehicle and the image of the object in the vehicle in an embodiment;

[0051] Figure 4 A flowchart of determining the current action information of the object in the vehicle based on the image in an embodiment;

[0052] Figure 5 A flowchart of determining the target control instruction in an embodiment;

[0053] Figure 6 A structural block diagram of the vehicle control device in an embodiment;

[0054] Figure 7 A structural diagram of the new energy vehicle in an embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0056] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0057] In the field of traditional vehicle control systems, there are mainly the following control technologies:

[0058] Physical button control: Various physical buttons are set on the instrument panel, steering wheel or center console of the vehicle to realize the operation of vehicle functions, such as controlling the window lifting, adjusting the air conditioning temperature and wind speed, turning on or off the audio system, etc. However, the layout of these physical buttons is fixed and distributed in different positions in the vehicle, and the driver needs to move his line of sight away from the road, find and press the corresponding button when operating, which will cause distraction during driving and increase the risk of accidents, especially at high speed or in complex road conditions. In addition, there are many physical buttons in the vehicle with different functions, and the driver needs to spend time to familiarize and remember the location and function of each button. For the function buttons that are not often used, it may be difficult to quickly find them when needed, and the operation is not convenient.

[0059] Touch screen control technology: With the development of electronic technology, more and more vehicles are equipped with touch screen control systems. The driver can operate the vehicle functions by touching the icons or menus on the screen, and the operation is relatively intuitive. This system usually has a graphical interface that can display more information and function options. Similar to physical buttons, the driver must turn his line of sight from the front road to the screen when operating the touch screen, and even a short line of sight away may miss important information on the road at a critical moment, which poses a safety hazard. At the same time, during vehicle driving, due to factors such as bumps and shakes, the driver may make mistakes when touching the screen, especially when clicking on small icons or making fine operations. Moreover, the touch screen may not be sensitive in some cases (such as screen wet, gloves operation).

[0060] Voice control technology: Using the microphone and voice recognition system in the vehicle, the driver can control the vehicle functions by speaking specific instructions. Voice control provides a convenient way for the driver without manual operation, especially when it is inconvenient to operate with both hands during driving. Current voice control systems can recognize some basic instructions, such as navigation destination input, making a call, adjusting the volume, etc. However, in noisy environments, such as passengers talking in the car, playing music, the vehicle driving on noisy roads, etc., the accuracy of voice recognition will be greatly reduced, and external noise may also mask the driver's voice instructions, causing the system to fail to correctly recognize or recognize errors. In addition, the existing voice control system has limited understanding of natural language, and the driver must use specific, system-preset instruction formats and vocabulary, otherwise the system may not accurately understand, and drivers with different accents and regions may encounter difficulties in instruction recognition.

[0061] Other control technologies: For example, some vehicles are equipped with DMS, which mainly monitors the driver's facial expressions, eye states, etc. to determine whether the driver is tired or distracted. Some DMS systems have begun to try to combine some simple control functions, such as implementing certain specific functions through blinking or head movements, but these functions are often limited and not accurate enough. However, controlling the vehicle through blinking, head movements, etc. is easily disturbed by the driver's normal physiological movements (such as normal blinking, slight head shaking, etc.), leading to misjudgment; at the same time, the functions realized by this combination are very limited and cannot meet the needs of the driver for diversified function control of the vehicle in different scenarios, lacking flexibility and scalability.

[0062] As can be seen, the above-mentioned several vehicle control technologies have obvious shortcomings in driving safety, operation convenience, function diversity and system integration, etc., and cannot meet the growing demand for intelligent driving and the expectations of drivers for convenient and safe operation of vehicles.

[0063] Therefore, based on the intelligent vehicle control system, the vehicle control method provided by the present application, under the condition that the vehicle opens the corresponding control mode, acquires the current scene information of the vehicle and the image of the object in the vehicle, determines the current action information of the object in the vehicle based on the image of the object in the vehicle, and matches the pre-defined rule library combining the current action information and the current scene information. The rule library saves multiple sets of control conditions and control instructions corresponding to each set of control conditions, and each set of control conditions is a combination of preset actions and scene conditions, so as to determine the target control instruction corresponding to the current action information and the current scene information. Based on the target control instruction, the vehicle is controlled to make a corresponding response, and through this scheme, the intelligent and personalized precise control of multiple operation functions of the vehicle under various scene conditions can be realized, providing a safer, more convenient and intelligent vehicle control solution for the object in the vehicle.

[0064] The vehicle control method provided by the embodiments of the present application can be applied to the application environment as shown in Figure 1 The vehicle includes a cabin software system 101, a telematics box (T-BOX) 102, a vehicle controller 103, a driver monitoring system (DMS) 104, a power battery system 105, a motor and a motor controller 106, which are connected through a bus 100.

[0065] The cabin software system 101 can be used to control the user interface in the vehicle, including the instrument panel, touch screen, entertainment system and navigation system, to provide interactive functions for the driver and passengers.

[0066] The T-BOX 102, as a remote information processing unit of the vehicle, can connect the vehicle and the cloud through a communication network, and is responsible for collecting, uploading and remotely controlling vehicle data.

[0067] The vehicle controller 103, as the brain of a new energy vehicle, is responsible for coordinating and controlling the operation of components such as the battery, motor, motor controller and thermal management system, to realize intelligent control of vehicle power distribution, energy management and vehicle functions.

[0068] The driver monitoring system (DMS) 104 is a system that monitors the driver's state in real time through cameras and sensors, such as eye movement, facial expression and driving behavior, to detect risks such as fatigue driving and distracted driving and provide timely reminders to improve driving safety.

[0069] The power battery system 105 is mainly used to provide the main power source for the vehicle and to provide power for the motor during driving.

[0070] The motor and motor controller 106, where the motor is the core component that converts electrical energy into mechanical energy, provides power during vehicle driving, and the motor controller is responsible for controlling the speed and torque output of the motor. It receives instructions from the vehicle control system to adjust the working state of the motor, so that the motor can realize the functions of acceleration, deceleration and braking. Specifically, the motor controller can change the working state of the motor by adjusting the current and voltage to determine whether the motor is driving the vehicle forward or performing energy recovery.

[0071] In an exemplary embodiment, as shown in Figure 2 A vehicle control method is provided, and the vehicle controller of the vehicle in Figure 1 is taken as an example to illustrate the method, and in other embodiments, the method can also be implemented by the vehicle controller and other controllers or control systems (such as DMS). The method includes the following steps S201 to S203. Wherein:

[0072] In step S201, in a case where a target control mode of the vehicle is started, current scene information of the vehicle and an image of an in-vehicle object are acquired, and current action information of the in-vehicle object is determined based on the acquired image of the in-vehicle object.

[0073] The in-vehicle object can be a driver or a passenger at a preset position in the vehicle, such as a co-driver. The target control mode can be a pre-configured intelligent control mode, in which relevant hardware and / or software systems in the vehicle are activated, the vehicle scene and the action of the in-vehicle object are collected and recognized based on the hardware and / or software systems, and a function of adaptively matching vehicle control instructions based on the recognized scene information and the action information of the in-vehicle object is enabled. In some embodiments, the target control mode can be an update based on an existing control mode of the vehicle, that is, the target control mode is integrated into the existing control mode. In other embodiments, the target control mode can also be an independent control mode in the vehicle. The in-vehicle object can start or stop the target control mode through a physical button, a touch button, or a voice instruction in the vehicle.

[0074] The acquired current scene information includes at least one of current time, current vehicle state parameters, and current environment information. The current time can be date information, week information, and / or specific time period information. The current vehicle state parameters can be any parameter related to the state of the vehicle, including but not limited to vehicle speed state, light state, steering wheel state, door state, window state, trunk state, wiper state, etc. The current environment information can be natural environment information of the vehicle, such as weather information and light information, and can also be road condition information of the vehicle, such as highway scene, mountain road scene, congested road section scene, and parking lot scene.

[0075] In one embodiment, the current scene information of the vehicle can be acquired by one or more sensors deployed in the vehicle. For example, the current illumination information can be measured by a light sensor, the current vehicle speed information can be acquired by a speed sensor, and the vehicle-mounted radar or traffic flow information can be acquired. In other embodiments, the current scene information of the vehicle can also be acquired through the Internet of Vehicles, such as the current illumination information and traffic flow information acquired through the Internet of Vehicles. In addition, the corresponding current scene information can also be acquired by one or more sensors and the Internet of Vehicles, respectively.

[0076] For example, the image of the driver can be acquired by the vehicle-mounted camera, for example, the image of the driver can be acquired by the camera associated with the DMS system, and the current action information of the driver can be recognized based on the DMS system. Unlike the image of the driver collected by the traditional DMS system, which focuses on the face region of the driver, the image of the driver collected in the embodiment of the application includes not only the image of the face region of the driver, but also the image of the hand region of the driver or the overall image of the upper body.

[0077] It should be noted that the image of the object in the vehicle is acquired and analyzed in the application, which is collected, used and processed with the authorization of the object in the vehicle, in accordance with relevant regulations.

[0078] In some embodiments, when the target control mode of the vehicle is started, the current scene information of the vehicle and the image of the object in the vehicle can be acquired based on a preset frequency or a dynamically adaptive frequency, and the characteristics of the current scene information can be analyzed based on a preset scene analysis algorithm or model, and the current action information of the object in the vehicle can be analyzed based on the acquired image of the object in the vehicle based on a preset image recognition algorithm or model.

[0079] In step S202, the target control instruction is determined based on the determined current action information of the object in the vehicle, the current scene information of the vehicle and the pre-defined rule base. The rule base stores a plurality of control conditions and control instructions corresponding to each group of control conditions, and each group of control conditions is a combination of preset actions and scene conditions. The combination of the current action information and the current scene information matches the control condition corresponding to the target control instruction.

[0080] In the embodiment of the application, the pre-defined rule base stores a plurality of corresponding relationships among scene conditions, action conditions and vehicle control instructions. The scene condition in each corresponding relationship can be a single scene condition, for example, the illuminance is greater than 1000 lux, or a combination of multiple scene conditions, for example, highway driving scene & 6:00-22:00 & illuminance greater than 1000 lux. The action condition in each corresponding relationship can be one action condition, for example, "hand rotation", or a combination of multiple action conditions, for example, hand rotation & rotation speed greater than a set value & rotation angle greater than a set angle. The vehicle control instruction in each corresponding relationship can be one control instruction, for example, the volume reduction instruction, or multiple control instructions, for example, the volume reduction instruction & the window closing instruction.

[0081] In some embodiments, the rule base can be customized by the user based on the user's driving habits. In other embodiments, a default rule base can be pre-configured for multiple vehicles based on big data analysis. In addition, the rule base can also be dynamically updated for different vehicles based on the matching of the corresponding relationship in the rule base by different users, so as to adapt to the driving habits of the vehicle user and improve the intelligent level.

[0082] In one embodiment, after the current action information of the object in the vehicle and the current scene information of the vehicle are determined, the current action information can be matched with the pre-defined rule base first to determine the matched candidate control condition, and then the target control condition and the target control instruction corresponding to the target control condition can be further matched based on the current scene information. This way is beneficial to improve the matching efficiency for the situation that the action condition is simple and the scene condition is complex. In addition, the current scene information can be matched with the pre-defined rule base first, and then the current action information can be matched. This way is more efficient for the situation that the scene condition is simple and the action condition is complex. The current scene information and the current action information can also be matched with the pre-defined rule base at the same time to determine the target control condition and the target control instruction corresponding to the target control condition. The corresponding matching method can be determined based on the specific situation of the scene condition and the action condition in the rule, so that the corresponding target control instruction can be adaptively matched from the pre-defined rule base.

[0083] Step S203, based on the determined target control instruction, the vehicle is controlled to make a corresponding response.

[0084] The target control instruction can be a vehicle speed control instruction, a vehicle body component control instruction, or a cockpit software control instruction. The vehicle body component control instruction includes but is not limited to a vehicle light control instruction, a vehicle window control instruction, a tailgate control instruction, a wiper control instruction, an air conditioner control instruction, etc. The cockpit software control instruction includes but is not limited to a volume control instruction, a play control instruction, an application software switching instruction, etc.

[0085] In one embodiment, based on the determined target control instruction, the target control instruction can be executed immediately to control the vehicle to make a corresponding response, or the current vehicle state can be further determined to determine whether the vehicle is currently suitable for executing the target control instruction, or the safety of the vehicle after executing the target control instruction is determined, and based on the further determination result, the vehicle is controlled to make a corresponding response, thereby improving the safety of intelligent control.

[0086] Through the vehicle control method of the above embodiments of the present application, in the case where the target control mode of the vehicle is turned on, the current scene information of the vehicle and the image of the in-vehicle object are acquired, and the current action information of the in-vehicle object is determined by analyzing the image; based on the acquired current action information of the in-vehicle object, the current scene information of the vehicle, and the pre-defined rule base, a target control instruction that matches both the current scene and the current action is adaptively determined, and the vehicle is controlled to make a corresponding response based on the target control instruction, without the need for the in-vehicle user to perform physical button operation and touch button operation, thereby improving the convenience of vehicle control during driving. At the same time, since the target control instruction is determined based on the combination of the current scene of the vehicle and the current action of the in-vehicle object, and is a control condition already existing in the pre-defined rule base, on the basis of improving convenience and the degree of intelligent control, the driving safety of the vehicle is also ensured.

[0087] In one embodiment, as shown in Figure 3 the above-mentioned current scene information of the vehicle and the image of the in-vehicle object are acquired in the case where the target control mode of the vehicle is turned on, the specific implementation can include:

[0088] Step S301, in the case where the target control mode of the vehicle is turned on, the current scene information of the vehicle is acquired, and based on the acquired current scene information, the frame rate and / or resolution of the driver monitoring system of the vehicle are determined.

[0089] Among them, the driver monitoring system can contain an image acquisition device, which can be an in-vehicle camera and / or an infrared sensor, and the frame rate and / or resolution of the image acquisition device are adjustable. In one embodiment, the frame rate and / or resolution under different scenes can be pre-set and saved as a corresponding reference table, and after the acquired current scene information, based on the reference table, the frame rate and / or resolution of the image acquisition device of the driver monitoring system of the vehicle can be quickly determined.

[0090] Step S302, control the driver monitoring system to acquire the image of the in-vehicle object based on the determined frame rate and / or resolution.

[0091] Among them, the image acquired by the driver monitoring system is an image containing hand information of the in-vehicle object, in addition to which, the image of the head and shoulder of the in-vehicle object can also be included. Image processing and image recognition can be performed on the acquired multiple frames of images of the in-vehicle object to determine the hand action of the in-vehicle object, or further refer to the relative position change of the hand, head and shoulder to obtain more accurate hand action recognition results.

[0092] By the method of the embodiment, the frame rate and / or resolution of the driver monitoring system can be dynamically adjusted according to different scenes. When the current scene information is a relatively complex scene, the frame rate and / or resolution of the driver monitoring system can be increased to obtain images of the objects in the vehicle at a higher frame rate and / or resolution, thereby improving the control sensitivity of the vehicle in a complex scene. When the current scene information is a relatively simple scene, the frame rate and / or resolution of the driver monitoring system can be appropriately reduced to obtain images of the objects in the vehicle at a lower frame rate and / or resolution, thereby saving resource consumption.

[0093] In one embodiment, the specific manner of determining the frame rate and / or resolution of the driver monitoring system based on the current scene information of the vehicle can include:

[0094] Based on a preset scene complexity differentiation standard, the scene complexity corresponding to the current scene information is determined; based on a preset mapping table, the frame rate and / or resolution corresponding to the scene complexity are obtained as the frame rate and / or resolution of the driver monitoring system; in the mapping table, the scene complexity and the frame rate and / or resolution are in a positive correlation relationship.

[0095] The scene complexity differentiation standard is used to determine the complexity of the current scene information. For example, three complexity levels can be set, corresponding to high complexity, medium complexity, and low complexity, respectively; and for each complexity level, a corresponding differentiation standard is set, for example, the standard corresponding to high complexity can be that if any of the following conditions is met, the scene belongs to high complexity: urban congestion environment, red light intersection, rainy and snowy weather, dense fog weather, light intensity lower than a set threshold, icy road surface, peak travel period, etc. By matching the current scene information with the preset scene complexity differentiation standard, the scene complexity corresponding to the current scene information can be determined. In one embodiment, if there is no information corresponding to the current scene information in the preset scene complexity differentiation standard, the current scene information can be determined as a default scene complexity, for example, low complexity.

[0096] In addition, the frame rate and / or resolution corresponding to different scene complexities can also be set based on empirical data and saved as a preset mapping table. After the scene complexity of the current scene information is determined by the above-mentioned manner, the corresponding frame rate and / or resolution can be quickly determined by table lookup.

[0097] By means of the above-mentioned embodiments, in combination with the driver monitoring system arranged in the vehicle, the scene complexity corresponding to the current scene information can be determined based on the preset standard, and on this basis, the frame rate and / or resolution of the driver monitoring system adapted to the current scene information can be quickly determined by looking up the table, which is conducive to guaranteeing the response sensitivity and safety of the intelligent control of the vehicle based on the action and scene in the case of frequent switching of the scene information of the vehicle, and meanwhile, only the corresponding regulation and control of the driver monitoring system is required, and the hardware and software implementation cost is relatively low.

[0098] In one embodiment, the image of the in-vehicle object acquired in the foregoing embodiments is an image containing hand information of the in-vehicle object, and the current action information determined based on the image includes current hand action information of the in-vehicle object. In this case, as shown in the foregoing embodiments, the process of determining the current action information of the in-vehicle object based on the image can include: Figure 4

[0099] Step S401, performing a grayscale processing on the image of the in-vehicle object to obtain a grayscale image, with the purpose of reducing the amount of image data and retaining the hand information therein.

[0100] Step S402, performing a background deletion processing on the grayscale image with the region where the hand information is located in the grayscale image as the foreground, to obtain a background-processed image.

[0101] Step S403, identifying hand key points based on at least two consecutive frames of the background-processed image, and determining the current hand action information of the in-vehicle object based on the key points; wherein the hand key points include finger key points, palm key points and wrist key points, and the current hand action information includes at least one of the following: hand overall displacement information, hand overall movement speed information, local displacement information of the hand key points and local movement speed information of the hand key points. For example, the current hand action information can contain a combination of at least two of the following: hand action direction, hand action displacement, hand action speed, hand action angle, and hand action duration.

[0102] ​Based on the above embodiments, taking a driver as an example, the image acquisition process may include capturing the driver's hand movements in real time using a DMS camera at an intelligently adjustable frame rate (dynamically adjustable within the range of 10-60 frames / second depending on the complexity of the scene and the activity of the hand movements). The acquired image data is transmitted to the vehicle controller in real time for subsequent processing. Next, image preprocessing is performed, including: high-precision grayscale conversion of the image to reduce the amount of data while retaining key hand information; then, a preset adaptive background subtraction algorithm is applied. This algorithm can automatically optimize the subtraction parameters based on different driving environment lighting conditions (from high-brightness environments under strong sunlight to low-brightness environments at night without lighting) and complex backgrounds (such as interior decorations, clutter, etc.), accurately removing irrelevant background images other than the hands, and retaining only clear hand image areas for subsequent analysis.

[0103] In this application, the recognition of hand gestures based on driver images employs a multi-parameter fine-grained gesture determination method. The corresponding gesture recognition algorithm performs a comprehensive assessment based on multiple parameters, including the driver's hand displacement, velocity, trajectory, duration, finger flexion and extension, the angle of the gesture relative to the driver's body, and the hand's position in the vehicle's three-dimensional space. By analyzing omnidirectional hand information across consecutive frames, various hand parameters are precisely calculated. For example:

[0104] (1) The hand displacement, velocity, and trajectory can be calculated in the following ways:

[0105] The initial hand position is (x0, y0, z0), and the current hand position is (xt, yt, zt). The displacement calculation formula is:

[0106] in This represents the 3D displacement (in pixels) of the hand in the current frame. Assume the time interval between two frames is... The formula for calculating hand speed is:

[0107] The hand movement speed (pixels per second) is represented by this parameter. A 3D motion trajectory of the hand is constructed by recording position data across multiple consecutive frames. These parameters and trajectory information are used to accurately determine the amplitude, direction, speed, and spatial variations of the hand movements.

[0108] (2) Judgment and detailed analysis of gestures.

[0109] This application can recognize basic gestures, such as:

[0110] Wave left hand (horizontal direction): when the horizontal displacement of the hand is greater than the horizontal displacement threshold, the displacement direction is negative (leftward), the speed is greater than the speed threshold, the motion trajectory of the hand in the horizontal direction is approximately linear, and the angle between the hand gesture and the front of the driver's body is within a first range (e.g., 30 degrees), it is recognized as "wave left hand horizontally".

[0111] Wave right hand (horizontal direction): when the horizontal displacement of the hand is greater than the horizontal displacement threshold, the displacement direction is positive (rightward), the speed is greater than the speed threshold, the motion trajectory of the hand in the horizontal direction is linear, and the angle between the hand gesture and the front of the driver's body is within a second range (e.g., 32 degrees), it is recognized as "wave right hand horizontally".

[0112] Wave up hand (vertical direction): when the vertical displacement of the hand is greater than the vertical displacement threshold, the displacement direction is positive (upward), the vertical motion trajectory of the hand is linear, and the angle between the hand gesture and the front of the driver's body is within a third range (e.g., 45 degrees), it is recognized as "wave up hand vertically".

[0113] Wave down hand (vertical direction): when the vertical displacement of the hand is greater than the vertical displacement threshold, the displacement direction is negative (downward), the vertical motion trajectory of the hand is linear, and the angle between the hand gesture and the front of the driver's body is within a fourth range (e.g., 45 degrees), it is recognized as "wave down hand vertically". The displacement threshold and speed threshold in the horizontal and vertical directions can be adaptively adjusted according to different driving scenarios, driver habits, and vehicle driving states.

[0114] The application can also implement recognition of complex gestures, for example:

[0115] Rotary gesture (two-dimensional plane): when it is detected that the hand makes circular or arc motion in a two-dimensional plane with the wrist as the center in a clockwise or counterclockwise direction, the fingers maintain a specific bending and stretching state (e.g., the index finger is straight, the other fingers are slightly bent, the angle between the index finger and the palm plane reaches a set angle threshold (e.g., 15 degrees), and the rotation speed reaches a rotation speed threshold (e.g., 30 degrees per second) or more, it is recognized as a rotary gesture. Moreover, according to the rotation direction, different control instructions can be corresponded. For example, clockwise rotation and the same finger state correspond to increasing the volume or increasing the air conditioning temperature; counterclockwise rotation and the same finger state correspond to decreasing the volume or decreasing the air conditioning temperature.

[0116] Gripping gesture (three-dimensional space): By analyzing the palm area change rate, the finger bending speed, and the palm position change in three-dimensional space, the gripping gesture is determined. When the palm area is significantly reduced (by a percentage threshold, such as 30%) in a short time (such as 0.5 seconds) and the fingers are quickly bent to make a fist, and the palm center position is within a spherical range with the driver's chest as the center and a radius r (such as 0.5 meters), the "fist" operation is recognized. When the palm area increases rapidly and the fingers are straightened, and the palm position is within the above range, the "open palm" operation is recognized. The time threshold and the percentage threshold of the gripping action can be set according to the average operation speed of the driver.

[0117] Through the above embodiments, based on image preprocessing and multi-parameter hand gesture recognition, accurate recognition of the driver's hand gestures during vehicle driving can be ensured, the probability of misrecognition can be reduced, and the vehicle control safety can be improved.

[0118] In one embodiment, each set of control conditions in the aforementioned rule base is a combination of a preset gesture action and a scene condition. As shown in Figure 5 The specific way of determining the target control instruction based on the current action information of the object in the vehicle, the current scene information of the vehicle, and the pre-defined rule base can include:

[0119] In step S501, the current scene information determined in the aforementioned embodiment is matched with the scene conditions included in each set of control conditions in the rule base to determine the candidate scene conditions matched with the current scene information.

[0120] The number of candidate scene conditions matched with the current scene information can be one or more. When there is no scene condition matched with the current scene information, the number of candidate scene conditions is 0. In this case, it is determined that there is no corresponding target control instruction, and there is no need to adaptively execute the target control instruction.

[0121] In step S502, it is determined whether the current hand gesture information belongs to the preset gesture action corresponding to the candidate scene condition.

[0122] In the rule base, each candidate scene condition can correspond to multiple different preset gesture actions and correspond to different control instructions. Moreover, the current hand gesture information can match one or more preset gesture actions corresponding to a certain candidate scene condition, or can not match the preset gesture actions of any candidate scene condition. If it is the latter case, it is determined that there is no corresponding target control instruction at present, and there is no need to adaptively execute the target control instruction.

[0123] Step S503, in the case that the current hand action information belongs to any one of the preset gesture actions corresponding to the candidate scene condition, obtaining the control instruction corresponding to the combination of the candidate scene condition and the any one of the preset gesture actions as the target control instruction.

[0124] In the case that the current hand action information belongs to one preset gesture action of one candidate scene condition, the control instruction corresponding to the combination of the candidate scene condition and the preset gesture action is taken as the target control instruction. In the case that the current hand action information belongs to multiple preset gesture actions of one candidate scene condition, multiple control instructions corresponding to the combination of the candidate scene condition and each of the preset gesture actions are taken as the target control instructions.

[0125] Through the above embodiment, the current scene information is matched with the control conditions in the rule library first, and then the current hand action information is matched with the candidate control conditions. Since the scene information is simpler than the gesture information, the efficiency of obtaining the candidate control conditions is higher, and the data amount of further gesture action matching is reduced, thereby facilitating the efficiency guarantee in the process of adaptively matching the target control instruction.

[0126] In order to better illustrate the embodiments of the present application, the rule library of the present application is exemplified as follows, which is constructed in the following manner.

[0127] Firstly, multiple preset gesture actions and multiple scene conditions are determined. The multiple preset gesture actions include hand gestures and finger gestures. Specifically, each preset gesture action includes at least two of the moving direction, the moving speed, the moving angle, and the moving time, so as to more accurately identify whether the current gesture of the driver is the preset gesture and reduce misidentification. The multiple scene conditions include at least two of the road condition, the time condition, the vehicle state condition, and the natural environment condition, so as to more accurately identify whether the current scene information is the preset scene condition and reduce misidentification.

[0128] In order to facilitate understanding, the preset gesture actions and the scene conditions are exemplified as follows.

[0129] The preset gesture actions include basic gestures, for example:

[0130] (1) waving the hand to the left (horizontal direction). The definition of the gesture action can include that when the horizontal displacement of the hand is greater than a horizontal displacement threshold, the displacement direction is negative (moving to the left), the moving speed is greater than a speed threshold, the motion trajectory of the hand in the horizontal direction is approximately linear, and the angle between the gesture and the front of the driver's body is within a range (such as 30 degrees), it is determined as "waving the hand to the left".

[0131] (2) Wave right hand (horizontal direction), the gesture action definition can be: when the horizontal displacement of the hand > horizontal displacement threshold, and the displacement direction is positive (rightward movement), the speed > speed threshold, the motion trajectory of the hand in the horizontal direction is a straight line, the angle between the gesture and the front of the driver's body is within (such as 30 degrees), it is determined that the "right horizontal wave" is determined.

[0132] (3) Wave up hand (vertical direction), the gesture action definition can be: when the vertical displacement of the hand > vertical displacement threshold, and the displacement direction is positive (upward movement), the vertical motion trajectory of the hand is a straight line, the angle between the gesture and the front of the driver's body is within (such as 45 degrees), it is determined that the "upward vertical wave" is determined.

[0133] (4) Wave down hand (vertical direction), the gesture action definition can be: when the vertical displacement of the hand > vertical displacement threshold, and the displacement direction is negative (downward movement), the vertical motion trajectory of the hand is a straight line, the angle between the gesture and the front of the driver's body is within (such as 45 degrees), it is determined that the "downward vertical wave" is determined.

[0134] Among them, the displacement threshold and the speed threshold can be adaptively adjusted according to different driving scenes, driver habits and vehicle driving states.

[0135] The preset gesture action can also include complex gestures, for example:

[0136] (5) Rotate gesture (two-dimensional plane), the gesture action definition can be: when it is detected that the hand makes circular or arc motion in a two-dimensional plane with the wrist as the center in a clockwise or counterclockwise direction, the fingers maintain a specific bending and stretching state (such as the index finger is straight, the other fingers are slightly bent, the angle between the index finger and the palm plane reaches a set angle threshold (such as 15 degrees), and different operations are determined according to the rotation direction. For example, when the index finger is straight and the other fingers are slightly bent, the rotation direction is clockwise, and the rotation angle is greater than (such as 90 degrees), it is determined that the volume is increased or the air conditioning temperature is increased. When the index finger is straight and the other fingers are slightly bent, the rotation direction is counterclockwise, and the rotation angle is greater than (such as 90 degrees), it is determined that the volume is decreased or the air conditioning temperature is decreased. Only when the rotation speed is greater than the rotation speed threshold (such as 30 degrees per second) and the rotation angle is greater than (such as 90 degrees) does the operation trigger.

[0137] (6) Grasp gesture (three-dimensional space), the gesture action definition can be: by analyzing the palm area change rate, the bending speed of the fingers and the position change of the palm in the three-dimensional space to judge the grasp gesture. When the palm area significantly decreases (the decrease ratio is greater than a set proportion threshold (such as 30%)) in a short time (for example, 0.5 seconds) and the fingers quickly bend to make a fist, the center position of the palm is within a spherical range with the driver's chest as the center and a radius r (such as 0.5 meters), it is determined that the "fist" operation is determined; when the palm area rapidly increases and the fingers are straight, the palm position is within the above range, it is determined that the "open palm" operation is determined.

[0138] Then, after determining the scene conditions and preset gesture actions, one or more preset gesture actions corresponding to various scene conditions can be configured respectively, thereby obtaining one or more sets of control conditions and establishing a mapping relationship between each set of control conditions and the corresponding control instructions. Further, based on the multiple sets of control conditions and the mapping relationship corresponding to each set of control conditions, the rule base is constructed.

[0139] As a specific example, the content of the rule base can include the following scene conditions, preset gesture actions, and control instructions.

[0140] Scene condition 1, highway driving scene (6:00 - 22:00), the preset gesture actions and control instructions corresponding to this scene condition 1 include: ① Left horizontal hand waving: Turn on the left turn signal. ② Right horizontal hand waving: Turn on the right turn signal. ③ Downward vertical hand waving: Reduce the vehicle speed by 5 km / h. ④ Clockwise rotating fingers (straight index finger): Fine-tune the air conditioning temperature (increase by 1.C each time) or the volume (increase by 2 bars each time).

[0141] Scene condition 2, urban congestion road condition scene (7:00 - 9:00, 17:00 - 19:00), the preset gesture actions and control instructions corresponding to this scene condition 2 include: ① Upward vertical hand waving: Start the automatic following vehicle function (when the vehicle has this function). ② Fists: Emergency brake assist (increase braking force based on normal brake system). ③ Left horizontal hand waving + Downward vertical hand waving: Switch the video recording mode of the driving recorder.

[0142] Scene condition 3, night mountain driving scene (19:00 - 6:00), the preset gesture actions and control instructions corresponding to this scene condition 3 include: ① Right horizontal hand waving: Turn on the high beam. ② Clockwise rotating fingers (straight index finger): Switch the fog lamp mode.

[0143] Scene condition 4, parking lot and vehicle speed less than a set threshold operation scene (no specific time limit), the preset gesture actions and control instructions corresponding to this scene condition 4 include: ① Open palm: Open the trunk. ② Downward vertical hand waving (small amplitude): Start the reversing radar.

[0144] In the above embodiments of the present application, the driver can also redefine the control instructions corresponding to various hand gestures in specific scenarios. For example, when driving on urban roads, the driver can remap "vertical downward waving of hands" to opening the car window for ventilation, or when driving on the highway, "clenching of fists" can be remapped to starting the cruise control function. In other embodiments, the preset gesture action can also be a combination of gestures, such as setting the combination gesture of "vertical upward waving of hands + horizontal right waving of hands + clockwise rotation of fingers (straight index finger)" to correspond to the control instruction of searching and navigating to a preset common destination (such as home address or work unit) in the navigation system, and the action requirements corresponding to the combination gesture can be defined as an action time interval of 1 second, and the action amplitude and speed of each gesture need to meet the basic requirements in the corresponding scenario. In addition, the driver can set that the self-defined gesture combination can only take effect in a specific time period (such as working hours on weekdays) or a specific geofence area (such as within 1 km around the company), to increase the flexibility and safety of operation; more complex function customization can also be performed in combination with environmental conditions and preset gesture actions. For example, when the vehicle is driving in the rain (judged by the vehicle-mounted rain sensor that the rainfall is greater than a certain value, such as more than 2 mm per minute), the driver can map the gesture action of "counterclockwise rotation of fingers (straight index finger) + vertical downward waving of hands" to the control instruction "turn on the windshield wiper and adjust to the fast wiper mode"; when the temperature in the vehicle is higher than a set value (judged by the vehicle temperature sensor) and the vehicle is in a parked state, the gesture action of "clenching of fists + horizontal left waving of hands" is mapped to the control instruction "start the vehicle ventilation system and set to the maximum ventilation mode".

[0145] Through the rule base construction method of the above embodiments, the precise position of the vehicle (through high-precision GPS positioning and map data), time, environmental state (including light, weather, temperature, road conditions, surrounding vehicle conditions, etc.), and complex features of gestures (combination, angle, position, speed, duration, etc.) are fully combined to achieve deep programming and personalized adjustment of multiple scenarios and multiple functions. Based on the rule base, deep programming of multiple scenario conditions, multiple preset gesture actions, and multiple control instructions can be achieved to meet the complex needs of different users in multiple scenarios.

[0146] In one embodiment, after determining the target control instruction based on the above embodiments, the specific case of controlling the vehicle to make a corresponding response based on the target control instruction can include:

[0147] determining a constraint condition corresponding to the target control instruction; the constraint condition includes at least one of a vehicle speed constraint condition and an interval duration constraint of the last execution of the same type of control instruction; if there is no situation corresponding to the constraint condition at present, controlling the vehicle to make a corresponding response based on the target control instruction; otherwise, not executing the target control instruction.

[0148] The vehicle control method based on the embodiments of the present application reduces misjudgment. On the one hand, multiple parameter thresholds such as speed, displacement, and angle of gesture operation are set. When the detected gesture action does not completely meet the set conditions, it is not determined to match the preset gesture in the rule base, and thus the related control instruction is not triggered. On the other hand, corresponding constraint conditions can also be set for the target control instruction. Before the target control instruction is matched and executed, it is also necessary to determine whether the corresponding constraint condition exists. If it exists, the target control instruction is not executed, so as to ensure the safety of the vehicle and the driving experience.

[0149] In one embodiment, the constraint conditions of the operation instructions corresponding to different control conditions (scene conditions + preset gesture actions) can also be different. For example:

[0150] The above scene condition 1 is a highway driving scene (6:00-22:00):

[0151] Left horizontal waving hand: turn on the left turn signal. The corresponding constraint condition can be that when the horizontal displacement of the hand is greater than 0.3 meters (the speed is greater than 5 meters / second), the gesture and the front of the driver's body form an angle within the first range, it is recognized as matching the preset gesture action of waving the hand to the left.

[0152] Right horizontal waving hand: turn on the right turn signal. The corresponding constraint condition can be that when the horizontal displacement of the hand is greater than 3 meters (the displacement speed is greater than 5 meters / second), the gesture and the front of the driver's body form an angle within the second range, it is recognized as matching the preset gesture action of waving the hand to the right.

[0153] Downward vertical waving hand: reduce the vehicle speed by 5 km / h. The corresponding constraint condition can be that when the vertical displacement is greater than 0.5 meters, the gesture and the front of the driver's body form an angle within the third range, it is recognized as matching the preset gesture action of waving the hand downward.

[0154] Clockwise rotating finger (straightening the index finger): fine tuning the air conditioning temperature (increasing by 1.C each time) or the volume (increasing by 2 steps each time). The corresponding constraint condition can be that when the rotation speed is above 45 degrees per second and the rotation angle is greater than the set angle threshold, it is recognized as matching the preset gesture action of rotating the finger clockwise.

[0155] Under the control condition of the above scene condition 1 and the preset gesture action combination, the accident avoidance measures can also include, for example, the speed adjustment operation has a minimum speed limit (cannot be lower than 60 km / h), and the operation has a certain time interval limit (such as an interval of 2 seconds between two speed reduction operations).

[0156] Scenario condition 2, urban traffic congestion scenario (7:00 - 9:00, 17:00 - 19:00):

[0157] Upward vertical waving: start the automatic following function (the vehicle has this function). The corresponding constraint conditions can be that the vertical displacement is greater than 0.1 meters, the speed is about 0.2 meters per second, and the angle between the gesture and the front of the driver's body is within the fourth range, and only then is it recognized as matching the preset gesture action of upward vertical waving.

[0158] Fist: emergency brake assist (increase brake force on the basis of normal brake system). The corresponding constraint conditions can be that the palm center position is within a spherical range with the driver's chest as the center and a radius of 0.4 meters, and the gripping action is completed in less than 0.4 seconds, and only then is it recognized as matching the preset gesture action of fist.

[0159] Left horizontal waving + downward vertical waving: switch the video recording mode of the driving recorder. The corresponding constraint conditions can be that the action sequence and time interval are within 0.5 seconds, the horizontal displacement is >0.2 meters (speed >0.2 meters / second), and the vertical displacement is >0.1 meters, and only then is it recognized as matching the preset gesture action.

[0160] Under the control conditions of scenario condition 2 and gesture action combination, the misjudgment processing and accident avoidance measures can also include: the fist gesture needs to meet multiple consecutive frames, and only then is it recognized as matching the preset gesture action. At the same time, the automatic following start operation has a speed limit (below 30 km / h) to prevent misstart when driving at high speed.

[0161] Scenario condition 3, night driving in mountainous areas (19:00 - 6:00):

[0162] Right horizontal waving: turn on high beam. The corresponding constraint conditions can be that the hand needs to maintain the waving action for more than 0.8 seconds, the speed is more than 0.3 meters per second, the angle between the gesture and the front of the driver's body is within the range, and the hand position is within the range of 0.3-0.6 meters from the front of the driver's body, and only then is it recognized as matching the preset gesture action.

[0163] Clockwise rotating fingers (straight index finger): switch fog lamp mode. The corresponding constraint conditions can be that the rotation angle needs to be greater than 180 degrees, and the speed is more than 20 meters per second, and only then is it recognized as matching the preset gesture action.

[0164] In the control condition of the above scenario condition 3 and the preset gesture action combination, the misjudgment processing and the accident avoidance measure can further include that the high beam and the fog lamp operation have strict gesture duration and speed requirements to avoid triggering due to a short hand motion. At the same time, the system detects the current light state of the vehicle, and if it is already in the corresponding light on state, it will not be triggered repeatedly to prevent interference to other drivers.

[0165] The above scenario condition 4 is a parking lot operation scenario (no specific time limit):

[0166] Open palm: open trunk. The corresponding constraint condition can be that the distance is within 0.2-0.5 meters, and the palm area increases by more than 50%, which is recognized as matching the preset gesture action.

[0167] Downward vertical hand wave (small amplitude): start the reversing radar. The corresponding constraint condition can be that the vertical displacement is between 0.05-0.1 meters, the speed is about 0.1 meters per second, the gesture and the driver's body are within the range, and the gesture is recognized as matching the preset gesture action.

[0168] In the control condition of the above scenario condition 4 and the preset gesture action combination, the misjudgment processing and the accident avoidance measure can further include that for the trunk opening operation, the palm position and area change have strict requirements to prevent the trunk from being opened by mistake during other operations. The reversing radar starting operation is only effective when the vehicle speed is very low (less than 5 km / h) to avoid misstarting during normal driving.

[0169] Through the above embodiments, the constraint condition corresponding to the target control instruction is added, that is, some avoidance measures are added to avoid potential dangers caused by automatic execution of the control instruction, further ensuring the safety of the vehicle.

[0170] In one embodiment, after controlling the vehicle to make a corresponding response based on the determined target control instruction, if an adjustment operation of the object in the vehicle is obtained for the response within a set time, the number of adjustments corresponding to the current target control instruction is recorded; in the case where the number of recorded adjustments is greater than a set number threshold, based on the target control instruction and the corresponding adjustment operation, update information corresponding to the target control instruction is determined, and the information corresponding to the target control instruction in the rule library is updated based on the update information. The target control instruction itself can be updated, or the control condition corresponding to the target control instruction can be updated.

[0171] As an example, based on the target control instruction and the corresponding adjustment operation, the update information corresponding to the target control instruction can be determined through machine learning or large model, for example, in the case of callback to the pre-response state, it indicates that the corresponding relationship between the scenes, gestures and control instructions in the rule library has not been suitable for the operation habits of the current driver, therefore the control condition corresponding to the information combination of the current action information and the current scene information in the rule library can be deleted; in the case of adjustment to other state, it indicates that the related control instruction in the rule library does not meet the expectation of the current driver, therefore the target control instruction itself in the rule library can be updated.

[0172] Through the above embodiments, an adaptive learning mechanism is introduced, based on the actual operation feedback of the driver, the rules in the rule library are continuously optimized, based on the continuously updated rule library, the adaptability, intelligent level and user experience of the system can be continuously improved.

[0173] In summary, the technical solutions obtained by the above embodiments and combinations of embodiments at least bring the following technical effects:

[0174] All-around multi-parameter gesture recognition technology: deeply integrates the three-dimensional displacement, speed, trajectory, duration, finger bending and stretching state, angle of gesture relative to the driver's body, and position of the hand in the three-dimensional space in the vehicle of the driver's hand, and uses algorithms to achieve high-precision and high-confidence gesture recognition, which can accurately distinguish various complex gestures and action details, and provides a basis for the adaptation of multi-functional and fine control instructions of the vehicle.

[0175] Multi-scene and multi-functional deep rule library arrangement: combined with the accurate position of the vehicle (through high-precision GPS positioning and map data), time, environmental state (including light, weather, temperature, road conditions, surrounding vehicle conditions, etc.) and complex features of gestures (combination, angle, position, speed, duration, etc.), it can meet the needs of multiple scenes and multiple users.

[0176] Dynamic resource allocation and adaptive learning optimization mechanism: introduces dynamic frame rate, resolution and gesture recognition sensitivity adjustment technology, optimizes system resource utilization according to hand action, scene complexity and recognition confidence. At the same time, the system has the ability to adaptively learn the operation habits of the driver, collects and analyzes the operation data of the driver, continuously optimizes the gesture recognition algorithm and the rule library, and continuously improves the system performance.

[0177] Complex gesture combination and environmental combination as control condition: supports combination operation of multiple complex gestures, and associates gestures with rich environmental conditions to realize more complex and more practical driving demand-oriented vehicle function control. It breaks through the limitations of traditional single gesture control, and provides the driver with more rich operation options and more intelligent interactive experience.

[0178] Seamless integration technology with existing vehicle systems: It can be seamlessly integrated with existing driver monitoring systems (DMS), vehicle electronic control units (ECUs), various sensors (such as light sensors, speed sensors, rain sensors, etc.), and other vehicle-mounted devices (such as navigation systems, audio systems, air conditioning systems, lighting systems, etc.). Through standardized interfaces and communication protocols, data exchange between systems is smooth, making full use of existing hardware resources, avoiding additional hardware investment and compatibility issues, and reducing system development and deployment costs. At the same time, this integration method ensures that new interactive control functions are expanded without affecting the original vehicle functions, improving the overall performance and value of the entire vehicle system.

[0179] It should be understood that although each step in the flowchart involved in the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0180] Based on the same inventive concept, the embodiments of the present application also provide a vehicle control device for implementing the above-mentioned vehicle control method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more vehicle control device embodiments provided below can refer to the limitations of the vehicle control method in the above text, which will not be repeated here.

[0181] In one exemplary embodiment, as shown in Figure 6 a vehicle control device is provided, comprising:

[0182] The information acquisition module 601 is configured to acquire current scene information of the vehicle and an image of an object in the vehicle when a target control mode of the vehicle is turned on, wherein the current scene information includes at least one of current time, current position, current vehicle state parameter, and current environment information.

[0183] The image recognition module 602 is configured to determine current action information of the object in the vehicle based on the image.

[0184] The instruction determination module 603 is configured to determine a target control instruction based on the current action information, the current scene information, and a pre-defined rule base. The rule base stores a plurality of sets of control conditions and control instructions corresponding to each set of control conditions, and each set of control conditions is a combination of action conditions and scene conditions. The combination of the current action information and the current scene information matches the control condition corresponding to the target control instruction.

[0185] The instruction execution module 604 is configured to control the vehicle to make a corresponding response based on the target control instruction.

[0186] The vehicle control device of the above embodiment, when the target control mode of the vehicle is turned on, acquires the current scene information of the vehicle and the image of the in-vehicle object through the information acquisition module, and analyzes the image to determine the current action information of the in-vehicle object through the image recognition module. The instruction determination module adaptively determines the target control instruction that matches the current scene and the current action based on the acquired current action information of the in-vehicle object, the current scene information of the vehicle, and the pre-defined rule base. The instruction execution module controls the vehicle to make a corresponding response based on the target control instruction. Based on the vehicle control device, the in-vehicle user does not need to perform physical button operation and touch button operation, thereby improving the convenience of vehicle control during driving. At the same time, since the target control instruction is determined based on the combination of the current scene of the vehicle and the current action of the in-vehicle object, and is a control condition already existing in the pre-defined rule base, the convenience and the intelligent degree of operation are improved, and the driving safety of the vehicle is also ensured.

[0187] In one embodiment, the information acquisition module 601 is specifically configured to:

[0188] When the target control mode of the vehicle is turned on, the current scene information of the vehicle is acquired. The frame rate and / or resolution of the driver monitoring system of the vehicle are determined based on the current scene information. The image of the in-vehicle object is acquired by the driver monitoring system based on the frame rate and / or resolution. The image is an image containing hand information of the in-vehicle object.

[0189] In one embodiment, the device further includes a scene complexity recognition module configured to determine the scene complexity corresponding to the current scene information based on a pre-set scene complexity differentiation standard, and acquire the frame rate and / or resolution corresponding to the scene complexity as the frame rate and / or resolution of the driver monitoring system based on a pre-set mapping table. In the mapping table, the scene complexity and the frame rate and / or resolution are in a positive correlation.

[0190] In one embodiment, the image is an image containing hand information of the in-vehicle object.

[0191] The image recognition module 602 is specifically configured to: perform grayscale processing on the image to obtain a grayscale image, with the purpose of reducing the amount of image data and retaining hand information therein; perform background reduction processing on the grayscale image to obtain a background-processed image, with the hand information region as the foreground; identify hand key points based on at least two consecutive background-processed images, determine the current hand action information of the in-vehicle object based on the hand key points; the hand key points include finger key points, palm key points, and wrist key points.

[0192] In one embodiment, each set of control conditions in the rule base is a combination condition of a preset gesture action and a scene condition; the instruction determination module 603 is specifically configured to:

[0193] match the current scene information with the scene conditions contained in each set of control conditions in the rule base, determine candidate scene conditions matched with the current scene information; compare the current hand action information with the preset gesture action corresponding to the candidate scene conditions; in the case that the current hand action information belongs to the preset gesture action corresponding to the candidate scene conditions, obtain the control instruction corresponding to the control condition combined by the candidate scene condition and the preset gesture action as the target control instruction.

[0194] In one embodiment, the instruction execution module 604 is specifically configured to:

[0195] determine the constraint condition corresponding to the target control instruction; the constraint condition includes at least one of a vehicle speed constraint condition and an interval duration constraint of the last execution of the same type of control instruction; if there is no corresponding situation of any constraint condition at present, control the vehicle to make a corresponding response based on the target control instruction; otherwise, do not execute the target control instruction.

[0196] In one embodiment, the foregoing device further includes an updating module configured to:

[0197] within a set time after controlling the vehicle to make a corresponding response based on the target control instruction, if an adjustment operation of the in-vehicle object is obtained for the response, record the adjustment number corresponding to the target control instruction; in the case that the adjustment number is greater than a set number threshold, determine the update information corresponding to the target control instruction based on the target control instruction and the adjustment operation; update the information corresponding to the target control instruction in the rule base based on the update information.

[0198] Each module in the vehicle control device described above can be realized by software, hardware, and combinations thereof, in whole or in part. Each module described above can be embedded in or independent of the processor in the new energy vehicle in hardware form, or can be stored in the memory in the new energy vehicle in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0199] In one exemplary embodiment, a new energy vehicle is provided, the internal structure of which can be shown in the following diagram. Figure 7 As shown, the new energy vehicle includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a vehicle control method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the new energy vehicle can be a touch layer covering the display screen, or it can be a button, trackball, or touchpad installed in the new energy vehicle.

[0200] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the new energy vehicle to which the present application is applied. A specific new energy vehicle may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0201] In one exemplary embodiment, a new energy vehicle is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described vehicle control method embodiments.

[0202] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0203] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0204] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0205] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A vehicle control method characterized by, The method comprises the following steps: In the case that the intelligent control mode of the vehicle is turned on, the frame rate and / or resolution of the driver monitoring system of the vehicle are determined based on the current scene information of the vehicle; The frame rate and / or resolution are positively correlated with the scene complexity corresponding to the current scene information; the driver monitoring system is controlled to acquire images of objects in the vehicle based on the frame rate and / or resolution; the current scene information includes at least one of the current time, the current location, the current vehicle state parameter, and the current environmental information; The current action information of the objects in the vehicle is determined based on the images; The target control instruction is determined based on the current action information, the current scene information, and a pre-defined rule base; the rule base stores a plurality of sets of control conditions and control instructions corresponding to each set of control conditions, and each set of control conditions is a combination of a preset action and a scene condition; the information combination of the current action information and the current scene information matches the control condition corresponding to the target control instruction; After the target control instruction is determined, the constraint condition corresponding to the target control instruction is determined, the constraint condition includes at least one of a vehicle speed constraint condition and an interval time constraint of the last execution of the same type of control instruction; if there is no corresponding situation of any constraint condition at present, the vehicle is controlled to respond correspondingly based on the target control instruction; otherwise, the target control instruction is not executed.

2. The method of claim 1, wherein, The images are images containing hand information of the objects in the vehicle.

3. The method of claim 2, wherein, The frame rate and / or resolution of the driver monitoring system are determined based on the current scene information of the vehicle, which comprises the following steps: The scene complexity corresponding to the current scene information is determined based on a pre-set scene complexity classification standard; The frame rate and / or resolution corresponding to the scene complexity are acquired as the frame rate and / or resolution of the driver monitoring system based on a pre-set mapping table; in the mapping table, the scene complexity and the frame rate and / or resolution are in a positive correlation.

4. The method of claim 2, wherein: The current action information of the objects in the vehicle is determined based on the images, which comprises the following steps: The image is gray-scale processed to obtain a gray-scale image, with the goal of reducing the amount of image data and retaining the hand information therein; The gray-scale image is background-subtracted to obtain a background-processed image, with the region where the hand information is located as the foreground; Hand key points are identified based on at least two consecutive background-processed images, and the current hand action information of the objects in the vehicle is determined based on the hand key points; the hand key points include finger key points, palm key points, and wrist key points.

5. The method of claim 4, wherein, Each set of control conditions in the rule base is a combination of a preset gesture action and a scene condition; The target control instruction is determined based on the current action information, the current scene information, and a pre-defined rule base, which comprises the following steps: The current scene information is matched with the scene conditions contained in each set of control conditions in the rule base to determine candidate scene conditions matching the current scene information; The current hand action information is compared with a preset gesture action corresponding to the candidate scene conditions. In a case where the current hand action information belongs to a preset gesture action corresponding to the candidate scene condition, a control instruction corresponding to a combination of the candidate scene condition and the preset gesture action is obtained as the target control instruction.

6. The method according to any one of claims 1 to 5, characterized in that, Further comprising: Within a set time after the vehicle is controlled to make a corresponding response based on the target control instruction, if an adjustment operation of the in-vehicle object is obtained for the response, a number of adjustments corresponding to the target control instruction is recorded; In a case where the number of adjustments is greater than a set number threshold, update information corresponding to the target control instruction is determined based on the target control instruction and the adjustment operation; Based on the update information, information corresponding to the target control instruction in the rule base is updated.

7. A vehicle control device characterized by comprising: Comprise: An information acquisition module is configured to, in a case where an intelligent control mode of a vehicle is turned on, determine a frame rate and / or resolution of a driver monitoring system of the vehicle based on current scene information of the vehicle; the frame rate and / or resolution are positively correlated with a scene complexity degree corresponding to the current scene information; and control the driver monitoring system to acquire an image of an in-vehicle object based on the frame rate and / or resolution; the current scene information comprises at least one of a current time, a current location, a current vehicle state parameter, and current environment information; An image recognition module is configured to determine current action information of the in-vehicle object based on the image; An instruction determination module is configured to determine a target control instruction based on the current action information, the current scene information, and a pre-defined rule base; the rule base stores a plurality of groups of control conditions and control instructions corresponding to each group of control conditions, and each group of the control conditions is a combination of an action condition and a scene condition; an information combination of the current action information and the current scene information matches a control condition corresponding to the target control instruction; An instruction execution module is configured to, after the target control instruction is determined, determine a constraint condition corresponding to the target control instruction, the constraint condition comprises at least one of a vehicle speed constraint condition and an interval duration constraint of a last execution of a same type of control instruction, and if there is no case corresponding to any of the constraint conditions, control the vehicle to make a corresponding response based on the target control instruction; otherwise, the target control instruction is not executed.

8. A new energy vehicle, comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Images