Vehicle control method and device, new energy vehicle and storage medium

Abstract

The invention relates to a vehicle control method and device, a new energy vehicle and a storage medium, and relates to the technical field of new energy vehicles. The method comprises the steps that under the condition that a target control mode of a vehicle is started, current scene information of the vehicle and an image of an object in the vehicle are obtained, and the image is analyzed to determine current action information of the object in the vehicle; based on the obtained current action information of the object in the vehicle, the current scene information of the vehicle and a predefined rule base, a target control instruction matched with the current scene and the current action is determined in a self-adaptive mode, and the vehicle is controlled to make a corresponding response based on the target control instruction. Based on the vehicle control method, a user in the vehicle does not need physical key operation or touch key operation, and the convenience of vehicle control in the driving process is improved; meanwhile, the target control instruction is jointly determined based on the combination of the current scene of the vehicle and the current action of the object in the vehicle, so that the driving safety of the vehicle is ensured.

Description

Technical Field

[0001] The present application relates to the field of new energy vehicle technology, and in particular to a vehicle control method, device, new energy vehicle, storage medium and computer program product. Background Art

[0002] In the current field of vehicle control systems, there are mainly several control technologies:

[0003] 1. Physical button control: Various physical buttons are set on the dashboard, steering wheel or center console to operate vehicle functions. The layout of these physical buttons is fixed, and the driver needs to be familiar with the location and function of each button.

[0004] 2. Touch screen control technology: More and more vehicles are equipped with touch screen control systems. Drivers can operate vehicle functions by touching icons or menus on the screen. This system usually has a graphical interface that can display more information and function options. However, due to factors such as bumps and shaking during vehicle driving, the driver may make mistakes when touching the screen.

[0005] 3. Voice control technology: Using the microphone and voice recognition system in the car, the driver can control the vehicle functions by speaking specific commands. However, the current voice control system has limited understanding of natural language, and the driver must use a specific, system-preset command format and vocabulary, otherwise the system may not be able to accurately understand it.

[0006] 4. Simple driver monitoring system (DMS) combined with limited control functions: Some vehicles are equipped with DMS, which is controlled by blinking, head movements, etc., but the recognition accuracy is poor and it is easily disturbed by the driver's normal physiological movements (such as normal blinking, slight head shaking, etc.), resulting in misjudgment.

[0007] In summary, the current vehicle control system can no longer meet the growing demand for intelligent driving. Summary of the invention

[0008] Based on this, it is necessary to provide a vehicle control method, device, new energy vehicle, computer-readable storage medium and computer program product to address the above technical problems.

[0009] In a first aspect, the present application provides a vehicle control method, comprising:

[0010] When the target control mode of the vehicle is turned on, obtaining the current scene information of the vehicle and the image of the object in the vehicle; the current scene information includes at least one of the current time, the current position, the current vehicle state parameter, and the current environment information;

[0011] determining current motion information of the object in the vehicle based on the image;

[0012] Determine a target control instruction based on the current action information, the current scene information and a predefined rule base; the rule base stores multiple groups of control conditions and control instructions corresponding to each group of control conditions, and each group of control conditions is a combination of preset action and scene conditions; the information combination of the current action information and the current scene information matches the control condition corresponding to the target control instruction;

[0013] Based on the target control instruction, the vehicle is controlled to make a corresponding response.

[0014] In one embodiment, when the target control mode of the vehicle is turned on, obtaining the current scene information of the vehicle and the image of the object in the vehicle includes:

[0015] When the target control mode of the vehicle is turned on, obtaining the current scene information of the vehicle;

[0016] determining a frame rate and / or resolution of a driver monitoring system of the vehicle based on the current scene information;

[0017] The driver monitoring system is controlled to obtain an image of the in-vehicle object based on the frame rate and / or resolution; the image is an image containing hand information of the in-vehicle object.

[0018] In one embodiment, determining a frame rate and / or a resolution of a driver monitoring system based on the current scene information includes:

[0019] Determining the scene complexity corresponding to the current scene information based on a preset scene complexity distinction standard;

[0020] Based on a preset mapping table, the frame rate and / or resolution corresponding to the scene complexity is obtained as the frame rate and / or resolution of the driver monitoring system; in the mapping table, there is a positive correlation between the scene complexity and the frame rate and / or resolution.

[0021] In one of the embodiments, the image is an image including hand information of the object in the vehicle;

[0022] The determining the current motion information of the object in the vehicle based on the image includes:

[0023] To reduce the amount of image data and retain the hand information therein, grayscale processing is performed on the image to obtain a grayscale image;

[0024] Taking the area where the hand information is located as the foreground, performing background subtraction processing on the grayscale image to obtain an image after background processing;

[0025] Based on at least two consecutive frames of the background-processed image, hand key points are identified, and current hand motion information of the object in the vehicle is determined based on the hand key points; the hand key points include finger key points, palm key points and wrist key points.

[0026] In one embodiment, each group of control conditions in the rule base is a combination of preset gesture actions and scene conditions;

[0027] The determining of the target control instruction based on the current action information, the current scene information and a predefined rule base includes:

[0028] Matching the current scene information with the scene conditions included in each group of control conditions in the rule base to determine candidate scene conditions matching the current scene information;

[0029] Comparing the current hand motion information with a preset hand gesture corresponding to the candidate scene condition;

[0030] In the case that the current hand motion information belongs to a preset gesture action corresponding to the candidate scene condition, a control instruction corresponding to a control condition of a combination of the candidate scene condition and the preset gesture action is obtained as the target control instruction.

[0031] In one embodiment, controlling the vehicle to make a corresponding response based on the target control instruction includes:

[0032] Determine the constraint condition corresponding to the target control instruction; the constraint condition includes at least one of a vehicle speed constraint condition and an interval time constraint of the last execution of the same type of control instruction;

[0033] If there is no situation corresponding to any of the constraint conditions at present, the vehicle is controlled to make a corresponding response based on the target control instruction; otherwise, the target control instruction is not executed.

[0034] In one embodiment, it also includes:

[0035] Within a set time after the vehicle is controlled to make a corresponding response based on the target control instruction, if an adjustment operation is obtained by the in-vehicle object in response to the response, the number of adjustments corresponding to the target control instruction is recorded;

[0036] In a case where the number of adjustments is greater than a set number threshold, determining update information corresponding to the target control instruction based on the target control instruction and the adjustment operation;

[0037] Based on the update information, the information corresponding to the target control instruction in the rule base is updated.

[0038] In a second aspect, the present application provides a vehicle control device, comprising:

[0039] An information acquisition module, used to acquire the current scene information of the vehicle and the image of the object in the vehicle when the target control mode of the vehicle is turned on; the current scene information includes at least one of the current time, the current position, the current vehicle state parameter, and the current environment information;

[0040] An image recognition module, used to determine current motion information of the object in the vehicle based on the image;

[0041] An instruction determination module is used to determine a target control instruction based on the current action information, the current scene information and a predefined rule base; the rule base stores multiple groups of control conditions and control instructions corresponding to each group of control conditions, and each group of control conditions is a combination of an action condition and a scene condition; the information combination of the current action information and the current scene information matches the control condition corresponding to the target control instruction;

[0042] The instruction execution module is used to control the vehicle to make a corresponding response based on the target control instruction.

[0043] In a third aspect, the present application provides a new energy vehicle, comprising a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of any of the methods described above when executing the computer program.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of any of the methods described above are implemented.

[0045] In a fifth aspect, the present application provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the steps of any of the methods described above are implemented.

[0046] The above-mentioned vehicle control method, device, new energy vehicle, storage medium and computer program product, when the target control mode of the vehicle is turned on, obtains the current scene information of the vehicle and the image of the object in the vehicle, and analyzes the image to determine the current action information of the object in the vehicle; based on the acquired current action information of the object in the vehicle, the current scene information of the vehicle and the pre-defined rule library, adaptively determines the target control instruction that matches both the current scene and the current action, and controls the vehicle to make a corresponding response based on the target control instruction. The user in the vehicle does not need to operate physical buttons or touch buttons, which improves the convenience of vehicle control during driving; at the same time, because the target control instruction is determined based on the combination of the current scene of the vehicle and the current action of the object in the vehicle, and is a control condition already in the pre-defined rule library, it is beneficial to ensure vehicle driving safety on the basis of improving convenience and intelligent control. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] FIG1 is an application environment diagram of a vehicle control method according to an embodiment;

[0049] Figure 2 A schematic flow chart of a vehicle control method in one embodiment;

[0050] Figure 3 A schematic diagram of a process for obtaining current scene information of a vehicle and images of objects inside the vehicle in one embodiment;

[0051] FIG4 is a schematic diagram of a process for determining current motion information of an object in a vehicle based on an image in one embodiment;

[0052] Figure 5 A schematic diagram of a flow chart of determining a target control instruction in an embodiment;

[0053] Figure 6 is a structural block diagram of a vehicle control device in one embodiment;

[0054] FIG. 7 is a structural diagram of a new energy vehicle in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0057] In the field of traditional vehicle control systems, there are mainly the following control technologies:

[0058] Physical button control: Various physical buttons are set on the dashboard, steering wheel or center console to operate vehicle functions, such as controlling window lifting, adjusting air conditioning temperature and wind speed, turning on or off the audio system, etc. However, these physical buttons have a fixed layout and are distributed in different locations in the car. When operating, the driver needs to take his eyes off the road to find and press the corresponding buttons, which will cause distraction during driving and increase the risk of accidents, especially when driving at high speeds or in complex road conditions. In addition, there are many physical buttons in the car with different functions. The driver needs to spend time to familiarize himself with and memorize the location and function of each button. For uncommon function buttons, it may be difficult to find them quickly when they are needed, and the operation is not convenient enough.

[0059] Touch screen control technology: With the development of electronic technology, more and more vehicles are equipped with touch screen control systems. The driver can operate the vehicle functions by touching the icons or menus on the screen, and the operation method is relatively intuitive. This system usually has a graphical interface that can display more information and function options. Similar to physical buttons, when operating the touch screen, the driver must shift his eyes from the road ahead to the screen. Even a short period of time away from the road may miss important information on the road at a critical moment, posing a safety hazard. At the same time, during the driving process of the vehicle, due to factors such as bumps and shaking, the driver may make mistakes when touching the screen, especially when clicking on smaller icons or performing delicate operations. Moreover, the touch screen may be insensitive in some cases (such as when the screen is wet or operated with gloves).

[0060] Voice control technology: Using the microphone and voice recognition system in the car, the driver can control the vehicle functions by speaking specific commands. Voice control provides a convenient way for drivers to operate without hands, which is especially practical when it is inconvenient to operate with both hands during driving. The current voice control system can recognize some basic commands, such as entering navigation destinations, making calls, adjusting volume, etc. However, in noisy environments, such as when passengers in the car are talking, playing music, and the vehicle is driving on noisy roads, the accuracy of voice recognition will be greatly reduced, and external noise may also mask the driver's voice commands, causing the system to fail to recognize correctly or recognize errors. In addition, the existing voice control system has limited understanding of natural language, and the driver must use specific, system-preset command formats and vocabulary, otherwise the system may not be able to accurately understand, and drivers from different regions and with different accents may encounter difficulties in command recognition.

[0061] Other control technologies: For example, some vehicles are equipped with DMS, which is mainly used to monitor the driver's facial expressions, eye conditions, etc. to determine whether the driver is tired or distracted. Some DMS systems have begun to try to combine some simple control functions, such as blinking or head movements to achieve certain specific functions, but these functions are often limited and not precise enough. However, vehicle control through blinking, head movements, etc. is easily interfered by the driver's normal physiological movements (such as normal blinking, slight head shaking, etc.), leading to misjudgment; at the same time, the functions currently achieved by this combination are very limited, and cannot meet the driver's needs for diversified vehicle function control in different scenarios, and lack flexibility and scalability.

[0062] It can be seen that the above-mentioned vehicle control technologies have obvious deficiencies in driving safety, operational convenience, functional diversity and system integration, and cannot meet the growing demand for intelligent driving and drivers' expectations for convenient and safe vehicle operation.

[0063] Based on this, the vehicle control method proposed in the present application is based on an intelligent vehicle control system. When the vehicle turns on the corresponding control mode, the current scene information of the vehicle and the image of the object in the vehicle are obtained, the current action information of the object in the vehicle is determined based on the image of the object in the vehicle, and the current action information and the current scene information are combined to match a predefined rule library, in which multiple groups of control conditions and control instructions corresponding to each group of control conditions are stored, and each group of control conditions is a combination of preset actions and scene conditions. Thereby, the target control instruction corresponding to the current action information and the current scene information can be determined, and the vehicle is controlled to make a corresponding response based on the target control instruction. Through this solution, the intelligent and personalized precise control of multiple operation functions of the vehicle under various scene conditions can be realized, providing a safer, more convenient and intelligent vehicle control solution for objects in the vehicle.

[0064] The vehicle control method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. The vehicle includes a cockpit software system 101, a vehicle interconnection terminal T-BOX (Telematics BOX) 102, a vehicle controller 103, a driver monitoring system (DMS) 104, a power battery system 105, a motor and a motor controller 106, etc. connected through a bus 100.

[0065] The cockpit software system 101 can be used to control the user interface in the car, including the instrument panel, touch screen, entertainment system and navigation system, and provide interactive functions for the driver and passengers.

[0066] As the vehicle's telematics unit, T-BOX 102 can connect the vehicle and the cloud through a communication network and is responsible for collecting, uploading and remotely controlling vehicle data.

[0067] The vehicle controller 103, as the brain of the new energy vehicle, is responsible for coordinating and controlling the operation of components such as batteries, motors, motor controllers, and thermal management systems, to achieve intelligent control of the vehicle's power distribution, energy management, and vehicle functions.

[0068] The driver monitoring system (DMS) 104 is a system that uses cameras and sensors to monitor the driver's status in real time, such as eye movements, facial expressions, and driving behavior, to detect risks such as fatigue driving and distracted driving and provide timely reminders, thereby improving driving safety.

[0069] The power battery system 105 is mainly used to provide a main power source for the vehicle and to provide power for the motor when driving.

[0070] The motor and motor controller 106, where the motor is the core component that converts electrical energy into mechanical energy and provides power when the vehicle is driving, and the motor controller is responsible for controlling the speed and torque output of the motor. It receives instructions from the vehicle control system and adjusts the working state of the motor so that the motor can achieve the functions of acceleration, deceleration and braking. Specifically, the motor controller can change the working state of the motor by adjusting the current and voltage, and decide whether the motor drives the vehicle forward or recovers energy.

[0071] In an exemplary embodiment, Figure 2 As shown, a vehicle control method is provided, and the method is applied to the vehicle controller of the vehicle in FIG. 1 as an example for explanation. In other embodiments, the method can also be implemented by the vehicle controller and other controllers or control systems (such as DMS). The method includes the following steps S201 to S203. Among them:

[0072] Step S201, when the target control mode of the vehicle is turned on, the current scene information of the vehicle and the image of the object inside the vehicle are obtained, and the current action information of the object inside the vehicle is determined based on the obtained image of the object inside the vehicle.

[0073] Among them, the in-car object can be the driver, or a passenger in a preset position in the car, such as a front passenger passenger. The target control mode can be a pre-configured intelligent control mode, in which the relevant hardware and / or software systems in the vehicle are activated, and the vehicle scene and the in-car object action are collected and identified based on the hardware and / or software system, and the function of adaptively matching the vehicle control instructions based on the identified scene information and the in-car object action information is enabled. In some embodiments, the target control mode can be an update based on the existing control mode of the vehicle, that is, the target control mode is integrated into the existing control mode. In other embodiments, the target control mode can also be an independent control mode in the vehicle. The in-car object can turn on or off the target control mode through physical buttons, touch buttons or voice commands in the car.

[0074] The current scene information obtained includes at least one of the current time, the current vehicle state parameters, and the current environment information. The current time may be year, month, day information, week information, and / or specific time period information. The current vehicle state parameters may be any parameters related to the vehicle state, including but not limited to the vehicle speed state, the light state, the steering wheel state, the door state, the window state, the trunk state, the wiper state, etc. The current environment information may be the natural environment information of the vehicle, such as weather information and light information, and may also be the road condition information of the vehicle, such as a highway scene, a mountain road scene, a congested road scene, a parking lot scene, etc.

[0075] In one embodiment, the current scene information of the vehicle can be obtained through one or more sensors deployed in the vehicle. For example, the current illumination information is measured by a light sensor, the current vehicle speed information is obtained by a speed sensor, and the vehicle-mounted radar or vehicle flow and pedestrian flow information is obtained. In other embodiments, the current scene information of the vehicle can also be obtained through the Internet of Vehicles, for example, the current illumination information and vehicle flow and pedestrian flow information are obtained through the Internet of Vehicles. In addition, the corresponding current scene information can also be obtained through one or more sensors and the Internet of Vehicles.

[0076] Among them, taking the driver as an example, the driver's image can be obtained through the vehicle camera, for example, the driver's image can be obtained through the camera associated with the DMS system, and the driver's current action information can be identified based on the DMS system. Different from the driver's image collected by the traditional DMS system, which is concentrated on the driver's facial area, the driver's image collected in the embodiment of the present application includes not only the driver's facial area image, but also the driver's hand area image or the upper body image.

[0077] It should be noted that this application obtains images of objects inside the car and performs image analysis, which is collected, used and processed with the authorization of the objects inside the car and complies with relevant regulations.

[0078] In some embodiments, when the above-mentioned target control mode of the vehicle is turned on, the current scene information of the vehicle and the images of objects inside the vehicle can be obtained based on a preset frequency or a dynamically adaptive frequency, and the characteristics of the current scene information can be analyzed based on a preset scene analysis algorithm or model, and the current action information of the objects inside the vehicle can be analyzed based on the acquired images of the objects inside the vehicle based on a preset image recognition algorithm or model.

[0079] Step S202, based on the determined current action information of the in-vehicle object, the current scene information of the vehicle and a predefined rule base, a target control instruction is determined. The rule base stores multiple groups of control conditions and control instructions corresponding to each group of control conditions, and each group of control conditions is a combination of preset action and scene conditions; and the information combination of the current action information and the current scene information matches the control condition corresponding to the target control instruction.

[0080] In an embodiment of the present application, a predefined rule base stores multiple correspondences between scene conditions + action conditions + vehicle control instructions. The scene condition in each correspondence may be a single scene condition, such as illuminance greater than 1000 lux, or a combination of multiple scene conditions, such as highway driving scene & 6:00 - 22:00) & illuminance greater than 1000 lux. The action condition in each correspondence may be an action condition, such as "hand rotation", or a combination of multiple action conditions, such as hand rotation & rotation speed greater than a set value & rotation angle greater than a set angle. The vehicle control instruction in each correspondence may be a control instruction, such as a volume reduction instruction, or multiple control instructions, such as a volume reduction instruction & a window closing instruction.

[0081] In some embodiments, the rule base can be customized by the user based on the user's driving habits. In other embodiments, a default rule base can be pre-configured for multiple vehicles based on big data analysis. In addition, the rule base for different vehicles can be dynamically updated based on the matching of the corresponding relationships in the rule base by different users to adapt to the driving habits of the vehicle user and improve the degree of intelligence.

[0082] In one embodiment, after determining the current action information of the object in the vehicle and the current scene information of the vehicle, the current action information can be matched with the predefined rule base to determine the matching candidate control conditions, and then on this basis, the target control conditions and the target control instructions corresponding to the target control conditions can be further matched in combination with the current scene information. This method is conducive to improving the matching efficiency for situations where the action conditions are simple and the scene conditions are complex. In addition, the current scene information can be matched with the predefined rule base first, and then the current action information can be matched. This method has a higher matching efficiency for situations where the scene conditions are simple and the action conditions are complex; the current scene information and the current action information can also be matched with the predefined rule base at the same time to determine the target control conditions and the target control instructions corresponding to the target control conditions. The corresponding matching method can be determined based on the specific circumstances of the scene conditions and action conditions in the rules, so as to adaptively match the corresponding target control instructions from the predefined rule base.

[0083] Step S203: Based on the determined target control instruction, the vehicle is controlled to make a corresponding response.

[0084] The target control command may be a vehicle speed control command, a body component control command or a cockpit software control command. The body component control command includes but is not limited to a light control command, a window control command, a tailgate control command, a wiper control command, an air conditioning control command, etc. The cockpit software control command includes but is not limited to a volume control command, a playback control command, an application software switching command, etc.

[0085] In one embodiment, based on the determined target control instruction, the target control instruction can be executed immediately to control the vehicle to make a corresponding response. The current vehicle state can also be further determined to determine whether the vehicle is currently suitable for executing the target control instruction, or to determine the safety of the vehicle after executing the target control instruction. Based on the further determination results, the vehicle is controlled to make a corresponding response, thereby improving the safety of intelligent control.

[0086] Through the vehicle control method of the above-mentioned embodiment of the present application, when the target control mode of the vehicle is turned on, the current scene information of the vehicle and the image of the object in the vehicle are obtained, and the current action information of the object in the vehicle is determined by analyzing the image; based on the obtained current action information of the object in the vehicle, the current scene information of the vehicle and the pre-defined rule library, a target control instruction that matches both the current scene and the current action is adaptively determined, and the vehicle is controlled to make a corresponding response based on the target control instruction. The user in the vehicle does not need to operate physical buttons or touch buttons, which improves the convenience of vehicle control during driving; at the same time, because the target control instruction is determined based on a combination of the current scene of the vehicle and the current action of the object in the vehicle, and is a control condition already in the pre-defined rule library, thereby, on the basis of improving convenience and the degree of intelligent control, it is also beneficial to ensure vehicle driving safety.

[0087] In one embodiment, Figure 3 As shown, in the above-mentioned case where the target control mode of the vehicle is turned on, the current scene information of the vehicle and the image of the object in the vehicle are obtained, and the specific implementation method may include:

[0088] Step S301, when the target control mode of the vehicle is turned on, the current scene information of the vehicle is obtained, and based on the obtained current scene information, the frame rate and / or resolution of the driver monitoring system of the vehicle is determined.

[0089] The driver monitoring system may include an image acquisition device, which may be an in-vehicle camera and / or an infrared sensor, and the frame rate and / or resolution of the image acquisition device may be adjustable. In one embodiment, the frame rate and / or resolution under different scenes may be preset and saved as a corresponding comparison table. After the current scene information is obtained, the frame rate and / or resolution of the driver monitoring system (image acquisition device) of the vehicle may be quickly determined based on the comparison table.

[0090] Step S302 : controlling the driver monitoring system to acquire images of objects inside the vehicle based on the determined frame rate and / or resolution.

[0091] The images acquired by the driver monitoring system include hand information of the object in the car, and may also include images of the head and shoulders of the object in the car. By performing image processing and image recognition on the acquired multiple frames of images of the object in the car, the hand movements of the object in the car can be determined, or the relative position changes of the hand, head and shoulders can be further referred to to obtain more accurate hand movement recognition results.

[0092] Through the method of this embodiment, the frame rate and / or resolution of the driver monitoring system can be dynamically adjusted for different scenarios. When the current scene information is a more complex scene, the frame rate and / or resolution of the driver monitoring system can be increased, and images of objects inside the vehicle can be acquired at a higher frame rate and / or resolution, which is beneficial to improving the vehicle control sensitivity in complex scenes; when the current scene information is a simpler scene, the frame rate and / or resolution of the driver monitoring system can be appropriately reduced, and images of objects inside the vehicle can be acquired at a lower frame rate and / or resolution, which is beneficial to saving resource overhead.

[0093] In one embodiment, the specific method of determining the frame rate and / or resolution of the driver monitoring system based on the current scene information of the vehicle may include:

[0094] Based on a preset scene complexity distinction standard, determine the scene complexity corresponding to the current scene information; based on a preset mapping table, obtain the frame rate and / or resolution corresponding to the scene complexity as the frame rate and / or resolution of the driver monitoring system; in the mapping table, there is a positive correlation between the scene complexity and the frame rate and / or resolution.

[0095] Among them, the scene complexity distinction standard is used to determine the complexity of the current scene information. For example, three complexity levels can be set, corresponding to high complexity, medium complexity and low complexity respectively; and for each complexity level, the corresponding distinction standard is set respectively. For example, the standard corresponding to the high complexity level can be that if any of the following items are met, it belongs to a scene of high complexity: urban congestion environment, traffic light intersection, rainy and snowy weather, dense fog weather, light intensity below the set threshold, icy road surface, peak travel period, etc. The current scene information is matched with the preset scene complexity distinction standard to determine the scene complexity corresponding to the current scene information. In one embodiment, if there is no information corresponding to the current scene information in the preset scene complexity distinction standard, the current scene information can be determined as the default scene complexity, such as low complexity.

[0096] In addition, based on empirical data, frame rates and / or resolutions corresponding to different scene complexities can be set and saved as a preset mapping table. After determining the scene complexity of the current scene information in the above manner, the corresponding frame rate and / or resolution can be quickly determined by looking up the table.

[0097] Through the above-mentioned embodiments, combined with the driver monitoring system configured in the vehicle, the scene complexity corresponding to the current scene information can be determined based on preset standards. On this basis, by looking up the table, the frame rate and / or resolution of the driver monitoring system adapted to the current scene information can be quickly determined. In the case where the scene information of the vehicle switches frequently, it is beneficial to ensure the response sensitivity and safety of the above-mentioned action- and scene-based intelligent control of the vehicle. At the same time, only the driver monitoring system needs to be adjusted accordingly, and the hardware and software implementation costs are low.

[0098] In one embodiment, the image of the in-vehicle object acquired in the aforementioned embodiment is an image containing hand information of the in-vehicle object, and the current action information determined based on the image includes the current hand action information of the in-vehicle object. In this case, as shown in FIG. 4 , the process of determining the current action information of the in-vehicle object based on the image mentioned in the aforementioned embodiment may include:

[0099] Step S401 , with the goal of reducing the amount of image data and retaining the hand information therein, grayscale processing is performed on the image of the object in the car to obtain a grayscale image.

[0100] Step S402, taking the area where the hand information is located in the grayscale image as the foreground, performing background subtraction processing on the grayscale image to obtain an image after background processing.

[0101] Step S403, based on at least two consecutive frames of the background-processed image, identify the hand key points, and determine the current hand motion information of the object in the car based on the key points; wherein the hand key points include finger key points, palm key points and wrist key points, and the current hand motion information includes at least one of the overall displacement information of the hand, the overall movement speed information of the hand, the local displacement information of the hand key points and the local movement speed information of the hand key points. For example, the current hand motion information may include a combination of at least two of the hand motion direction, hand motion displacement, hand motion speed, hand motion angle and hand motion duration.

[0102] Based on the above embodiment, taking the driver as an example, the image acquisition process may include capturing the driver's hand movements in real time through the DMS camera at an intelligently adjusted frame rate (the frame rate can be dynamically adjusted within the range of 10-60 frames / second according to the complexity of the scene and the activity of the hand movements). The collected image data is transmitted to the vehicle controller in real time for subsequent processing. Then the image preprocessing is performed, including: high-precision grayscale processing of the image to reduce the amount of data while retaining key hand information, and then using a preset background adaptive subtraction algorithm. This algorithm can automatically optimize the subtraction parameters according to different driving environment lighting conditions (from high-brightness environment under strong sunlight to low-brightness environment without lighting at night) and complex backgrounds (such as interior decoration, sundries, etc.), accurately eliminate irrelevant background images other than the hand, and only retain clear hand image areas for subsequent analysis.

[0103] Among them, the hand motion information is recognized based on the driver's image. This application can adopt a multi-parameter fine gesture determination method. The corresponding gesture recognition algorithm is based on the driver's hand displacement, speed, trajectory, duration, finger bending and extension state, gesture angle relative to the driver's body, and the position of the hand in the three-dimensional space in the car. Through the analysis of the full range of hand information between consecutive frames, various parameters of the hand are accurately calculated. For example:

[0104] (1) The hand displacement, velocity and trajectory can be calculated as follows:

[0105] The initial hand position is (x0, y0, z0), the hand position in the current frame is (xt, yt, zt), and the displacement calculation formula is:

[0106] in is the 3D displacement distance (pixels) of the hand in the current frame. Assume that the time interval between two frames is , the hand speed calculation formula is:

[0107] is the hand movement speed (pixels / second). The three-dimensional motion trajectory of the hand is constructed by recording the position of multiple frames. These parameters and trajectory information are used to accurately determine the amplitude, direction, speed and changes in space of the hand movement.

[0108] (2) Gesture action determination and detail analysis.

[0109] This application can recognize basic gestures, such as:

[0110] Waving left (horizontally): When the horizontal displacement of the hand is greater than the horizontal displacement threshold, and the displacement direction is negative (leftward), the speed is greater than the speed threshold, the horizontal movement trajectory of the hand is approximately straight, and the angle between the gesture and the front of the driver's body is within the first range (such as 30 degrees), it is recognized as "waving horizontally to the left".

[0111] Waving right (horizontally): When the horizontal displacement of the hand is greater than the horizontal displacement threshold, and the displacement direction is positive (rightward movement), the speed is greater than the speed threshold, the horizontal movement trajectory of the hand is a straight line, and the angle between the gesture and the front of the driver's body is within the second range (such as 32 degrees), it is recognized as "waving horizontally to the right".

[0112] Waving upward (vertical direction): When the vertical displacement of the hand is greater than the vertical displacement threshold, and the displacement direction is positive (moving upward), the vertical movement trajectory of the hand is a straight line, and the angle between the gesture and the front of the driver's body is within the third range (such as 45 degrees), it is recognized as "waving vertically upward".

[0113] Waving downward (vertical direction): When the vertical displacement of the hand is greater than the vertical displacement threshold, and the displacement direction is negative (moving downward), the vertical movement trajectory of the hand is a straight line, and the angle between the gesture and the front of the driver's body is within the fourth range (such as 45 degrees), it is recognized as "waving vertically downward". Among them, the displacement thresholds and speed thresholds in the horizontal and vertical directions can be adaptively adjusted according to different driving scenarios, driver habits and vehicle driving conditions.

[0114] This application can also realize the recognition of complex gestures, such as:

[0115] Rotation gesture (two-dimensional plane): When the hand is detected to make a circular or arc motion in a clockwise or counterclockwise direction in a two-dimensional plane with the wrist as the center, and the fingers maintain a specific bending and stretching state (such as the index finger is straight, the other fingers are slightly bent, and the angle between the index finger and the palm plane reaches the set angle threshold (such as 15 degrees), and the rotation speed reaches the rotation speed threshold (such as 30 degrees per second), it is recognized as a rotation gesture. In addition, different control instructions can be corresponding to the rotation direction. For example, when the rotation is clockwise and the above finger states are met, it corresponds to increasing the volume or raising the air conditioning temperature; when the rotation is counterclockwise and the finger states are the same, it corresponds to decreasing the volume or lowering the air conditioning temperature.

[0116] Grasping gesture (three-dimensional space): The grasping gesture is judged by analyzing the change rate of the palm area, the bending speed of the fingers, and the change of the position of the palm in three-dimensional space. When the palm area decreases significantly in a short time (such as 0.5 seconds) (the reduction ratio is greater than the set ratio threshold (such as 30%)) and the fingers are quickly bent and clenched into a fist, and the center position of the palm is within a spherical range with a radius of r (such as 0.5 meters) centered on the driver's chest, it is recognized as a "clenched fist" operation; when the palm area increases rapidly and the fingers are straightened, and the palm position is within the above range, it is recognized as an "open palm" operation. Among them, the time threshold and ratio threshold of the grasping action can be set according to the average operation speed of the driver.

[0117] Through the above embodiments, based on image preprocessing and multi-parameter hand motion recognition, accurate recognition of the driver's hand motions can be ensured during vehicle driving, reducing the probability of misrecognition and improving vehicle control safety.

[0118] In one embodiment, each group of control conditions in the aforementioned rule base is a combination of preset gesture actions and scene conditions. Figure 5 As shown, the specific method of determining the target control instruction based on the current action information of the object in the vehicle, the current scene information of the vehicle and the predefined rule library may include:

[0119] Step S501 : Match the current scene information determined in the above-mentioned embodiment with the scene conditions included in each group of control conditions in the rule base to determine the candidate scene conditions matching the current scene information.

[0120] Among them, the number of candidate scene conditions matching the current scene information can be one or more. When there is no scene condition matching the current scene information, the number of candidate scene conditions is 0. In this case, it can be determined that there is no corresponding target control instruction, and there is no need to adaptively execute the target control instruction.

[0121] Step S502, determining whether the current hand motion information belongs to a preset gesture action corresponding to the candidate scene condition.

[0122] Among them, in the rule base, each candidate scene condition may correspond to multiple different preset gesture actions, and correspond to different control instructions. In addition, the current hand movement information may match one or more preset gesture actions corresponding to a candidate scene condition, or may not match the preset gesture actions of any candidate scene condition. If it is the latter case, it is determined that there is no corresponding target control instruction at present, and there is no need to adaptively execute the target control instruction.

[0123] Step S503, when the current hand motion information belongs to any preset gesture action corresponding to the candidate scene condition, obtain a control instruction corresponding to the control condition of the combination of the candidate scene condition and the any preset gesture action as a target control instruction.

[0124] Wherein, when it is determined that the current hand motion information belongs to a preset gesture action of a candidate scene condition, the control instruction corresponding to the candidate scene condition + the preset gesture action is used as the target control instruction. When it is determined that the current hand motion information belongs to multiple preset gesture actions of a candidate scene condition, the multiple control instructions corresponding to the candidate scene condition + each preset gesture action are used as the target control instruction.

[0125] Through the above embodiment, the current scene information is first matched with the control conditions in the rule library, and then the candidate control conditions are matched with the current hand movement information. Since the scene information is simpler than the gesture information, the efficiency of obtaining the candidate control conditions is higher, and the amount of data for further gesture movement matching is reduced, which is beneficial to ensuring efficiency in the process of adaptively matching target control instructions.

[0126] In order to better illustrate the embodiments of the present application, the rule base of the present application is illustrated below by way of example. The rule base can be constructed in the following manner.

[0127] First, multiple preset gesture actions and multiple scene conditions are determined; wherein the multiple preset gesture actions include hand gestures and finger gestures, and specifically, each preset gesture action includes at least two information of moving direction, moving speed, moving angle, and moving time, so as to more accurately identify whether the driver's current gesture is a preset gesture and reduce misidentification. Among them, various scene conditions include at least two of road conditions, time conditions, vehicle state conditions, and natural environment conditions, so as to more accurately identify whether the current scene information is a preset scene condition and reduce misidentification.

[0128] For ease of understanding, the preset gesture actions and scene conditions are respectively illustrated below with examples.

[0129] The preset gesture actions include basic gestures, such as:

[0130] (1) Waving to the left (horizontally). The definition of this gesture action may include: when the horizontal displacement of the hand is greater than the horizontal displacement threshold, and the displacement direction is negative (leftward), the movement speed is greater than the speed threshold, the movement trajectory of the hand in the horizontal direction is approximately straight, and the angle between the gesture and the front of the driver's body is within a range (such as 30 degrees), it is judged as "waving to the left horizontally".

[0131] (2) Waving to the right (horizontally). The definition of this gesture action can be: when the horizontal displacement of the hand is greater than the horizontal displacement threshold, and the displacement direction is positive (rightward movement), the speed is greater than the speed threshold, the horizontal movement trajectory of the hand is a straight line, and the angle between the gesture and the front of the driver's body is within a range (such as 30 degrees), it is judged as "waving to the right horizontally".

[0132] (3) Waving upward (vertical direction). The definition of this gesture action can be: when the vertical displacement of the hand is greater than the vertical displacement threshold, and the displacement direction is positive (moving upward), the vertical movement trajectory of the hand is a straight line, and the angle between the gesture and the front of the driver's body is within a range (such as 45 degrees), it is judged as "waving vertically upward".

[0133] (4) Waving downward (vertical direction). The definition of this gesture action can be: when the vertical displacement of the hand is greater than the vertical displacement threshold, and the displacement direction is negative (moving downward), the vertical movement trajectory of the hand is a straight line, and the angle between the gesture and the front of the driver's body is within a range (such as 45 degrees), it is judged as "waving vertically downward".

[0134] Among them, the displacement threshold and speed threshold can be adaptively adjusted according to different driving scenarios, driver habits and vehicle driving status.

[0135] Preset gesture actions can also include complex gestures, such as:

[0136] (5) Rotation gesture (two-dimensional plane). The definition of this gesture action can be: when the hand is detected to make a circular or arc motion in a clockwise or counterclockwise direction in a two-dimensional plane with the wrist as the center, and the fingers maintain a specific bending and stretching state (such as the index finger is straight, the other fingers are slightly bent, and the angle between the index finger and the palm plane reaches the set angle threshold (such as 15 degrees), different operations are determined according to the rotation direction. For example, when the rotation is clockwise and the above finger states are met, it is determined to increase the volume or increase the air conditioning temperature; when the rotation is counterclockwise and the finger states are the same, it is determined to decrease the volume or reduce the air conditioning temperature. The operation is triggered only when the rotation speed is above the rotation speed threshold (such as 30 degrees per second) and the rotation angle is greater than (such as 90 degrees).

[0137] (6) Grasping gesture (three-dimensional space): This gesture action can be defined as: judging the grasping gesture by analyzing the change rate of the palm area, the bending speed of the fingers, and the change of the palm position in the three-dimensional space. When the palm area decreases significantly in a short period of time (e.g., 0.5 seconds) (the reduction ratio is greater than the set ratio threshold (e.g., 30%)) and the fingers are quickly bent and clenched into a fist, and the center position of the palm is within a spherical range with a radius of r (e.g., 0.5 meters) centered on the driver's chest, it is determined to be a "clenched fist" operation; when the palm area increases rapidly and the fingers are straightened, and the palm position is within the above range, it is determined to be an "open palm" operation.

[0138] Next, after determining the scene conditions and preset gesture actions, one or more corresponding preset gesture actions can be configured for various scene conditions, thereby obtaining one or more groups of control conditions, and establishing a mapping relationship between each group of control conditions and the corresponding control instructions. Furthermore, based on the multiple groups of control conditions and the mapping relationship corresponding to each group of control conditions, the rule base is constructed.

[0139] As a specific example, the content of the rule library may include the following scene conditions, preset gesture actions, and control instructions.

[0140] Scenario condition 1, highway driving scenario (6:00 - 22:00), the preset gesture actions and control instructions corresponding to scenario condition 1 include: ① Wave horizontally to the left: turn on the left turn signal. ② Wave horizontally to the right: turn on the right turn signal. ③ Wave vertically downward: reduce the speed by 5km / h. ④ Rotate your finger clockwise (index finger straight): fine-tune the air conditioning temperature (increase 1.C each time) or the volume (increase 2 grids each time).

[0141] Scenario 2, urban congestion scenario (7:00 - 9:00, 17:00 - 19:00), the preset gesture actions and control commands corresponding to scenario 2 include: ① Vertically wave upward: start the automatic following function (if the vehicle has this function). ② Make a fist: emergency brake assist (increase the braking force on the basis of the normal braking system). ③ Wave horizontally to the left + wave vertically downward: switch the recording mode of the driving recorder.

[0142] Scene condition 3, nighttime mountain driving scene (19:00 - 6:00), the preset gesture actions and control instructions corresponding to this scene condition 3 include: ① Waving horizontally to the right: turn on the high beam. ② Rotate the finger clockwise (index finger straight): switch to fog light mode.

[0143] Scenario 4: Parking lot and vehicle speed is less than the set threshold operation scenario (no specific time limit). The preset gesture actions and control instructions corresponding to scenario 4 include: ① Open your palm: open the trunk. ② Waving your hand vertically downward (small amplitude): start the reversing radar.

[0144] In the above embodiments of the present application, the driver can also redefine the control instructions corresponding to various hand movements in specific scenarios. For example, when driving on urban roads, the driver can remap "waving vertically downward" to opening the car window for ventilation, or when driving on a highway, the driver can remap "making a fist" to start the cruise control function. In other embodiments, the preset gesture action can also be a gesture combination, for example, a combination gesture of "waving vertically upward + waving horizontally to the right + rotating fingers clockwise (index finger straight)" is set, and the corresponding control instruction is to search and navigate to a preset common destination (such as a home address or workplace) in the navigation system. At the same time, the action requirements corresponding to the combined gesture can be defined as the action time interval within 1 second, and the action amplitude and speed of each gesture must meet the basic requirements of the corresponding scene. In addition, the driver can set the custom gesture combination to take effect in a specific time period (such as working hours on weekdays) or a specific geographical fence area (such as within 1 km around the company) to increase the flexibility and safety of the operation; more complex function customization can also be performed in combination with environmental conditions and preset gesture actions. For example, when the vehicle is driving in the rain (the rainfall is greater than a certain value, such as rainfall greater than 2 mm per minute, as determined by the on-board rain sensor), the driver can map the gesture of "rotating fingers counterclockwise (index finger straightened) + waving vertically downward" to the control command "turn on the wipers and adjust to fast wiping mode"; when the temperature inside the vehicle is higher than the set value (determined by the temperature sensor inside the vehicle) and the vehicle is parked, the gesture of "making a fist + waving horizontally to the left" is mapped to the control command "start the vehicle ventilation system and set it to maximum ventilation mode".

[0145] Through the rule base construction method of the above embodiment, by fully combining the vehicle's precise location (through high-precision GPS positioning and map data), time, environmental conditions (including light, weather, temperature, road conditions, surrounding vehicle conditions, etc.) and complex characteristics of gestures (combination, angle, position, speed, duration, etc.), multi-scenario, multi-functional deep orchestration and personalized adjustment can be achieved. Based on the rule base, deep orchestration of multiple scene conditions, multiple preset gesture actions, and multiple control instructions can be achieved to meet the complex needs of different users in multiple scenarios.

[0146] In one embodiment, after the target control instruction is determined based on the above embodiment, the specific situation of controlling the vehicle to make a corresponding response based on the target control instruction may include:

[0147] Determine the constraint conditions corresponding to the target control instruction; the constraint conditions include at least one of the vehicle speed constraint condition and the interval time constraint of the last execution of the same type of control instruction; if there is no corresponding situation of the constraint condition at present, control the vehicle to make a corresponding response based on the target control instruction; otherwise, do not execute the target control instruction.

[0148] Based on the vehicle control method of the embodiment of the present application, in order to reduce misjudgment, on the one hand, multi-parameter thresholds such as speed, displacement, and angle of gesture operation are set. When the detected gesture action does not fully meet the set conditions, it will not be determined to match the preset gesture in the rule library, and thus the relevant control instructions will not be triggered. On the other hand, corresponding constraints can also be set for the target control instructions. Before matching the target control instructions and executing the target control instructions, it is necessary to determine whether there are corresponding constraints. If so, the target control instructions will not be executed to ensure vehicle safety and driving experience.

[0149] In one embodiment, the constraints of the operation instructions corresponding to different control conditions (scene conditions + preset gesture actions) may also be different. For example:

[0150] The above scenario condition 1, highway driving scenario (6:00 - 22:00):

[0151] Waving horizontally to the left: Turn on the left turn signal. The corresponding constraint condition can be that when the horizontal displacement of the hand is greater than 0.3 meters (speed greater than 5 meters / second) and the angle between the gesture and the front of the driver's body is within the first range, it is recognized as matching the preset gesture action of waving horizontally to the left.

[0152] Waving right horizontally: turn on the right turn signal. The corresponding constraint condition can be that when the horizontal displacement of the hand is greater than 3 meters (the displacement speed is greater than 5 meters / second) and the angle between the gesture and the front of the driver's body is within the second range, it is recognized as matching the preset gesture action of waving right horizontally.

[0153] Waving vertically downward: reduce the vehicle speed by 5 km / h. The corresponding constraint condition may be that when the vertical displacement is greater than 0.5 meters and the angle between the gesture and the front of the driver's body is within the third range, it is recognized as matching the preset gesture action of waving vertically downward.

[0154] Rotate your finger clockwise (with your index finger extended): fine-tune the air conditioning temperature (increase by 1°C each time) or the volume (increase by 2 levels each time). The corresponding constraint condition can be that only when the rotation speed is above 45 degrees per second and the rotation angle is greater than the set angle threshold, it is recognized as matching the preset gesture action of rotating your finger clockwise.

[0155] Under the control conditions of the above-mentioned scenario condition 1 and the preset gesture action combination, accident avoidance measures may also include, for example, a minimum speed limit for vehicle speed adjustment operations (not less than 60km / h) and a certain time interval limit for operations (such as seconds between two speed reduction operations).

[0156] The above scenario condition 2, urban congested traffic scenario (7:00 - 9:00, 17:00 - 19:00):

[0157] Waving vertically upwards: Activate the automatic following function (if the vehicle has this function). The corresponding constraint conditions can be that the vertical displacement is greater than 0.1 meters, the speed is about 0.2 meters per second, and the angle between the gesture and the front of the driver's body is within the fourth range, and it is recognized as matching the preset gesture action of waving vertically upwards.

[0158] Fist clenching: Emergency brake assist (increases braking force on the basis of the normal braking system). The corresponding constraint condition can be that the center of the palm is within a spherical range with a radius of 0.4 meters centered on the driver's chest, and the grasping action is completed in less than 0.4 seconds, which is recognized as matching the preset gesture action of fist clenching.

[0159] Waving left horizontally + waving downward vertically: Switch the recording mode of the dashcam. The corresponding constraint conditions can be that the action sequence and time interval are within 0.5 seconds, the horizontal displacement is >0.2 meters (speed >0.2 meters / second), and the vertical displacement is >0.1 meters, only then it is recognized as matching the preset gesture action.

[0160] Under the above scenario condition 2 and the control conditions of gesture action combination, the misjudgment handling and accident avoidance measures can also include: for example, a fist gesture requires multiple consecutive frames to meet the conditions before it is recognized as matching the preset gesture action. At the same time, the automatic follow-up start operation has a speed limit (less than 30km / h) to prevent accidental start when driving at high speed.

[0161] The above scenario condition 3, nighttime mountain driving scenario (19:00 - 6:00):

[0162] Waving horizontally to the right: turn on the high beam. The corresponding constraint conditions can be that the hand needs to maintain the waving action for more than 0.8 seconds, the speed is more than 0.3 meters per second, the angle between the gesture and the front of the driver's body is within the range, and the hand position is within the range of 0.3-0.6 meters in front of the driver's body, only then it is recognized as matching the preset gesture action.

[0163] Rotate your finger clockwise (with your index finger extended): switch to fog light mode. The corresponding constraint conditions may be that the rotation angle must be greater than 180 degrees and the speed must be above 20 meters per second to be recognized as matching the preset gesture action.

[0164] Under the above scenario condition 3 and the control conditions of the preset gesture action combination, the misjudgment handling and accident avoidance measures may also include: the operation of the high beam and fog lights has strict gesture duration and speed requirements to avoid false triggering due to short hand movements. At the same time, the system will detect the current light status of the vehicle. If the corresponding light is already on, it will not be triggered repeatedly to prevent interference to other drivers.

[0165] Scenario condition 4 above, parking lot operation scenario (no specific time limit):

[0166] Open your palm: open the trunk. The corresponding constraint condition can be that the distance is within the range of 0.2-0.5 meters, and the palm area increases by more than 50%, which is recognized as matching the preset gesture action.

[0167] Waving your hand vertically downward (with a small amplitude): activate the reversing radar. The corresponding constraint conditions can be that the vertical displacement is between 0.05-0.1 meters, the speed is about 0.1 meters per second, and the angle between the gesture and the front of the driver's body is within the range, and it is recognized as matching the preset gesture action.

[0168] Under the above scenario condition 4 and the control conditions of the preset gesture action combination, the misjudgment handling and accident avoidance measures may also include: For the trunk opening operation, there are strict requirements for the palm position and area change to prevent the trunk from being opened by mistake during other operations. The reversing radar activation operation is only effective when the vehicle speed is extremely low (less than 5km / h) to avoid accidental activation during normal driving.

[0169] Through the above-mentioned embodiments, constraint conditions corresponding to the target control instructions are added, that is, some avoidance measures are added, which can avoid potential dangers caused by automatically executing the control instructions and further ensure the safety of the vehicle.

[0170] In one embodiment, within a set time after the vehicle is controlled to make a corresponding response based on the determined target control instruction, if it is obtained that the object in the vehicle has made an adjustment operation based on the response, the number of adjustments corresponding to the current target control instruction is recorded; when the recorded number of adjustments is greater than the set number threshold, based on the target control instruction and the corresponding adjustment operation, the update information corresponding to the target control instruction is determined, and the information corresponding to the target control instruction in the rule library is updated based on the update information. The target control instruction itself can be updated, and the control condition corresponding to the target control instruction can also be updated.

[0171] As an example, based on the target control instruction and the corresponding adjustment operation, the update information corresponding to the target control instruction can be determined through machine learning or a large model. For example, when the adjustment information is a callback to the state before response, it indicates that the correspondence between the scenes, gestures, and control instructions in the rule library is no longer suitable for the current driver's operating habits. Therefore, the control conditions corresponding to the information combination of the current action information and the current scene information in the rule library can be deleted; when the adjustment information is an adjustment to other states, it indicates that the relevant control instructions in the rule library no longer meet the current driver's expectations, so the target control instruction itself in the rule library can be updated.

[0172] Through the above embodiments, an adaptive learning mechanism is introduced to continuously optimize the rules in the rule base based on the actual operation feedback of the driver. Based on the continuously updated rule base, the adaptability, intelligence level and user experience of the system can be sustainably improved.

[0173] In summary, the technical solutions obtained through the above embodiments and combinations of the embodiments at least bring about the following technical effects:

[0174] All-round multi-parameter gesture recognition technology: deeply integrates multi-dimensional parameters such as the driver's hand's three-dimensional displacement, speed, trajectory, duration, finger bending and extension state, the angle of the gesture relative to the driver's body, and the position of the hand in the three-dimensional space inside the car, and uses algorithms to achieve high-precision and high-confidence gesture recognition. It can accurately distinguish various complex gestures and action details, and provide a basis for the adaptation of the vehicle's multi-functional and refined control commands.

[0175] Multi-scenario, multi-functional deep rule base orchestration: Combining the vehicle's precise location (through high-precision GPS positioning and map data), time, environmental conditions (including light, weather, temperature, road conditions, surrounding vehicle conditions, etc.) and complex features of gestures (combination, angle, position, speed, duration, etc.), it can meet the diverse needs of multiple scenarios and multiple users.

[0176] Dynamic resource allocation and adaptive learning optimization mechanism: Introducing dynamic frame rate, resolution and gesture recognition sensitivity adjustment technology to optimize system resource utilization based on hand movements, scene complexity and recognition confidence. At the same time, the system has the ability to adaptively learn the driver's operating habits, and continuously optimizes the gesture recognition algorithm and rule base by collecting and analyzing the driver's operating data, and continuously improves system performance.

[0177] Complex gesture combinations and environmental combinations as control conditions: Supports the combination of multiple complex gestures, and associates gestures with a variety of environmental conditions to achieve more complex vehicle function control that is more in line with actual driving needs. It breaks through the limitations of traditional single gesture control and provides drivers with more abundant operation options and a smarter interactive experience.

[0178] Seamless integration technology with existing vehicle systems: It can be seamlessly integrated with existing driver monitoring systems (DMS), vehicle electronic control units (ECUs), various sensors (such as light sensors, speed sensors, rain sensors, etc.) and other vehicle equipment (such as navigation systems, audio systems, air conditioning systems, lighting systems, etc.). Through standardized interfaces and communication protocols, it ensures smooth data interaction between systems, fully utilizes existing hardware resources, avoids additional hardware investment and compatibility issues, and reduces system development and deployment costs. At the same time, this integration method ensures that new interactive control functions are expanded without affecting the original vehicle functions, thereby improving the overall performance and value of the entire vehicle system.

[0179] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0180] Based on the same inventive concept, the embodiment of the present application also provides a vehicle control device for implementing the vehicle control method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more vehicle control device embodiments provided below can refer to the limitations on the vehicle control method above, and will not be repeated here.

[0181] In an exemplary embodiment, Figure 6 As shown, a vehicle control device is provided, comprising:

[0182] The information acquisition module 601 is used to acquire the current scene information of the vehicle and the image of the object in the vehicle when the target control mode of the vehicle is turned on; the current scene information includes at least one of the current time, the current position, the current vehicle state parameter, and the current environment information;

[0183] An image recognition module 602 is used to determine the current motion information of the object in the vehicle based on the image;

[0184] The instruction determination module 603 is used to determine the target control instruction based on the current action information, the current scene information and a predefined rule base; the rule base stores multiple groups of control conditions and control instructions corresponding to each group of control conditions, and each group of control conditions is a combination of an action condition and a scene condition; the information combination of the current action information and the current scene information matches the control condition corresponding to the target control instruction;

[0185] The instruction execution module 604 is used to control the vehicle to make a corresponding response based on the target control instruction.

[0186] The vehicle control device of the above embodiment, when the target control mode of the vehicle is turned on, obtains the current scene information of the vehicle and the image of the object in the vehicle through the information acquisition module, and determines the current action information of the object in the vehicle by analyzing the image through the image recognition module; the instruction determination module adaptively determines the target control instruction that matches both the current scene and the current action based on the acquired current action information of the object in the vehicle, the current scene information of the vehicle and the predefined rule base, and the instruction execution module controls the vehicle to make a corresponding response based on the target control instruction. Based on the vehicle control device, the user in the vehicle does not need to operate physical buttons and touch buttons, which improves the convenience of vehicle control during driving; at the same time, because the target control instruction is determined based on the combination of the current scene of the vehicle and the current action of the object in the vehicle, and is a control condition already in the predefined rule base, thus, on the basis of improving convenience and intelligent control, it is also conducive to ensuring vehicle driving safety.

[0187] In one embodiment, the information acquisition module 601 is specifically used to:

[0188] When the target control mode of the vehicle is turned on, the current scene information of the vehicle is obtained; the frame rate and / or resolution of the driver monitoring system of the vehicle is determined based on the current scene information; the driver monitoring system is controlled to obtain an image of the object inside the vehicle based on the frame rate and / or resolution; the image is an image containing hand information of the object inside the vehicle.

[0189] In one embodiment, the above-mentioned device also includes a scene complexity identification module, which is used to: determine the scene complexity corresponding to the current scene information based on a preset scene complexity distinction standard; based on a preset mapping table, obtain the frame rate and / or resolution corresponding to the scene complexity as the frame rate and / or resolution of the driver monitoring system; in the mapping table, there is a positive correlation between the scene complexity and the frame rate and / or resolution.

[0190] In one embodiment, the image is an image including hand information of an object in the vehicle;

[0191] The image recognition module 602 is specifically used for: graying the image to obtain a grayscale image with the goal of reducing the amount of image data and retaining the hand information therein; performing background subtraction processing on the grayscale image with the area where the hand information is located as the foreground to obtain a background-processed image; identifying hand key points based on at least two consecutive frames of background-processed images, and determining the current hand motion information of the object in the car based on the hand key points; the hand key points include finger key points, palm key points and wrist key points.

[0192] In one embodiment, each group of control conditions in the rule base is a combination of preset gesture actions and scene conditions; the instruction determination module 603 is specifically used to:

[0193] Match the current scene information with the scene conditions contained in each group of control conditions in the rule library to determine the candidate scene conditions that match the current scene information; compare the current hand movement information with the preset gesture actions corresponding to the candidate scene conditions; when the current hand movement information belongs to the preset gesture actions corresponding to the candidate scene conditions, obtain the control instructions corresponding to the control conditions of the combination of the candidate scene conditions and the preset gesture actions as the target control instructions.

[0194] In one embodiment, the instruction execution module 604 is specifically used to:

[0195] Determine the constraint conditions corresponding to the target control instruction; the constraint conditions include at least one of the vehicle speed constraint condition and the interval time constraint of the last execution of the same type of control instruction; if there is no corresponding situation of any constraint condition at present, control the vehicle to make a corresponding response based on the target control instruction; otherwise, do not execute the target control instruction.

[0196] In one embodiment, the aforementioned device further includes an updating module, which is used to:

[0197] Within a set time after the vehicle is controlled to make a corresponding response based on the target control instruction, if an adjustment operation is obtained from the object in the vehicle in response to the response, the number of adjustments corresponding to the target control instruction is recorded; when the number of adjustments is greater than the set number threshold, based on the target control instruction and the adjustment operation, the update information corresponding to the target control instruction is determined; based on the update information, the information corresponding to the target control instruction in the rule library is updated.

[0198] Each module in the above vehicle control device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the new energy vehicle in the form of hardware, or can be stored in the memory in the new energy vehicle in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0199] In an exemplary embodiment, a new energy vehicle is provided, and its internal structure diagram can be shown in FIG7. The new energy vehicle includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the new energy vehicle is used to provide computing and control capabilities. The memory of the new energy vehicle includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the new energy vehicle is used to exchange information between the processor and the external device. The communication interface of the new energy vehicle is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a vehicle control method is implemented. The display unit of the new energy vehicle is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the new energy vehicle can be a touch layer covered on the display screen, or it can be a button, trackball or touchpad set in the new energy vehicle.

[0200] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the new energy vehicle to which the scheme of the present application is applied. A specific new energy vehicle may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0201] In an exemplary embodiment, a new energy vehicle is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned vehicle control method embodiments when executing the computer program.

[0202] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0203] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0204] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0205] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A vehicle control method, characterized in that: include: When the target control mode of the vehicle is turned on, obtaining the current scene information of the vehicle and the image of the object inside the vehicle; The current scene information includes at least one of the current time, current location, current vehicle state parameters, and current environment information; determining current motion information of the object in the vehicle based on the image; Determine a target control instruction based on the current action information, the current scene information and a predefined rule base; the rule base stores multiple groups of control conditions and control instructions corresponding to each group of control conditions, and each group of control conditions is a combination of preset action and scene conditions; the information combination of the current action information and the current scene information matches the control condition corresponding to the target control instruction; Based on the target control instruction, the vehicle is controlled to make a corresponding response.

2. The method according to claim 1, characterized in that The method of obtaining the current scene information of the vehicle and the image of the object in the vehicle when the target control mode of the vehicle is turned on includes: When the target control mode of the vehicle is turned on, obtaining the current scene information of the vehicle; determining a frame rate and / or resolution of a driver monitoring system of the vehicle based on the current scene information; The driver monitoring system is controlled to obtain an image of the in-vehicle object based on the frame rate and / or resolution; the image is an image containing hand information of the in-vehicle object.

3. The method according to claim 2, characterized in that The determining the frame rate and / or resolution of the driver monitoring system based on the current scene information includes: Determining the scene complexity corresponding to the current scene information based on a preset scene complexity distinction standard; Based on a preset mapping table, the frame rate and / or resolution corresponding to the scene complexity is obtained as the frame rate and / or resolution of the driver monitoring system; in the mapping table, there is a positive correlation between the scene complexity and the frame rate and / or resolution.

4. The method according to claim 1, characterized in that: The image is an image containing hand information of the object in the vehicle; The determining the current motion information of the object in the vehicle based on the image includes: To reduce the amount of image data and retain the hand information therein, grayscale processing is performed on the image to obtain a grayscale image; Taking the area where the hand information is located as the foreground, performing background subtraction processing on the grayscale image to obtain an image after background processing; Based on at least two consecutive frames of the background-processed image, hand key points are identified, and current hand motion information of the object in the vehicle is determined based on the hand key points; the hand key points include finger key points, palm key points and wrist key points.

5. The method according to claim 4, characterized in that Each group of control conditions in the rule base is a combination of preset gesture actions and scene conditions; The determining of the target control instruction based on the current action information, the current scene information and a predefined rule base includes: Matching the current scene information with the scene conditions included in each group of control conditions in the rule base to determine candidate scene conditions matching the current scene information; Comparing the current hand motion information with a preset hand gesture corresponding to the candidate scene condition; In the case that the current hand motion information belongs to a preset gesture action corresponding to the candidate scene condition, a control instruction corresponding to a control condition of a combination of the candidate scene condition and the preset gesture action is obtained as the target control instruction.

6. The method according to any one of claims 1 to 5, characterized in that: The controlling the vehicle to make a corresponding response based on the target control instruction includes: Determine the constraint condition corresponding to the target control instruction; the constraint condition includes at least one of a vehicle speed constraint condition and an interval time constraint of the last execution of the same type of control instruction; If there is no situation corresponding to any of the constraint conditions at present, the vehicle is controlled to make a corresponding response based on the target control instruction; otherwise, the target control instruction is not executed.

7. The method according to any one of claims 1 to 5, characterized in that: Also includes: Within a set time after the vehicle is controlled to make a corresponding response based on the target control instruction, if an adjustment operation is obtained by the in-vehicle object in response to the response, the number of adjustments corresponding to the target control instruction is recorded; In a case where the number of adjustments is greater than a set number threshold, determining update information corresponding to the target control instruction based on the target control instruction and the adjustment operation; Based on the update information, the information corresponding to the target control instruction in the rule base is updated.

8. A vehicle control device, characterized in that: include: An information acquisition module, used to acquire the current scene information of the vehicle and the image of the object in the vehicle when the target control mode of the vehicle is turned on; The current scene information includes at least one of the current time, current location, current vehicle state parameters, and current environment information; An image recognition module, used to determine current motion information of the object in the vehicle based on the image; An instruction determination module is used to determine a target control instruction based on the current action information, the current scene information and a predefined rule base; the rule base stores multiple groups of control conditions and control instructions corresponding to each group of control conditions, and each group of control conditions is a combination of an action condition and a scene condition; the information combination of the current action information and the current scene information matches the control condition corresponding to the target control instruction; The instruction execution module is used to control the vehicle to make a corresponding response based on the target control instruction.

9. A new energy vehicle, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images