A sensor and vision recognition based vehicle door anti-pinch method, system and vehicle

Abstract

The application relates to the technical field of vehicle door equipment, and discloses a vehicle door anti-pinch method and system based on sensors and visual recognition and a vehicle. The method comprises the following steps: acquiring a limb tracking image around a vehicle door through an imager; inputting the limb tracking image into a pre-trained neural network model to output limb state information; acquiring limb contact information around the vehicle door through a sensor; determining a limb pinching risk state according to the limb state information and the limb contact information; and generating an anti-pinch instruction to control the vehicle door to open or stop closing when the limb is in the pinching risk state. The method solves the problem of inaccurate anti-pinch process determination caused by mechanical sensor detection in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle door equipment, in particular to a vehicle door anti-pinch method and system based on sensors and visual recognition. BACKGROUND

[0002] At present, with the rapid development of automobile industry technology and intelligent control method, automobiles are widely used in life. In the process of using the vehicle, door pinch accidents occur from time to time, especially children and the elderly are easily pinched in the case of inattention, causing great harm and property loss.

[0003] In order to avoid the occurrence of door pinch accidents, various door anti-pinch devices have emerged, but these devices usually use mechanical sensors and switches for sensing and control. The mechanical sensor cannot determine whether it is a human body part, and there is a problem of inaccurate judgment.

[0004] Therefore, the existing situation and technology need to be improved and developed. SUMMARY

[0005] In view of the above shortcomings of the prior art, the purpose of the present application is to provide a vehicle door anti-pinch method and system based on sensors and visual recognition, which solves the problem of inaccurate judgment in the anti-pinch process caused by mechanical sensor detection in the prior art.

[0006] The technical scheme of the present application is as follows:

[0007] On the one hand, the present application provides a vehicle door anti-pinch method based on sensors and visual recognition, comprising the steps of:

[0008] acquiring a limb tracking image around the door through an imager;

[0009] inputting the limb tracking image into a pre-trained neural network model to output limb state information;

[0010] acquiring limb contact information around the door through a sensor;

[0011] determining a limb pinch risk state according to the limb state information and the limb contact information;

[0012] when the limb is in the pinch risk state, generating an anti-pinch instruction to control the door to open or stop closing action.

[0013] According to the above technical means, the limbs around the door are recognized by using a neural network model, and the limb state information is obtained through visual recognition; the limb contact information around the door is obtained through the sensor; through the joint action of the limb state information obtained through the image recognition technology and the limb contact information obtained through the sensor, the environment around the door is detected, the limbs are more accurately recognized and tracked, and when it is detected that the limbs are in a risk state of being clamped, the door is accurately controlled to avoid clamping people or objects.

[0014] Further, in the step of inputting the limb tracking image into the pre-trained neural network model to output the limb state information, the training step of the neural network model comprises:

[0015] Collecting training data around the door;

[0016] Preprocessing the training data to obtain training standard image data;

[0017] Extracting useful target features from the training standard image data through a deep learning algorithm;

[0018] Labeling the limb state in the target feature, and converting the target feature into target data understandable by the neural network model;

[0019] Using the target data with limb state labels as a training set, and training the neural network model using a deep learning algorithm, wherein the limb state includes limb images and limb positions;

[0020] Using a test set to evaluate the trained neural network model to determine whether the accuracy and recall rate of the trained neural network model meet the preset standard.

[0021] According to the above technical means, the deep learning algorithm and the neural network model can improve the recognition accuracy of the limb image, the accuracy of the judgment result of the trained neural network model is greatly improved, thereby improving the convenience and intelligence of the door, and the method has the advantages of high accuracy, high intelligence, high safety and wide applicability.

[0022] Further, the steps in the step of inputting the limb tracking image into the pre-trained neural network model to output the limb state information comprise:

[0023] Inputting the limb tracking image into the pre-trained neural network model, recognizing the limbs in the limb tracking image, and determining the position information of the limbs;

[0024] Processing the continuous frames of limb tracking images through a computer vision processing algorithm to capture dynamic change information of the limbs, wherein the dynamic change information of the limbs includes the movement direction and speed of the limbs.

[0025] According to the above technical means, the limb position and the dynamic change information of the limb can be recognized, so that the limb action process can be predicted, and the risk probability of the limb being clamped can be more accurately evaluated, and the accuracy of the visual recognition judgment structure is improved.

[0026] Further, in the step of acquiring the limb contact information around the door by the sensor:

[0027] The pressure change data around the door is acquired by the pressure sensor;

[0028] The temperature change data around the door is acquired by the thermal sensor;

[0029] According to the pressure change data and the temperature change data, the limb contact information is obtained.

[0030] According to the above technical means, the force sensor can measure the pressure applied by the object, when the limb contacts the door, a certain pressure will be applied on the sensor, by detecting and measuring the pressure change, it can be judged whether there is a limb contacting the door; and the thermal sensor can measure the temperature change of the object, when the limb contacts the door, it will affect the temperature around the sensor, by detecting and measuring the temperature change, it can be judged whether there is a limb contacting the door. The contact behavior of the limb is perceived by two kinds of sensors, and is used in cooperation with visual recognition, so as to more accurately perceive the environment and objects around the door.

[0031] Further, according to the limb state information and the limb contact information, the step of determining the limb clamping risk state comprises:

[0032] The judgment result based on the limb state information and the judgment result based on the limb contact information are assigned a weight ratio, and the limb clamping risk state is determined.

[0033] According to the above technical means, the judgment result of visual recognition and the judgment result of the sensor are respectively set to a certain weight ratio, and the output of visual recognition and the output of other sensors are weighted and comprehensively judged to obtain the final hand position judgment result.

[0034] Further, the step of assigning a weight ratio to the judgment result based on the limb state information and the judgment result based on the limb contact information to determine the limb clamping risk state comprises:

[0035] The limb state information is compared with the preset clamped condition limb information to obtain a first limb clamping probability value a1;

[0036] The limb contact information is compared with the preset clamped condition contact information to obtain a second limb clamping probability value a2;

[0037] respectively, the first limb clamping probability value is assigned a first weight coefficient K1, and the second limb clamping probability value is assigned a second weight coefficient K2, wherein K1+K2=1;

[0038] The judgment result F=a1*K1+a2*K2 is compared with a preset threshold value;

[0039] When F is greater than or equal to the preset threshold value, the limb is in a clamping risk state; when F is less than the preset threshold value, the limb is not in a clamping risk state.

[0040] According to the above technical means, the comprehensive evaluation of the judgment result is realized, and more accurate judgment is made on whether the limb is in a clamping risk state.

[0041] Further, the step of respectively assigning the first weight coefficient K1 to the first limb clamping probability value and the second weight coefficient K2 to the second limb clamping probability value further comprises:

[0042] According to different working conditions, the values of the first weight coefficient K1 and the second weight coefficient K2 are adjusted.

[0043] According to the above technical means, in different working conditions, the judgment results of visual recognition and sensors have different biases, so the values of the first weight coefficient K1 and the second weight coefficient K2 are adjusted according to different working conditions, so that the weight distribution is more reasonable, and more accurate evaluation of the limb in a clamping risk state is realized.

[0044] Further, in the step of respectively assigning the first weight coefficient K1 to the first limb clamping probability value and the second weight coefficient K2 to the second limb clamping probability value:

[0045] The first weight coefficient and the second weight coefficient are determined by comparing the model performance under different weight ratios or using machine learning algorithm for automatic learning.

[0046] According to the above technical means, the best weight is determined by comparing the model performance under different weight ratios or using machine learning algorithm for automatic learning. Thus, the accuracy of the judgment of the limb in a clamping risk state is improved.

[0047] Further, according to the limb state information and the limb contact information, the step of identifying that the limb is in a clamping risk state further comprises the step of:

[0048] Through the use data in the verification set or actual scene, the evaluation result of identifying that the limb is in a clamping risk state is obtained, and the evaluation result includes accuracy and reliability;

[0049] According to the evaluation result, the weight ratio of the first weight coefficient and the second weight coefficient is adjusted, the deep learning model is changed, and the method of changing the sensor to obtain the body contact information around the door is changed, so as to iterate the process of identifying that the body is in the risk of being clamped for multiple times.

[0050] According to the above technical means, the accuracy and reliability of the body being in the risk of being clamped are evaluated through verification set or actual scene test, and the adjustment is made according to the evaluation result, so as to further improve the accuracy of anti-clamping, so as to realize multiple iterations and optimization of the method, so as to find the best weight ratio and comprehensive judgment method, so as to ensure that the expected effect can be achieved in actual application.

[0051] In a second aspect, the present application further provides a door anti-pinch system based on sensor and visual recognition, which comprises:

[0052] An image acquisition module is configured to acquire body tracking images around the door through an imager;

[0053] A body state information acquisition module is configured to input the body tracking images into a pre-trained neural network model and output body state information;

[0054] A body contact information acquisition module is configured to acquire body contact information around the door through a sensor;

[0055] A clamped risk state recognition module is configured to determine the clamped risk state of the body according to the body state information and the body contact information;

[0056] A door control module is configured to generate an anti-pinch instruction to control the door to open or stop closing when the body is in the clamped risk state.

[0057] In a third aspect, the present application further provides a vehicle, which comprises an imager, a sensor, a door control unit and a host controller.

[0058] The host controller is communicatively connected to the imager, the sensor and the door control unit, and is configured to implement the door anti-pinch method based on sensor and visual recognition as described above.

[0059] The present application has the following advantages:

[0060] (1) The neural network model is used to identify the limbs around the door, and the limb state information is obtained through the visual recognition function. The limb contact information around the door is obtained through the sensor. Through the joint action of the limb state information obtained by the image recognition technology and the limb contact information obtained by the sensor, the environment around the door is detected, the limbs are more accurately identified and tracked, and when the limbs are detected in the risk state of being clamped, the door is accurately controlled to avoid the door clamping or clamping. Thus, the intelligent control of the door anti-pinch can be realized, the accuracy and reliability of the door anti-pinch are greatly improved, and the complex traffic environment and driving state can be adapted.

[0061] (2) Visual recognition is used for main judgment, and sensors are used for auxiliary judgment. Even if the sensor is mis-triggered and damaged, the judgment of the limb in the risk state of being clamped can be made through visual recognition, which has the advantages of high intelligence, high safety, and wide applicability. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 A flowchart of main steps of a door anti-pinch method based on sensors and visual recognition of an embodiment of the present application;

[0063] Figure 2 A flowchart of detailed steps of a door anti-pinch method based on sensors and visual recognition of an embodiment of the present application;

[0064] Figure 3 A flowchart of the training process of a neural network model in a door anti-pinch method based on sensors and visual recognition of an embodiment of the present application;

[0065] Figure 4 A circuit principle block diagram of a vehicle of an embodiment of the present application.

[0066] In the figure, 100, imager; 200, sensor; 210, pressure sensor; 220, thermal sensor; 300, door control unit; 400, main controller; 500, door state detector; 600, in-vehicle environment detector; 700, human-computer interaction module. DETAILED DESCRIPTION

[0067] The embodiments of the present application will be described below with reference to the accompanying drawings and preferred embodiments, and other advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure in the specification. The present application can also be implemented or applied by different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustration of the present application, but not for limitation of the protection scope of the present application.

[0068] The existing mechanical sensor is usually installed on the vehicle door, which has the defects of inaccurate detection, easy mis-triggering and damage, and poor stability. To solve the above problems, the following embodiments are proposed in the present application:

[0069] Embodiment one

[0070] As shown in Figure 1 , Figure 2 , the present embodiment proposes a sensor and visual recognition based vehicle door anti-pinch method, which is applied to a vehicle and can realize anti-pinch protection of a limb, which can be a human hand or a human foot. The present embodiment mainly takes a human hand as an example for illustration. The method mainly includes the following steps:

[0071] Step S100, acquiring a limb tracking image around the vehicle door through an imager.

[0072] In the specific process, the camera provided on the vehicle can collect images or videos around the vehicle door in real time.

[0073] Step S200, inputting the limb tracking image into a pre-trained neural network model to output limb state information, wherein the limb state information includes limb position and limb dynamic change state.

[0074] The trained neural network model can directly obtain the position of the human hand and the dynamic change state of the human hand, which can improve the recognition accuracy of the human hand image. The accuracy of the judgment result of the trained neural network model is greatly improved, thereby improving the convenience and intelligence of the vehicle door use, and having the advantages of high accuracy, high intelligence, high safety and wide applicability.

[0075] As shown in Figure 3 , in the specific process, in step S200, the neural network model needs to be pre-trained, and the training steps of the neural network model specifically include:

[0076] Step S211, collecting training data around the vehicle door, wherein the training data includes image or / and video data.

[0077] Step S212, pre-processing the training data to obtain training standard image data.

[0078] In the specific process, the collected training data is subjected to image noise reduction, background interference removal and image size adjustment operations, thereby obtaining training standard image data that can be used for neural network training.

[0079] Step S213, extracting useful target features from the training standard image data through a deep learning algorithm.

[0080] In the specific process, the convolutional neural network (CNN) is used for feature extraction of the hand, the target features are extracted from the pre-trained standard image data, the target features are useful for neural network training, and the shape, color, texture and other information of the hand are captured, and the shape, color, texture and other features of the hand are recognized and tracked. The convolutional neural network (CNN) model is used for feature extraction and recognition of the hand to improve the accuracy and robustness.

[0081] In step S214, the limb state in the target feature is labeled, and the target feature is converted into target data understandable by the neural network model.

[0082] In the specific process, the target feature is converted into a vector or matrix form, which is the target data understood by the neural network model, and the target data is used as the training data of the subsequent neural network model.

[0083] In step S215, the target data with the limb state label is used as the training set, and the neural network model is trained using the deep learning algorithm, wherein the limb state includes the limb image and the limb position.

[0084] In the specific process, the hand image and the hand position are labeled in the plurality of target data in the training set, the labeled data is used as the training set, the neural network model is trained by the deep learning algorithm, and the convolutional neural network is used for supervised learning.

[0085] In step S216, the trained neural network model is evaluated using the test set, and the accuracy and recall rate of the trained neural network model are determined to reach the preset standard.

[0086] In the specific process, the trained neural network model is continuously updated and improved to adapt to different limb shapes and dynamic changes. For example, more recognition features (more hand features are labeled) can be added, such as the joint angle of the fingers and the area of the palm, to improve the recognition accuracy and robustness.

[0087] As shown in Figure 1 , Figure 2 In step S220, the limb tracking image is input into the pre-trained neural network model, the limb in the limb tracking image is recognized, and the position information of the limb is determined.

[0088] In step S230, the limb tracking image with continuous frames is processed by the computer vision processing algorithm to capture the dynamic change information of the limb.

[0089] In the specific process, the hand and hand position information output by the neural network is continuously tracked, so that the action and trajectory of the hand can be predicted, and the positional relationship between the hand and the door and the risk of being pinched can be judged. The dynamic change information of the hand includes the movement direction and speed of the hand. By identifying the hand position and the dynamic change information of the hand, the hand action process can be predicted, so that the risk probability of the hand being pinched can be more accurately evaluated, and the accuracy of the visual recognition and judgment structure can be improved.

[0090] As shown in Figure 1 , Figure 2 Step S300, the limb contact information around the door is obtained through the sensor.

[0091] In the specific process, in order to further improve the accuracy of the judgment, the limb contact information can be obtained by the pressure sensor, the thermal sensor or the infrared sensor to obtain various monitoring data around the door. In order to more accurately perceive the environment and objects around the door, the positional relationship between the hand and the door is judged by the limb contact information, and the risk probability of the hand being pinched is evaluated.

[0092] Step S310, the pressure change data around the door is obtained by the pressure sensor, and the pressure sensor is arranged at the door edge or the door handle position.

[0093] In the specific process, the force sensor can measure the pressure applied by the object, and the pressure sensor can be placed at the door edge or the door handle position around the door. When the hand contacts the door, a certain pressure will be applied on the sensor. By detecting and measuring the pressure change, it can be judged whether the hand contacts the door.

[0094] Step S320, the temperature change data around the door is obtained by the thermal sensor, and the thermal sensor is installed at the door handle position.

[0095] In the specific process, the thermal sensor can measure the temperature change of the object, and the thermal sensor can be installed at the door handle position around the door. When the hand contacts the door, it will affect the temperature around the sensor. By detecting and measuring the temperature change, it can be judged whether the hand contacts the door.

[0096] Step S330, the limb contact information is obtained according to the pressure change data and the temperature change data.

[0097] Specifically, the contact behavior of the human hand is assisted to be perceived by two sensors, and is used in cooperation with visual recognition, so as to more accurately perceive the environment and objects around the vehicle door. The various sensors in the embodiment are all auxiliary to perceive the contact behavior of the hand, and are not directly used to identify the position and dynamic change of the hand, but play a role in assisting to evaluate the risk probability of the hand being pinched. The pressure sensor and the thermal sensor can be used in combination with computer vision recognition to provide more comprehensive environmental perception and object recognition capabilities.

[0098] Step S400, determining the limb pinching risk state according to the limb state information and the limb contact information.

[0099] In a specific process, the judgment result based on the limb state information and the judgment result based on the limb contact information are respectively assigned a weight proportion, and the limb pinching risk state is determined according to the judgment result after the weight proportion is assigned. According to the weight proportion, the output of the visual recognition process and the output of the other sensors are weighted and averaged or other integrated manner to obtain the final hand position judgment result. The limb state information and the limb contact information are combined to evaluate the limb pinching risk, and the accuracy of the judgment is improved.

[0100] The limb pinching risk state includes two forms of being in a pinching risk state and not being in a pinching risk state.

[0101] In the following specific process, a weight evaluation method is provided, and the implementation mode of step S400 includes the following steps:

[0102] Step S410, comparing the limb state information with the preset pinching condition limb information to obtain a first limb pinching probability value a1.

[0103] In a specific process, the possible limb state information is classified, and a probability value (first limb pinching probability value) is set for each class. The pinching condition limb information is formed after presetting. When the visual recognition process outputs the limb state information, the pinching condition limb information is matched, so that the first limb pinching probability value a1 can be matched.

[0104] Step S420, comparing the limb contact information with the preset pinching condition contact information to obtain a second limb pinching probability value a2.

[0105] In a specific process, the possible limb contact information is classified, and a probability value (second limb pinching probability value) is set for each class. The pinching condition contact information is formed after presetting. When the sensor recognition process outputs the limb contact information, the pinching condition contact information is matched, so that the second limb pinching probability value a2 can be matched.

[0106] Step S430, respectively, the first limb is clamped probability value is given the first weight coefficient K1, the second limb back clamping probability value is given the second weight coefficient K2, wherein K1+K2=1.

[0107] In the specific process, since the main recognition process is visual recognition through a neural network, K2 in the embodiment is greater than K1, and in actual application, K1 can be 0.7-0.8. The sensor plays an auxiliary role.

[0108] The first weight coefficient K1 and the second weight coefficient K2 can be adjusted. There are several ways to adjust the process:

[0109] The first way is that the first weight coefficient and the second weight coefficient are determined by comparing the model performance under different weight ratios or using machine learning algorithm for automatic learning.

[0110] By comparing the model performance under different weight ratios or using machine learning algorithm for automatic learning to determine the best weight, the accuracy of judging the limb in the risk state of being clamped is improved.

[0111] The second way is to adjust the values of the first weight coefficient K1 and the second weight coefficient K2 according to different working conditions.

[0112] In different working conditions, the judgment results of visual recognition and the judgment results of the sensor have different biases, so the values of the first weight coefficient K1 and the second weight coefficient K2 are adjusted according to different working conditions, so that the weight distribution is more reasonable, and a more accurate assessment of the limb in the risk state of being clamped is realized.

[0113] The specific process is: 1. According to the working condition scene classification: classify different working condition scenes, such as indoor scene, outdoor scene, low light scene, high light scene, etc. By classifying the working condition scene, different weight ratios can be set for each working condition scene. For example, in low light conditions, more reliance on sensor data may be needed, so the weight ratio of the sensor can be increased. 2. Dynamic adjustment: dynamically adjust the weight ratio according to the real-time scene information and environmental conditions in the working condition scene. For example, when the light condition changes are detected, the weight ratio can be automatically adjusted to adapt to the new environment. This can be achieved by using sensor data or other environmental perception methods. 3. Customization according to user feedback: collect user feedback and opinions in different scenes, and adjust the weight ratio accordingly. Users may have different needs and preferences in specific scenarios, and their feedback can provide valuable information about weight distribution.

[0114] Step S440, compare the judgment result F=a1*K1+a2*K2 with the preset threshold value; when F is greater than or equal to the preset threshold value, it is recognized that the limb is in a pinching risk state; when F is less than the preset threshold value, the limb is not in a pinching risk state.

[0115] The judgment result of visual recognition and the judgment result of the sensor are respectively set with a certain weight ratio, the output of visual recognition and the output of other sensors are weighted and then comprehensively judged to obtain a final hand position judgment result. More accurate judgment of whether the human hand is in a pinching risk state is realized. The above-mentioned weight ratio makes the sensor can be used in combination with computer vision technology, and provides more comprehensive environmental perception and object recognition capability.

[0116] Specific application examples are as follows:

[0117] The first weight coefficient K1=0.7, the second weight coefficient K2=0.3; the first limb pinching probability value in the preset pinching condition limb information is divided into multiple levels, for example, 10%-100%; and the second limb pinching probability value in the preset pinching condition contact information is divided into multiple levels, for example, 10%-100%. The preset threshold value is set to 45%. Therefore, when the first limb pinching probability value of visual recognition is 70%, even if the second weight coefficient recognized by the sensor is not used, the vehicle door stopping or opening condition can be triggered.

[0118] Therefore, even in the case that the sensor fails due to instability, the judgment can be made by using the above-mentioned method. The stability of the vehicle door anti-pinch system is improved.

[0119] Step S500, when the limb is in a pinching risk state, an anti-pinch instruction is generated to control the vehicle door to open or stop closing action.

[0120] If a human hand is detected close to the vehicle door, the main control unit will send an instruction to the vehicle door control unit to automatically open or stop closing action, so as to avoid pinching accident. When the human hand completely leaves the vehicle door area, the vehicle door is automatically closed.

[0121] Step S600, by verifying the set or actual scene use data, an evaluation result of recognizing that the limb is in a pinching risk state is obtained, and the evaluation result includes accuracy and reliability;

[0122] Step S610, according to the evaluation result, the weight ratio of the first weight coefficient and the second weight coefficient is adjusted, the deep learning model is changed, and the method of obtaining the limb contact information around the vehicle door by the sensor is changed, so as to iterate the process of recognizing that the limb is in a pinching risk state multiple times.

[0123] In a specific process, the accuracy and reliability of the hand being in a pinch risk state are evaluated through a verification set or testing in an actual scene, and adjustments are made according to the evaluation results to further improve the accuracy of pinch prevention, so as to realize multiple iterations and optimization of the method to find the best weight ratio and comprehensive judgment method to ensure that the expected effect can be achieved in actual application.

[0124] Embodiment Two

[0125] Based on the vehicle door pinch prevention method in Embodiment One, this embodiment proposes a vehicle door pinch prevention system based on sensors and visual recognition, which includes an image acquisition module, a limb state information acquisition module, a limb contact information acquisition module, a pinch risk state recognition module, and a vehicle door control module. The image acquisition module is used to acquire limb tracking images around the vehicle door through an imager; the limb state information acquisition module is used to input the limb tracking images into a pre-trained neural network model to output limb state information, wherein the limb state information includes limb position and limb dynamic change state; the limb contact information acquisition module is used to acquire limb contact information around the vehicle door through a sensor; the pinch risk state recognition module is used to determine the limb pinch risk state according to the limb state information and the limb contact information; and the vehicle door control module is used to generate a pinch prevention instruction to control the vehicle door to open or stop closing action when the limb is in a pinch risk state. The vehicle door control module sends the pinch prevention instruction to the vehicle door control unit, and the vehicle door control unit controls the vehicle door to open or stop closing action according to the pinch prevention instruction.

[0126] Embodiment Three

[0127] As shown in Figure 4 , this embodiment also proposes a vehicle, which includes an imager 100, a sensor 200, a vehicle door control unit 300, and a host controller 400; the host controller 400 is communicatively connected to the imager 100, the sensor 200, and the vehicle door control unit 300, and is used to implement the vehicle door pinch prevention method based on the sensor 200 and visual recognition as described above.

[0128] The vehicle also includes a vehicle door state detector 500 for detecting the opening and closing state of the vehicle door and feeding back the state information to the user through the host controller 400 and a human-computer interaction module 700. In the implementation process of the vehicle door state detection, Hall sensors, photoelectric sensors, proximity switches, and other sensors can be used to detect the opening and closing state of the vehicle door. The human-computer interaction module 700 uses a multi-language human-computer interaction interface, so that the vehicle owner and passengers can interact with the vehicle door control system through voice instructions or touch screens, etc., to further improve the convenience and intelligent degree of use.

[0129] The vehicle further comprises an in-vehicle environment detector 600 for detecting parameters such as temperature, humidity, oxygen concentration, etc. of the in-vehicle environment, and feeding back to the user through the host controller 400 and the human-computer interaction module 700.

[0130] During the closing of the door, the data collected by the intelligent sensor 200 (pressure sensor 210, thermal sensor 220) is transmitted to the host controller 400 for processing. The host controller 400 uses deep learning algorithms and computer vision technology to analyze and process the data, determine whether there is a body around the door, and identify the position and dynamic changes of the body. If a body is detected close to the door, the host controller 400 will send instructions to the door control unit 300 to automatically open or stop the closing action of the door to avoid injury accidents. When the body completely leaves the door area, the door will automatically close.

[0131] The intelligent sensor 200 (pressure sensor 210, thermal sensor 220) and the host controller 400 used can be connected through a network to realize remote monitoring and control of the door anti-pinch device. In the process of realizing remote monitoring and control, the door anti-pinch device can be connected to a cloud server through a network connection to realize remote monitoring and control of the device, improving the convenience and intelligence of use.

[0132] In the specific implementation process: install the sensor 200 and imager 100 (camera) on the door, control the opening and closing state of the door through the door control unit 300; image collection and processing of the environment around the door are realized through the body recognition technology based on deep learning algorithm, to realize accurate recognition and tracking of the body; the auxiliary detection of the body position is realized through the sensor 200, when the body is detected close to the door, the door control unit 300 is used to determine whether the body enters the pinch area, if it enters, the movement of the door is immediately stopped to prevent injury; a variety of language voice prompts, graphical interfaces or touch screen interfaces are provided through the human-computer interaction module 700 to facilitate the user to operate and understand the state of the door; the data of the door state and the in-vehicle environment are processed and analyzed in the host controller 400, and the results are fed back to the user.

[0133] To sum up, the vehicle window light language control method and system and vehicle provided by the application obtain the limb state information through the visual recognition function; the limb contact information around the vehicle door is obtained through the sensor, more accurate recognition and tracking of the limb are realized, and the vehicle door is accurately controlled when the limb is detected to be in the risk state of being clamped, so that the risk of the vehicle door clamping people or objects is avoided. Thus, intelligent control of the vehicle door anti-clamping can be realized, the accuracy and reliability of the vehicle door anti-clamping are greatly improved, and the vehicle door anti-clamping can adapt to complex traffic environments and driving states. Visual recognition is mainly used for judgment, and the sensor is used for auxiliary judgment. Even if the sensor is mis-triggered or damaged, the judgment of the limb being in the risk state of being clamped can be performed through visual recognition, and the application has the advantages of high intelligence, high safety, and wide applicability.

[0134] The above embodiments are only preferred embodiments for fully illustrating the application, and the protection scope of the application is not limited thereto. Any equivalent replacement or transformation of the application made by those skilled in the art based on the application is within the protection scope of the application.

Claims

1. A method for preventing door pinching based on sensors and visual recognition, characterized in that, Including the following steps: The imager acquires images of the limbs around the car door; The limb tracking image is input into a pre-trained neural network model, which outputs limb state information. Sensors are used to acquire information about body contact around the car door; Based on the limb status information and the limb contact information, the risk status of limb being trapped is determined; When a limb is at risk of being pinched, an anti-pinch command is generated to control the opening or stop the closing action of the car door. The step of determining the risk status of limb being trapped based on the limb status information and the limb contact information specifically includes: assigning weight ratios to the judgment results based on the limb status information and the judgment results based on the limb contact information to determine the risk status of limb being trapped.

2. The door anti-pinch method based on sensors and visual recognition according to claim 1, characterized in that, In the step of inputting the limb tracking image into a pre-trained neural network model and outputting limb state information, the training steps of the neural network model include: Collect training data around the car doors; The training data is preprocessed to obtain training standard image data; Useful target features are extracted from the training standard image data using deep learning algorithms; The limb states in the target features are labeled, and the target features are transformed into target data that can be understood by the neural network model; The target data with limb status labels is used as the training set, and a neural network model is trained using a deep learning algorithm. The limb status includes limb images and limb positions. The trained neural network model is evaluated using a test set to determine if its accuracy and recall meet the preset standards.

3. The door anti-pinch method based on sensors and visual recognition according to claim 1, characterized in that, The steps in the step of inputting the limb tracking image into a pre-trained neural network model and outputting limb state information specifically include: The limb tracking image is input into a pre-trained neural network model to identify the limbs in the limb tracking image and determine the position information of the limbs; Computer vision processing algorithms are used to process continuous frames of images with limb tracking to capture dynamic change information of the limbs, including the direction and speed of limb movement.

4. The door anti-pinch method based on sensors and visual recognition according to claim 3, characterized in that, In the step of acquiring limb contact information around the car door through sensors: Pressure change data around the car door is acquired using pressure sensors; Temperature change data around the car door is obtained through thermal sensors; Based on pressure and temperature change data, limb contact information is obtained.

5. The door anti-pinch method based on sensors and visual recognition according to claim 1, characterized in that, The step of assigning weight ratios to the judgment results based on the limb state information and the judgment results based on the limb contact information to determine the risk state of limb being trapped specifically includes: The limb status information is compared with the preset limb information of the clamping condition to obtain the first limb clamping probability value a1. The limb contact information is compared with the preset clamping condition contact information to obtain the second limb clamping probability value a2. The probability value of the first limb being trapped is assigned a first weighting coefficient K1, and the probability value of the second limb being trapped is assigned a second weighting coefficient K2, where K1+K2=1; The judgment result F = a1*K1+ a2*K2 is compared with the preset threshold. When F is greater than or equal to the preset threshold, the limb is at risk of being pinched. When F is less than the preset threshold, the limb is not at risk of being pinched.

6. The door anti-pinch method based on sensors and visual recognition according to claim 5, characterized in that, After the steps of assigning a first weighting coefficient K1 to the probability value of the first limb being trapped and assigning a second weighting coefficient K2 to the probability value of the second limb being trapped, the method further includes: The values ​​of the first weighting coefficient K1 and the second weighting coefficient K2 are adjusted according to different working conditions. or In the steps of assigning a first weighting coefficient K1 to the probability value of the first limb being trapped and assigning a second weighting coefficient K2 to the probability value of the second limb being trapped: The first weighting coefficient and the second weighting coefficient are determined by comparing the model performance under different weighting ratios or by automatically learning using machine learning algorithms.

7. The vehicle door anti-pinch method based on sensors and visual recognition according to claim 5, characterized in that, Following the step of identifying a limb at risk of being trapped based on the limb state information and the limb contact information, the method further includes the following step: The assessment results of identifying a limb at risk of being trapped are obtained by using validation sets or usage data in real-world scenarios. The assessment results include accuracy and reliability. Based on the evaluation results, the weight ratio of the first and second weight coefficients is adjusted, the deep learning model is changed, and the method by which the sensor obtains limb contact information around the car door is changed, so as to iterate the process of determining whether a limb is in a state of risk of being trapped multiple times.

8. A vehicle door anti-pinch system based on sensors and visual recognition, characterized in that, include: An image acquisition module is used to acquire limb tracking images around the car door through an imager; The limb state information acquisition module is used to input the limb tracking image into a pre-trained neural network model and output limb state information. A limb contact information acquisition module, which is used to acquire limb contact information around the car door through sensors; A limb trapping risk state identification module is used to determine the limb trapping risk state based on the limb state information and the limb contact information, wherein a weight ratio is assigned to the judgment result based on the limb state information and the judgment result based on the limb contact information to determine the limb trapping risk state. The door control module is used to generate an anti-pinch command to control the door to open or stop closing when a limb is in a risk of being pinched.

9. A vehicle, characterized in that, include: Imager, sensors, door control unit, and main controller; The main controller is communicatively connected to the imager, the sensor, and the door control unit, and is used to implement the door anti-pinch method based on sensor and visual recognition as described in any one of claims 1-7.

Images