Wearing monitoring method of safety belt, electronic equipment and vehicle

Abstract

According to the safety belt wearing monitoring method, the electronic equipment and the vehicle provided by the invention, when the color similarity of the safety belt and the foreign matter of the driver is relatively high, the multi-mode verification data is adopted for cross verification, so that the monitoring problem caused by the same color of the safety belt and clothes is fundamentally solved, the safety belt wearing monitoring accuracy is greatly improved, and the safety belt wearing monitoring efficiency is improved. And misjudgment is effectively avoided. Multi-modal verification data are adopted for cross verification, heterogeneous multi-modal verification data are deeply fused and mutually backed up, even if identification of a single mode fails or is interfered, other modes can still provide effective information, the overall robustness of a wearing monitoring task is guaranteed, and the stability of full-scene and all-weather safety belt wearing monitoring is guaranteed. And after the wearing state of the safety belt is determined, graded early warning prompt is performed, so that a stiff alarm is changed into natural guidance under different monitoring results, and the user experience and safety are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of safety belt wearing monitoring, in particular to a safety belt wearing monitoring method, an electronic device and a vehicle. BACKGROUND

[0002] During vehicle driving, a safety belt is a primary passive safety device to protect the life safety of passengers. Mainstream safety belt monitoring technology mainly relies on physical sensors (such as a buckle switch, a pressure sensor) or a visual recognition system based on a camera. However, when the color of the safety belt is highly similar or completely consistent with the color of the clothes of the passenger, a pure visual recognition system is prone to misjudgment, resulting in false prompts, which seriously weakens the reliability and user trust of the monitoring system. SUMMARY

[0003] Therefore, the present application aims to provide a safety belt wearing monitoring method, an electronic device and a vehicle, which improve the accuracy of safety belt wearing identification in a color similar scene through multi-modal fusion safety belt wearing monitoring.

[0004] To achieve the above purpose, the present application provides a safety belt wearing monitoring method, comprising: determining a color similarity between a theoretical safety belt region and a clothes region according to a monitoring image of a safety belt wearing region; in response to the color similarity being greater than or equal to a preset first similarity threshold, acquiring multi-modal verification data, and determining a plurality of single modal determination results according to the monitoring image and the multi-modal verification data; determining a safety belt wearing state according to the plurality of determination results and the color similarity, and performing a hierarchical warning prompt according to the safety belt wearing state.

[0005] Optionally, the determining of the color similarity between the theoretical safety belt region and the clothes region according to the monitoring image of the safety belt wearing region comprises: identifying a first identified color of the theoretical safety belt region and a second identified color of a clothes background region in the monitoring image; performing similarity calculation on the first identified color and the second identified color to obtain the color similarity.

[0006] Optionally, the multi-modal verification data comprises thermal imaging data of the safety belt wearing region and pressure distribution data of a driver seat, and the determination result comprises a structure modal wearing state, a thermal imaging modal wearing state and a pressure modal wearing state; the determining of the plurality of single modal determination results according to the monitoring image and the multi-modal verification data comprises: extract visible light features from the theoretical seat belt region in the monitoring image to obtain edge continuity features and texture structure features, and determine the structure modal wearing state according to the edge continuity features and the texture structure features; determine a thermal distribution image and a temperature gradient map of the seat belt wearing region according to the thermal imaging data, and determine the thermal imaging modal wearing state according to the thermal distribution image and the temperature gradient map; determine a pressure distribution center and a pressure distribution uniformity according to the pressure distribution data, and determine the pressure modal wearing state according to the pressure distribution center and the pressure distribution uniformity.

[0007] Optionally, the determining the structure modal wearing state according to the edge continuity features and the texture structure features comprises: in response to the edge continuity features being edge continuity and the texture structure features being texture difference, determining that a seat belt is worn as the structure modal wearing state; in response to the edge continuity features being edge fracture or the texture structure features being no texture difference, determining that a seat belt is not worn as the structure modal wearing state.

[0008] Optionally, the determining the thermal imaging modal wearing state according to the thermal distribution image and the temperature gradient map comprises: in response to there being a peak-valley feature in the temperature gradient map, matching a target graph corresponding to a seat belt in the thermal distribution image according to graph similarity to obtain a graph matching result; in response to the graph matching result being that the target graph exists in the thermal distribution image, determining that a seat belt is worn as the thermal imaging modal wearing state; in response to the graph matching result being that the target graph does not exist in the thermal distribution image, determining that a seat belt is not worn as the thermal imaging modal wearing state.

[0009] Optionally, the determining the pressure modal wearing state according to the pressure distribution center and the pressure distribution uniformity comprises: determining a standard pressure center range and a standard uniformity range corresponding to a standard sitting posture; in response to the pressure distribution center existing in the standard pressure center range and the pressure distribution uniformity existing in the standard uniformity range, determining that a seat belt is worn as the pressure modal wearing state; in response to the pressure distribution center not existing in the standard pressure center range or the pressure distribution uniformity not existing in the standard uniformity range, determining that a seat belt is not worn as the pressure modal wearing state.

[0010] Optionally, the determining the safety belt wearing state according to the plurality of determination results and the color similarity comprises: in response to the color similarity being less than or equal to a preset second similarity threshold, determining a determination result as a result of wearing a full belt, wherein the second similarity threshold is greater than the first similarity threshold; determining a ratio of the result quantity to the total modal quantity; in response to the ratio being greater than a preset ratio threshold, determining that a safety belt is worn as the safety belt wearing state; in response to the ratio being less than or equal to the preset ratio threshold, determining that a safety belt is not worn as the safety belt wearing state.

[0011] Optionally, the determining the safety belt wearing state according to the plurality of determination results and the color similarity comprises: in response to the color similarity being greater than a preset second similarity threshold and the ratio being greater than a preset ratio threshold, determining a determination result as a target determination result of wearing a full belt; determining a confidence degree of each of the target determination results to obtain a confidence degree set; in response to a maximum confidence degree in the confidence degree set being greater than or equal to a preset confidence degree threshold, determining that a safety belt is worn as the safety belt wearing state; in response to the maximum confidence degree in the confidence degree set being less than the preset confidence degree threshold, determining that a safety belt is abnormally worn as the safety belt wearing state.

[0012] Based on the same inventive concept, the present application further provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable by the processor, wherein the processor implements the method as described above when executing the computer program.

[0013] Based on the same inventive concept, the present application further provides a vehicle comprising the electronic device as described above.

[0014] As can be seen from the above, the seat belt wearing monitoring method, electronic device, and vehicle provided in this application can determine the color similarity between the theoretical seat belt area and the clothing area based on the monitoring image of the seat belt wearing area; when the color similarity is greater than or equal to a preset first similarity threshold, multimodal verification data is acquired, and multiple single-modal judgment results are determined based on the monitoring image and multimodal verification data; the seat belt wearing status is determined based on multiple judgment results and color similarity, and graded warning prompts are given according to the seat belt wearing status. When the color similarity between the seat belt and the driver's foreign object is high, multimodal verification data is used for cross-verification, fundamentally overcoming the monitoring problem caused by the seat belt and clothing being the same color, significantly improving the accuracy of seat belt wearing monitoring, and effectively eliminating false judgments. By using multimodal verification data for cross-verification, heterogeneous multimodal verification data are deeply integrated and mutually backed up. Even if the recognition of a single modality fails or is interfered with, other modalities can still provide effective information, ensuring the overall robustness of the wearing monitoring task and ensuring the stability of seat belt wearing monitoring in all scenarios and all weather conditions. After determining the seatbelt wearing status, tiered warning prompts are provided, transforming rigid alarms into natural guidance based on different monitoring results, significantly improving user experience and safety. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a seatbelt wearing monitoring method according to an embodiment of this application; Figure 2 This is a schematic diagram of a seatbelt wearing monitoring device according to an embodiment of this application; Figure 3 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0018] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the embodiments of the present application shall have the common meaning understood by one of ordinary skill in the art to which the embodiments of the present application belong. The terms "first", "second", and similar terms used in the embodiments of the present application do not denote any order, quantity, or importance, but are merely used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms do not mean physical or mechanical connection, but can include electrical connection, whether direct or indirect. The terms "upper", "lower", "left", "right", and the like are merely used to represent relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships can also change accordingly.

[0019] In this document, it should be understood that any number of elements in the drawings are used for illustration only and not limitation, and any naming is merely for differentiation and does not have any limiting meaning.

[0020] Based on the description of the above background art, there are also the following cases in the related art: The safety belt monitoring technology in the related art mainly relies on physical sensors or camera-based visual recognition methods. The method of monitoring by physical sensors detects by installing sensors inside the vehicle. The most common is to install sensors (such as Hall sensors or micro switches) in the safety belt buckle, and to judge whether the safety belt is worn by detecting whether the lock tongue is inserted into the lock. This is the most basic and widely used method. The tension change when the safety belt is rolled back can also be detected by torque sensors, etc. When the safety belt is pulled out and worn, its rebound force will show a specific change sequence, and by comparing it with the preset reference sequence, the accuracy of the judgment can be improved.

[0021] The camera-based visual recognition method mainly uses cameras to take pictures and analyzes through image processing and pattern recognition technology. First, high-definition cameras are used to capture images of the driver and passengers in the vehicle. These cameras have high resolution and can clearly capture details. The next step is a key and complex step - accurately locating the area where the safety belt may be worn. Based on face or upper body detection, in the located window area, further use algorithms (such as Viola-Jones algorithm, deformable part model) to detect faces or upper bodies, and then according to the relative position, the safety belt detection area (ROI) is delimited. In the delimited area, advanced image analysis algorithms are used to identify the safety belt.

[0022] Detecting whether the lock tongue is inserted into the lock catch to determine whether the safety belt is worn can cause false recognition due to some inserted non-safety belt props, and when the safety belt color is highly similar or completely consistent with the color of the passenger's clothes, the pure visual recognition system is prone to misjudgment, resulting in false prompts, which seriously weakens the reliability and user trust of the monitoring system.

[0023] The safety belt wearing monitoring method, the electronic device and the vehicle provided by the embodiments of the present application can determine the color similarity between the theoretical safety belt region and the clothing region according to the monitoring image of the safety belt wearing region; when the color similarity is greater than or equal to a preset first similarity threshold, the multi-modal verification data are acquired, and the determination results of a plurality of single modes are determined according to the monitoring image and the multi-modal verification data; the safety belt wearing state is determined according to the plurality of determination results and the color similarity, and a hierarchical early warning prompt is performed according to the safety belt wearing state. When the color similarity between the safety belt and the driver's foreign matter is high, the multi-modal verification data are used for cross verification, which fundamentally solves the monitoring problem caused by the same color of the safety belt and the clothes, greatly improves the accuracy of safety belt wearing monitoring, and effectively prevents misjudgment. The multi-modal verification data are used for cross verification, the heterogeneous multi-modal verification data are deeply fused and backed up for each other, even if the recognition of a single mode fails or is disturbed, other modes can still provide effective information, ensuring the robustness of the overall wearing monitoring task and the stability of the all-scene and all-weather safety belt wearing monitoring. After the safety belt wearing state is determined, a hierarchical early warning prompt is performed, which changes the harsh alarm into natural guidance under different monitoring results, significantly improving the user experience and safety.

[0024] The safety belt wearing monitoring method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0025] In some embodiments, as shown in Figure 1 A safety belt wearing monitoring method includes steps 101-103.

[0026] Step 101: Determine the color similarity between the theoretical safety belt region and the clothing region according to the monitoring image of the safety belt wearing region.

[0027] In specific implementation, after the vehicle is powered on, a function self-check is performed first. After the vehicle is started or it is detected that the driver is seated, the safety belt wearing monitoring system is powered on for self-check to ensure that the data acquisition modules such as the high-definition camera, the infrared thermal imaging sensor, the seat pressure sensor and the microphone array are working normally. The multi-modal fusion decision module is initialized, and the pre-trained deep learning model and decision logic parameters are loaded.

[0028] Then the multi-modal data is synchronously collected, the multi-modal data including monitoring images of the whole wearing area and multi-modal verification data, the multi-modal verification data including thermal imaging data of the safety belt wearing area and pressure distribution data of the driver seat (more modal data can be added for assistance, which is not limited here).

[0029] The monitoring image is a visible light image, which is collected by a high-definition camera deployed on the A-pillar, the interior rearview mirror or the center console, covering the RGB image of the upper body of the occupant and the safety belt area. The thermal imaging data is collected by an infrared thermal imager, and the infrared sensor synchronously collects the thermal imaging data of the safety belt wearing area to obtain temperature distribution information. The pressure distribution data collection requires the use of a pressure sensor matrix installed in the seat to collect data such as pressure distribution and center of gravity position of the occupant. The vehicle microphone array continuously collects in-vehicle voice commands and environmental sounds for timely response to user interaction requests.

[0030] Then, the multi-modal data is pre-processed and standardized. The monitoring image is denoised, grayscale, histogram equalization, etc. to enhance the image quality and reduce the influence of light changes. The thermal imaging data is corrected for non-uniformity and calibrated for temperature to ensure the accuracy of the temperature data. The pressure distribution data is filtered to eliminate noise interference such as vehicle vibration. The voice signal is denoised and endpoint detected to prepare for subsequent recognition.

[0031] When the occupant is detected to be seated, the safety belt wearing state is determined through multi-modal data, and high-reliability recognition is performed especially for the extreme scenario of the safety belt and the clothes being the same color. The color similarity between the theoretical safety belt area and the clothing area needs to be determined according to the monitoring image. First, region positioning is performed in the monitoring image, and a target detection network (such as YOLO or SSD) is used to accurately identify the "occupant upper body", "safety belt shoulder strap area", "safety belt waist strap area" and "clothing main area". Then the color feature extraction is performed, the RGB and HSV color space histograms of the positioned areas are calculated, and the color similarity of the safety belt area and the clothing area is calculated using the CIEDE2000 color difference formula.

[0032] In some embodiments, determining the color similarity between the theoretical safety belt area and the clothing area according to the monitoring image of the safety belt wearing area includes: identifying a first identified color of the theoretical safety belt area and a second identified color of the clothing background area in the monitoring image; calculating the similarity of the first identified color and the second identified color to obtain the color similarity.

[0033] In implementation, first, the first color of the theoretical safety belt region and the second color of the clothing background region are identified, where the theoretical safety belt region includes the safety belt shoulder strap region and the safety belt waist belt region, and the first color and the second color are represented by RGB values. The similarity calculation process includes: color space conversion, which requires converting the first color and the second color (usually RGB values) to CIELAB color space. CIELAB color space is an approximately uniform color space, where L represents lightness, and A and B represent red-green and yellow-blue opposition dimensions. Then calculate the auxiliary variables, based on the L, A, and B values, calculate a set of intermediate variables such as L' (lightness average), C' (chroma average), and h' (average hue angle) to prepare for subsequent calculations. Then, calculate the difference components, respectively calculate the differences in lightness (ΔL'), chroma (ΔC'), and hue (ΔH') of the first color and the second color. Finally, weighting and correction are performed, and weighting coefficients (S_L, S_C, S_H) are introduced to correct the differences in sensitivity of the human eye to different lightness, chroma, and hue. For example, the human eye is more sensitive to changes in the hue of the blue region, and the formula corrects this by S_H and the rotation function R_T, and all parameters are substituted into the CIEDE2000 formula. The output result obtained by calculation is the color difference ΔE. Color similarity and color difference are negatively related, that is, the smaller the color difference ΔE, the greater the color similarity T, indicating that the first color and the second color are more similar.

[0034] Exemplarily, if the color similarity is simply represented as the reciprocal of the color difference, when ΔE∈(10.0, +∞), T∈(0, 1 / 10), it indicates that the color difference between the first color and the second color is large, and the color similarity between the first color and the second color is small, the first color and the second color have significant differences, and the first color and the second color are completely different. When ΔE∈(3.5, 10.0], T∈[1 / 10, 1 / 7), it indicates that the color difference between the first color and the second color is slightly large, and the color similarity between the first color and the second color is slightly small, the first color and the second color have perceptible differences, and the first color and the second color have obvious differences. When ΔE∈(2.0, 3.5], T∈[1 / 7, 1 / 2), it indicates that the color difference between the first color and the second color is small, and the color similarity between the first color and the second color is large, the first color and the second color have slight differences, and the first color and the second color can be perceived when displayed side by side. When ΔE∈(1.0, 2.0], T∈[1 / 2, 1), it indicates that the color difference between the first color and the second color is very small, and the color similarity between the first color and the second color is very large, the first color and the second color are highly similar, and the first color and the second color can only be perceived when carefully observed. When ΔE∈(0, 1.0], T∈[1, +∞), it indicates that the color difference between the first color and the second color is almost zero, and the color similarity between the first color and the second color is extremely high, the first color and the second color are extremely similar, and the first color and the second color cannot be distinguished by the naked eye and ordinary equipment, and the difference can only be perceived when professional color analysis equipment is used. Then 1 / 2 can be taken as the first similarity threshold.

[0035] Step 102: In response to the color similarity being greater than or equal to the preset first similarity threshold, acquiring multi-modal verification data, and determining a plurality of single-modal determination results according to the monitoring image and the multi-modal verification data.

[0036] In specific implementation, if the color similarity is greater than or equal to the preset first similarity threshold, it indicates that the safety belt color and the occupant clothing color are highly similar or extremely similar, and it is impossible to distinguish them by simple vision, so it is necessary to further interactively verify by the multi-modal verification data and the monitoring image to determine the preliminary monitoring result and obtain the determination result.

[0037] In some embodiments, the multi-modal verification data includes thermal imaging data of a safety belt wearing area and pressure distribution data of a driver seat; the determination result includes a structure modal wearing state, a thermal imaging modal wearing state, and a pressure modal wearing state; and determining the plurality of single-modal determination results according to the monitoring image and the multi-modal verification data includes: extract visible light features from the theoretical safety belt region in the monitoring image to obtain edge continuity features and texture structure features, and determine the structure modal wearing state according to the edge continuity features and the texture structure features; determine a thermal distribution image and a temperature gradient map of the safety belt wearing region according to the thermal imaging data, and determine the thermal imaging modal wearing state according to the thermal distribution image and the temperature gradient map; determine a pressure distribution center and a pressure distribution uniformity according to the pressure distribution data, and determine the pressure modal wearing state according to the pressure distribution center and the pressure distribution uniformity.

[0038] In specific implementation, for the monitoring image, edge continuity and structure verification are performed through visible light feature extraction to analyze whether the edge of the safety belt is continuous and complete, and whether the geometric structure conforms to the wearing feature (such as the diagonal shape of the shoulder strap and the horizontal shape of the waist strap). If the edge is broken, blurred, or fused with the edge of the clothes, it is inclined to determine that the safety belt is not worn. The thermal distribution of the safety belt wearing region is analyzed using thermal imaging data. Because the material and heat capacity of the fabric safety belt and ordinary clothes are different, there are usually differences in their thermal radiation characteristics. If a continuous belt-shaped region with a significant temperature difference from the surrounding clothes is detected, it strongly supports the determination result that the safety belt has been worn. If the thermal distribution is uniform and has no belt-shaped feature, it supports the determination result that the safety belt has not been worn. Pressure and posture are verified using pressure distribution data. The body posture of the occupant is determined according to the pressure distribution data, and it is judged whether the occupant is in a standard sitting posture that can be effectively restrained by the safety belt. If the pressure center deviates seriously (such as excessive forward leaning) or the posture is abnormal, even if the safety belt is detected, there is a possibility of misjudgment, and it is more inclined to determine that the safety belt has not been worn. If it meets the requirements of the standard sitting posture, it is more inclined to determine that the safety belt has been worn. The data of each modal will get a determination result, and the combination of multiple determination results and color similarity can determine a more accurate recognition result. The specific determination process of each modal is shown in the following embodiments.

[0039] In some embodiments, determining the structure modal wearing state according to the edge continuity features and the texture structure features includes: in response to the edge continuity feature being edge continuous and the texture structure feature being texture difference, determining that the safety belt is worn as the structure modal wearing state; in response to the edge continuity feature being edge broken or the texture structure feature being no texture difference, determining that the safety belt is not worn as the structure modal wearing state.

[0040] In specific implementation, for the judgment of the edge structure mode, since the color of the safety belt and the clothing of the occupant is highly similar, at this time, the texture feature and the structure feature of the safety belt are needed to be analyzed as input. If the edge continuity feature is edge continuity, it indicates whether the edge of the safety belt is continuous and complete. If the texture structure feature is texture difference, it indicates that there is a certain difference in the geometric structure between the safety belt and the clothing, and the edge fusion does not occur. When the two conditions are met at the same time, it is determined that the edge of the safety belt is continuous, complete, and the geometric structure is obvious, and the edge is not fused with the edge of the clothing, which meets the wearing feature (such as the diagonal line of the shoulder strap and the horizontal line of the waist belt), and the safety belt is determined as the structure mode wearing state. If there is edge fracture, blur or edge fusion with the clothing, it is inclined to determine that the safety belt is not worn, and any one of the edge continuity feature being edge fracture and the texture structure feature being no texture difference can determine that the safety belt is not worn as the structure mode wearing state.

[0041] In some embodiments, the thermal imaging mode wearing state is determined according to the thermal distribution image and the temperature gradient map, including: In response to the existence of the peak-valley feature in the temperature gradient map, a target graph corresponding to the safety belt is matched in the thermal distribution image according to the graph similarity, and a graph matching result is obtained; In response to the graph matching result being that the target graph exists in the thermal distribution image, the safety belt is determined as the thermal imaging mode wearing state; In response to the graph matching result being that the target graph does not exist in the thermal distribution image, the safety belt is determined as the thermal imaging mode wearing state.

[0042] In specific implementation, for the judgment process of the infrared thermal imaging mode, first, the thermal region registration is performed, the thermal distribution image is pixel-level registered with the monitoring image, and it is ensured that the same physical region is analyzed. Then, the absolute temperature analysis is performed, and the absolute temperature values of the safety belt wearing region and the clothing region are read. Since the safety belt is usually not directly attached to the skin, and the materials are different, the temperature thereof may be different from that of the clothing.

[0043] Then, the relative temperature gradient analysis is performed, and the temperature gradient perpendicular to the safety belt direction is calculated. A safety belt that is pulled tight will form a "thermal groove" in contact with the body or a "low temperature zone" due to the fabric characteristics, so that an obvious peak-valley feature will be displayed on the gradient map. The "fake safety belt" pattern printed on the clothes will not have this gradient feature. Finally, the thermal distribution pattern recognition is performed, and a small neural network is used to judge whether there is a continuous and narrow band-shaped target graph in the thermal distribution image, which is consistent with the physical form of the safety belt.

[0044] If there is a peak-valley feature and there is a target pattern in the thermal distribution image, it means that the thermal imaging data clearly determines that the seat belt is worn, and the seat belt is determined as the thermal imaging modal wearing state, and the confidence is high. If there is no peak-valley feature or there is no target pattern in the thermal distribution image, it means that the thermal imaging feature of wearing a seat belt is not obvious, and the seat belt is determined as the thermal imaging modal wearing state.

[0045] In some embodiments, the pressure modal wearing state is determined according to the pressure distribution center and the pressure distribution uniformity, comprising: determining a standard pressure center range and a standard uniformity range corresponding to a standard sitting posture; in response to the presence of the pressure distribution center in the standard pressure center range and the presence of the pressure distribution uniformity in the standard uniformity range, the seat belt is determined as the pressure modal wearing state; in response to the absence of the pressure distribution center in the standard pressure center range or the absence of the pressure distribution uniformity in the standard uniformity range, the seat belt is determined as the pressure modal wearing state.

[0046] In specific implementation, the sitting posture classification based on pressure distribution needs to use a classifier (such as SVM or lightweight CNN) to classify the occupant posture into "standard sitting posture", "forward leaning", "lateral leaning", "lying back", etc. The restraint state and visible form of the seat belt are different under different postures.

[0047] Optionally, it can also be complementary to visual posture estimation. If a skeleton recognition algorithm based on a camera (such as OpenPose) is configured, the joint information output by the algorithm can be fused with the pressure distribution information to more accurately judge the body angle, for example, whether the torso is at a reasonable angle with the seat back, which is a prerequisite for effective restraint of the seat belt.

[0048] When classifying the sitting posture, it is only necessary to judge whether the user's sitting posture is relative to the standard, so first determine the standard pressure center range and the standard uniformity range corresponding to the standard sitting posture. If there is a pressure distribution center in the standard pressure center range and a pressure distribution uniformity in the standard uniformity range, it means that the occupant is in a standard sitting posture that can be effectively restrained by the seat belt, and the seat belt is determined as the pressure modal wearing state. If there is no pressure distribution center in the standard pressure center range or no pressure distribution uniformity in the standard uniformity range, it means that the occupant is in a non-standard sitting posture that cannot be effectively restrained by the seat belt, and the seat belt is determined as the pressure modal wearing state.

[0049] Optionally, if the color similarity is less than a preset first similarity threshold, it means that the computer vision-based judgment is relatively accurate, and the identification result can be directly used to judge whether the seat belt is worn. Optionally, in order to ensure accuracy, in the case of sufficient computing resources, further verification can also be performed according to multi-modal verification data.

[0050] Step 103: Determine the seat belt wearing state according to the plurality of determination results and the color similarity, and give a pre-warning prompt according to the seat belt wearing state.

[0051] In specific implementation, if the color similarity is less than or equal to a preset second similarity threshold (for example, 1), it indicates that there is a certain possibility of wearing a seat belt when computer vision recognition is performed, but the probability is relatively small, so the influence of the color similarity is not considered when multi-modal judgment is performed, and only the determination results of the structural modal, the thermal imaging modal and the pressure modal are needed to determine the seat belt wearing state.

[0052] If the determination result of the structural modal is wearing a seat belt, the determination result of the thermal imaging modal is wearing a seat belt, and the determination result of the pressure modal is wearing a seat belt, it indicates that the determination results of the three modes are all wearing a seat belt, the result is reliable, and the seat belt wearing state is wearing a seat belt.

[0053] If the determination result of the structural modal is not wearing a seat belt, the determination result of the thermal imaging modal is wearing a seat belt, and the determination result of the pressure modal is wearing a seat belt, it indicates that among the determination results of the three modes, the determination results of two modes are wearing a seat belt, and the determination result of one mode is not wearing a seat belt, and the judgment based on computer vision is not reliable because the color of the seat belt and the foreign matter is similar, the determination result of the seat belt occupies the majority, and the determination result of the majority is used as the criterion to determine that the seat belt wearing state is wearing a seat belt.

[0054] If the determination result of the structural modal is wearing a seat belt, the determination result of the thermal imaging modal is not wearing a seat belt, and the determination result of the pressure modal is wearing a seat belt, it indicates that among the determination results of the three modes, the determination results of two modes are wearing a seat belt, and the determination result of one mode is not wearing a seat belt, and the judgment based on computer vision is not reliable because the color of the seat belt and the foreign matter is similar, the determination result of the seat belt occupies the majority, and the determination result of the majority is used as the criterion to determine that the seat belt wearing state is wearing a seat belt.

[0055] If the determination result of the structural modal is wearing a seat belt, the determination result of the thermal imaging modal is wearing a seat belt, and the determination result of the pressure modal is not wearing a seat belt, it indicates that among the determination results of the three modes, the determination results of two modes are wearing a seat belt, and the determination result of one mode is not wearing a seat belt, and the judgment based on computer vision is not reliable because the color of the seat belt and the foreign matter is similar, the determination result of the seat belt occupies the majority, and the determination result of the majority is used as the criterion to determine that the seat belt wearing state is wearing a seat belt.

[0056] If the determination result of the structural modal is that the safety belt is not worn, the determination result of the thermal imaging modal is that the safety belt is not worn, and the determination result of the pressure modal is that the safety belt is worn, it indicates that the determination result of one of the three modalities is that the safety belt is worn, and the determination results of two modalities are that the safety belt is not worn, and the judgment based on computer vision is not reliable because the color of the safety belt and the foreign matter is similar, and the determination result of the safety belt is not worn occupies the majority, and the majority of the determination result is used as the criterion, and it is determined that the safety belt wearing state is that the safety belt is not worn.

[0057] If the determination result of the structural modal is that the safety belt is not worn, the determination result of the thermal imaging modal is that the safety belt is worn, and the determination result of the pressure modal is that the safety belt is not worn, it indicates that the determination result of one of the three modalities is that the safety belt is worn, and the determination results of two modalities are that the safety belt is not worn, and the judgment based on computer vision is not reliable because the color of the safety belt and the foreign matter is similar, and the determination result of the safety belt is not worn occupies the majority, and the majority of the determination result is used as the criterion, and it is determined that the safety belt wearing state is that the safety belt is not worn.

[0058] If the determination result of the structural modal is that the safety belt is not worn, the determination result of the thermal imaging modal is that the safety belt is not worn, and the determination result of the pressure modal is that the safety belt is not worn, it indicates that the determination result of one of the three modalities is that the safety belt is worn, and the determination results of two modalities are that the safety belt is not worn, and the judgment based on computer vision is not reliable because the color of the safety belt and the foreign matter is similar, and the determination result of the safety belt is not worn occupies the majority, and the majority of the determination result is used as the criterion, and it is determined that the safety belt wearing state is that the safety belt is not worn. When the color similarity is less than or equal to the preset second similarity threshold, the process of determining the safety belt wearing state is shown in the following embodiment.

[0059] In some embodiments, determining the safety belt wearing state according to the plurality of determination results and the color similarity includes: In response to the color similarity being less than or equal to the preset second similarity threshold, determining that the determination result is that the safety belt is worn, the number of results, and the total number of modalities; wherein the second similarity threshold is greater than the first similarity threshold; determining the ratio of the number of results to the total number of modalities; In response to the ratio being greater than a preset ratio threshold, determining that the safety belt is worn as the safety belt wearing state; In response to the ratio being less than or equal to the preset ratio threshold, determining that the safety belt is not worn as the safety belt wearing state.

[0060] In a specific implementation, the computer vision-based judgment is not reliable because the seat belt and the foreign object have similar colors, so at this time, the seat belt wearing state is determined only according to the determination results of the structural modal, the thermal imaging modal, and the pressure modal, and the total number of modes is 3. The number of determination results that determine the seat belt to be worn is further determined, and then the ratio of the number of results to the total number of modes is used to determine the seat belt wearing state. When the total number of modes is odd, the number of determination results that can distinguish between the seat belt being worn and the seat belt not being worn is determined, and the majority of the results is used as the final seat belt configuration state. When the total number of modes is even, the number of determination results that determine the seat belt to be worn and the number of determination results that determine the seat belt not to be worn are equal, and at this time, the vision-based judgment result determines that the colors are similar, and the tendency of the seat belt not being worn is greater. The seat belt not being worn is determined as the seat belt wearing state. Then, 0.5 can be used as the ratio threshold to determine the seat belt wearing state. When the ratio is greater than the preset ratio threshold, the determination result of the seat belt being worn occupies the majority, and the seat belt being worn is determined as the seat belt wearing state. When the ratio is less than or equal to the preset ratio threshold, the determination result of the seat belt not being worn occupies the majority, or the number of the two determination results is the same, and the seat belt not being worn is determined as the seat belt wearing state.

[0061] Alternatively, if the scenario in which the number of the two determination results is the same is determined as the seat belt being worn, 0.5 can also be selected as the ratio threshold to determine the seat belt wearing state. At this time, when the ratio is greater than or equal to the preset ratio threshold, the determination result of the seat belt being worn occupies the majority, or the number of the two determination results is the same, and the seat belt being worn is determined as the seat belt wearing state. When the ratio is less than the preset ratio threshold, the determination result of the seat belt not being worn occupies the majority, and the seat belt not being worn is determined as the seat belt wearing state.

[0062] For the case in which the color similarity is greater than the preset second similarity threshold, the color similarity is too high, which is equivalent to the determination result of the seat belt not being worn. However, in order to avoid the coincidence that the occupant wears clothes that are extremely similar in color to the seat belt, multi-modal verification is still needed. The process of determining the seat belt wearing state in this scenario is shown in the following embodiment.

[0063] In some embodiments, determining the seat belt wearing state according to the plurality of determination results and the color similarity includes: in response to the color similarity being greater than the preset second similarity threshold and the ratio being greater than the preset ratio threshold, determining the determination result as a target determination result that determines the seat belt to be worn; determining the confidence of each target determination result to obtain a confidence set; in response to the maximum confidence in the confidence set being greater than or equal to a preset confidence threshold, determining the seat belt to be worn as the seat belt wearing state; In response to the maximum confidence in the confidence set being less than the preset confidence threshold, the wearing anomaly is determined as the seat belt wearing state.

[0064] In a specific implementation, since the situation in which the clothes are similar in color to the seat belt and can be distinguished only by professional equipment is almost zero (unless individual accidental phenomena), the determination result of the computer vision is more inclined to be no seat belt wearing at this time, and at this time, the accuracy of the determination result with the ratio greater than the preset ratio threshold in the multi-modal verification is affected, that is, the seat belt wearing state of wearing a seat belt may not be accurate enough (for the situation of no seat belt wearing, there is no effect, but the credibility is increased). Because if the determination result of the computer vision is counted, the opposite seat belt wearing state may be obtained, so at this time, the credibility of each determination result needs to be further determined. The confidence is used to measure the credibility of each target determination result, and by determining the confidence of each target determination result, a confidence set is obtained, and the confidence set includes the confidence of each determination result as wearing a seat belt.

[0065] If the maximum confidence in the confidence set is greater than or equal to the preset confidence threshold, it indicates that there is a determination result with very high credibility to support the final determination of wearing a seat belt; and in this scenario, the confidence of the determination result of the computer vision dimension is low, and the determination with high confidence is more inclined, and the seat belt wearing is determined as the seat belt wearing state.

[0066] If the maximum confidence in the confidence set is less than the preset confidence threshold, it indicates that there is no determination result with very high credibility to support the final determination of wearing a seat belt, and a more conservative control strategy is adopted for safety consideration, and the wearing anomaly is determined as the seat belt wearing state. Because the user safety is more important than the user experience, the safety is given priority to, and the seat belt wearing prompt is triggered when the wearing anomaly occurs, to ensure the safety.

[0067] After the seat belt wearing state is determined, the hierarchical warning prompt according to the seat belt wearing state can be completed through the head-up display and voice interaction, and the hierarchical warning prompt mechanism of the head-up display is as follows. First-level prompt (preventive): After the vehicle starts or the occupant is seated, the head-up display displays the icon and text of “Please fasten the seat belt” with a gentle prompt sound.

[0068] Second-level prompt (warning): When the system determines that the seat belt wearing state is a wearing anomaly, the head-up display displays “Seat belt state detection, please check if it is fastened” or “The seat belt is the same color as the clothes, please confirm wearing”, and flashes a yellow warning icon.

[0069] Level 3 prompt (mandatory): When the system determines that the seat belt wearing state is not wearing a seat belt, the head-up display displays a red "seat belt not fastened" icon, a text prompt, and a countdown (such as "Please fasten your seat belt within 3 seconds"), accompanied by a continuous buzzing or vibration reminder.

[0070] The system determines that the seat belt wearing state is wearing a seat belt, and no pre-warning prompt is needed.

[0071] Voice interaction mechanism: Voice command recognition: The system listens in real time and recognizes passenger voice commands such as "fasten seat belt", "how to fasten seat belt", "seat belt not fastened properly", etc.

[0072] Voice feedback and guidance: If the seat belt wearing state is not wearing a seat belt, the system immediately issues a voice reminder through the car audio: "I detect that you are not wearing a seat belt, please fasten it immediately to ensure safety." If "how to fasten seat belt" is recognized, the system can provide voice guidance: "Please cross the shoulder strap from the shoulder to the hip, the waist strap close to the hip bone, and buckle the lock." Voice confirmation and closed loop: After the passenger fastens the seat belt, if he says "seat belt fastened", the system will restart the multi-modal verification once, and if the verification is passed, the head-up display and voice will simultaneously prompt "seat belt fastened, have a pleasant journey!", completing the closed loop and ensuring driving safety.

[0073] Exemplarily, taking the scenario of the passenger wearing a black sweatshirt and wearing a black seat belt as an example.

[0074] The computer vision recognition module detects that the seat belt has a very high similarity to the color of the clothes, triggering the high-risk process of the same color.

[0075] The edge detection module finds that the seat belt edge is almost invisible in the image, with poor continuity.

[0076] The infrared thermal imaging module detects that there is a continuous band-shaped thermal zone (temperature slightly lower, consistent with the heat dissipation characteristics of the belt) with a clear (high confidence) thermal gradient in the seat belt wearing area compared to the uniform temperature background of the clothes.

[0077] The pressure sensor shows that the passenger's posture is a standard sitting posture, with normal pressure distribution.

[0078] Comprehensive judgment: The edge structure verification result is "not wearing a seat belt", the infrared thermal imaging and posture pressure verification results are both "wearing a seat belt", and the infrared feature is obvious with high confidence. The final determination is that the seat belt wearing state is "wearing a seat belt", the head-up display displays a green "seat belt fastened" icon, and the voice has no prompt.

[0079] The corresponding voice interaction example is: Passenger: "The seat belt is not fastened." Voice recognition module recognizes and analyzes, triggering voice reminder: "Detection of your seat belt is not fastened, please fasten immediately to ensure safety." After the passenger fastens the seat belt, the response is: "The seat belt is fastened." After the system verifies the seat belt wearing state as "wearing seat belt" again, the head-up display shows "The seat belt is fastened" and the voice prompt is "The seat belt is fastened, have a good trip!", completing the closed loop.

[0080] In summary, the embodiments of the present application have the following technical effects: Accurate response to same color interference: Through the four-dimensional cross-verification logic of "color similarity → edge structure → infrared thermal imaging → posture pressure", the monitoring accuracy in the same color scene of seat belt and clothes is significantly improved, avoiding the misjudgment of single visual recognition.

[0081] Multi-modal fusion, strong robustness: Comprehensive use of visible light, infrared, pressure, voice and other multi-source heterogeneous data, even if one mode fails or is disturbed (such as insufficient light, clothes blocking), other modes can still provide effective information to ensure the overall robustness of the system.

[0082] Active interaction, better experience: Integrated with head-up display and voice interaction, realizing multi-channel, hierarchical and interactive prompts of "vision + hearing", improving user perception and response efficiency, and avoiding "harsh prompts".

[0083] Safety closed loop management: Support voice command input and system feedback, forming a safety closed loop of "detection → reminder → user response → re-detection → confirmation", ensuring the effectiveness of seat belt wearing.

[0084] Strong adaptability and high universality: Suitable for different vehicle models, different passengers (adults / children), different environments (day / night, sunny / rainy), with wide practical value.

[0085] It should be noted that the method of the embodiments of the present application can be executed by a single device, such as a computer or a server, etc. The method of the embodiments of the present application can also be applied in a distributed scenario, completed by multiple devices cooperating with each other. In this distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiments of the present application, and the multiple devices will interact with each other to complete the method.

[0086] It is to be understood that the foregoing description is directed to embodiments of the application. Various embodiments can be devised without departing from the scope of the application. Some of the embodiments of the present application are described in the following detailed description. It should be appreciated that $ specific exemplary processes illustrated in the figures, and described below, can represent $ process that can be employed in connection with some embodiments of the present application. It should be appreciated that some embodiments of the present application can be practiced without several of the acts or steps described in the figures, and without including several of the advantages identified in the background section, and described below. In some instances, well-known structures and functions have not been described in detail in order to avoid obscuring the subject matter of the present application. It is to be understood that the acts or device components illustrated in the figures, and described below, can be implemented with computer software, hardware, or a combination of hardware and software to achieve the results described in connection with the below figures.

[0087] Based on the same inventive concept, the present application also provides a safety belt wearing monitoring device corresponding to the method of any of the above embodiments.

[0088] Reference Figure 2 , the safety belt wearing monitoring device comprises: The computer vision module 10 is configured to determine the color similarity between the theoretical safety belt area and the clothing area according to the monitoring image of the safety belt wearing area; The multi-modal verification module 20 is configured to, in response to the color similarity being greater than or equal to a preset first similarity threshold, acquire multi-modal verification data, and determine a plurality of single-modal determination results according to the monitoring image and the multi-modal verification data; The comprehensive decision module 30 is configured to determine the safety belt wearing state according to the plurality of determination results and the color similarity, and to perform a graded early warning prompt according to the safety belt wearing state.

[0089] Optionally, the computer vision module 10 is further configured to: identify a first identified color of the theoretical safety belt area and a second identified color of the clothing background area in the monitoring image; perform similarity calculation on the first identified color and the second identified color to obtain the color similarity.

[0090] Optionally, the multi-modal verification module 20 is further configured to: extract visible light features from the theoretical safety belt area in the monitoring image to obtain edge continuity features and texture structure features, and determine a structure modal wearing state according to the edge continuity features and the texture structure features; determine a thermal distribution image and a temperature gradient map of the safety belt wearing area according to thermal imaging data, and determine a thermal imaging modal wearing state according to the thermal distribution image and the temperature gradient map; determine a pressure distribution center and a pressure distribution uniformity according to pressure distribution data, and determine a pressure modal wearing state according to the pressure distribution center and the pressure distribution uniformity.

[0091] Optionally, the multi-modal verification module 20 is further configured to: In response to the edge continuity feature being edge continuity and the texture structure feature being texture difference, the safety belt wearing state is determined as the structure modal wearing state. In response to the edge continuity feature being edge fracture or the texture structure feature being no texture difference, the safety belt wearing state is determined as the structure modal wearing state.

[0092] Optionally, the multi-modal verification module 20 is further configured to: In response to the temperature gradient map having a peak-valley feature, a target pattern corresponding to the safety belt is matched in the thermal distribution image according to the pattern similarity, to obtain a pattern matching result; In response to the pattern matching result being that the target pattern exists in the thermal distribution image, the safety belt wearing state is determined as the thermal imaging modal wearing state; In response to the pattern matching result being that the target pattern does not exist in the thermal distribution image, the safety belt wearing state is determined as the thermal imaging modal wearing state.

[0093] Optionally, the multi-modal verification module 20 is further configured to: determine a standard pressure center range and a standard uniformity range corresponding to a standard sitting posture; In response to the standard pressure center range having a pressure distribution center and the standard uniformity range having a pressure distribution uniformity, the safety belt wearing state is determined as the pressure modal wearing state; In response to the standard pressure center range not having a pressure distribution center or the standard uniformity range not having a pressure distribution uniformity, the safety belt wearing state is determined as the pressure modal wearing state.

[0094] Optionally, the comprehensive decision module 30 is further configured to: In response to the color similarity being less than or equal to a preset second similarity threshold, the determination result is determined as the result of wearing the full belt, and the total number of modalities; wherein the second similarity threshold is greater than the first similarity threshold; determine a ratio of the result number to the total number of modalities; In response to the ratio being greater than a preset ratio threshold, the safety belt wearing state is determined as the safety belt wearing state; In response to the ratio being less than or equal to the preset ratio threshold, the safety belt wearing state is determined as the safety belt wearing state.

[0095] Optionally, the comprehensive decision module 30 is further configured to: In response to the color similarity being greater than a preset second similarity threshold and the ratio being greater than a preset ratio threshold, the determination result is determined as the target determination result of wearing the full belt; determine a confidence of each target determination result, to obtain a confidence set; In response to the maximum confidence in the confidence set being greater than or equal to a preset confidence threshold, the wearing of the safety belt is determined as the safety belt wearing state; In response to the maximum confidence in the confidence set being less than the preset confidence threshold, the wearing anomaly is determined as the safety belt wearing state.

[0096] For the convenience of description, the above apparatus is described in various modules in terms of functions. Of course, the functions of the modules can be implemented in one or more software and / or hardware in the implementation of the present application.

[0097] The apparatus of the above embodiments is used to implement the corresponding safety belt wearing monitoring method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here again.

[0098] Based on the same inventive concept, the present application also provides an electronic device corresponding to the method of any of the above embodiments, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the safety belt wearing monitoring method of any of the above embodiments.

[0099] Figure 3 A more specific hardware structure schematic diagram of an electronic device provided by the present embodiment is shown, which can include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for communication within the device.

[0100] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the present embodiment.

[0101] The memory 1020 can be implemented by a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs, and when the technical solutions provided by the present embodiment are implemented by software or firmware, the related program codes are saved in the memory 1020 and executed by the processor 1010.

[0102] The input / output interface 1030 is configured to connect an input / output module to realize information input and output. The input / output module can be configured in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0103] The communication interface 1040 is configured to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as a USB, a network cable, etc.) or a wireless manner (such as a mobile network, WIFI, Bluetooth, etc.).

[0104] The bus 1050 includes a channel to transmit information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0105] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only include components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0106] The electronic device of the above embodiments is used to implement the safety belt wearing monitoring method of any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0107] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to execute the safety belt wearing monitoring method of any of the above embodiments.

[0108] The computer readable medium of the embodiments includes permanent and non-permanent, removable and non-removable media, which can realize information storage by any method or technology. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0109] The storage medium of the above-mentioned embodiments stores computer instructions for causing the computer to execute the safety belt wearing monitoring method of any one of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0110] Based on the same inventive concept, the present application also provides a vehicle comprising the electronic device or the safety belt wearing monitoring device of the above-mentioned embodiments, and executing the safety belt wearing monitoring method of any one of the above-mentioned embodiments by the electronic device or the safety belt wearing monitoring device of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0111] It can be understood that before using the technical solutions of the embodiments of the present application, the user will be informed of the type, use range, use scenario, etc. of the personal information involved by appropriate means, and the authorization of the user will be obtained.

[0112] For example, in response to receiving the active request of the user, the user is sent prompt information to explicitly prompt the user that the operation requested to be executed will require the acquisition and use of personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as electronic device, application program, server or storage medium that executes the operation of the technical solutions of the present application according to the prompt information.

[0113] As an optional but not limited implementation manner, in response to accepting the active request of the user, the way of sending prompt information to the user may, for example, be the way of pop-up window, and the prompt information can be presented in the form of text in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0114] It can be understood that the above notification and obtaining user authorization process is only illustrative, and does not limit the implementation of the present application, and other ways meeting relevant laws and regulations can also be applied to the implementation of the present application.

[0115] Those skilled in the art should understand that the discussion of any embodiment is merely exemplary and is not intended to limit the scope of the present application; the above embodiments or technical features in different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the embodiments of the present application as described above, which are not provided in details for the sake of brevity.

[0116] In addition, in order to simplify the description and discussion, and so as not to make the embodiments of the present application difficult to understand, the well-known power / ground connections of integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. In addition, the apparatus can be shown in the form of block diagrams in order to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform to be implemented for the embodiments of the present application (i.e., these details should be fully within the understanding of those skilled in the art). Where specific details (e.g., circuitry) are set forth in order to describe an illustrative embodiment of the present application, it will be apparent to those skilled in the art that the present application can be practiced without these specific details or with an implementation varying from these specific details. Therefore, these descriptions should be considered as illustrative rather than limiting.

[0117] Although the present application has been described in conjunction with the specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0118] The embodiments of the present application are intended to cover all such alternatives, modifications and variations as falling within the broad scope of the application claimed. Accordingly, any omission, modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A seat belt wearing monitoring method characterized by, The method comprises the following steps: determining color similarity between a theoretical seat belt area and a clothing area according to a monitoring image of the seat belt wearing area; in response to the color similarity being greater than or equal to a preset first similarity threshold, acquiring multi-modal verification data, and determining a plurality of single-modal determination results according to the monitoring image and the multi-modal verification data; determining a seat belt wearing state according to the plurality of determination results and the color similarity, and performing a hierarchical warning prompt according to the seat belt wearing state.

2. The seat belt wearing monitoring method according to claim 1, characterized by, The method of determining color similarity between a theoretical seat belt area and a clothing area according to a monitoring image of the seat belt wearing area comprises the following steps: identifying a first identified color of the theoretical seat belt area and a second identified color of a clothing background area in the monitoring image; performing similarity calculation on the first identified color and the second identified color to obtain the color similarity.

3. The seat belt wearing monitoring method according to claim 1, characterized by, The multi-modal verification data comprises thermal imaging data of the seat belt wearing area and pressure distribution data of a driver's seat; the determination results comprise a structure modal wearing state, a thermal imaging modal wearing state and a pressure modal wearing state; the method of determining a plurality of single-modal determination results according to the monitoring image and the multi-modal verification data comprises the following steps: extracting visible light features of the theoretical seat belt area in the monitoring image to obtain edge continuity features and texture structure features, and determining the structure modal wearing state according to the edge continuity features and the texture structure features; determining a thermal distribution image and a temperature gradient map of the seat belt wearing area according to the thermal imaging data, and determining the thermal imaging modal wearing state according to the thermal distribution image and the temperature gradient map; determining a pressure distribution center and a pressure distribution uniformity according to the pressure distribution data, and determining the pressure modal wearing state according to the pressure distribution center and the pressure distribution uniformity.

4. The seat belt wearing monitoring method according to claim 3, characterized by, The method of determining the structure modal wearing state according to the edge continuity features and the texture structure features comprises the following steps: in response to the edge continuity features being edge continuous and the texture structure features being texture difference, determining that a seat belt is worn as the structure modal wearing state; in response to the edge continuity features being edge broken or the texture structure features being no texture difference, determining that a seat belt is not worn as the structure modal wearing state.

5. The seat belt wearing monitoring method according to claim 3, characterized by, The method of determining the thermal imaging modal wearing state according to the thermal distribution image and the temperature gradient map comprises the following steps: in response to the temperature gradient map having a peak-valley feature, matching a target pattern corresponding to a seat belt in the thermal distribution image according to pattern similarity to obtain a pattern matching result; in response to the pattern matching result being that the target pattern exists in the thermal distribution image, determining that a seat belt is worn as the thermal imaging modal wearing state; in response to the pattern matching result being that the target pattern does not exist in the thermal distribution image, determining that a seat belt is not worn as the thermal imaging modal wearing state.

6. The seat belt wearing monitoring method according to claim 3, characterized by, The method of determining the pressure modal wearing state according to the pressure distribution center and the pressure distribution uniformity comprises the following steps: determining a standard pressure center range and a standard uniformity range corresponding to a standard sitting posture; determining the safety belt wearing state as the pressure modal wearing state in response to the pressure distribution center being within the standard pressure center range and the pressure distribution uniformity being within the standard uniformity range; determining the safety belt not being worn as the pressure modal wearing state in response to the pressure distribution center not being within the standard pressure center range or the pressure distribution uniformity not being within the standard uniformity range.

7. The seat belt wearing monitoring method according to claim 1, characterized by, The determining the safety belt wearing state according to the multiple determination results and the color similarity includes: determining the result quantity and the total modal quantity in response to the color similarity being less than or equal to a preset second similarity threshold value, wherein the second similarity threshold value is greater than the first similarity threshold value; determining a ratio of the result quantity to the total modal quantity; determining the safety belt being worn as the safety belt wearing state in response to the ratio being greater than a preset ratio threshold value; determining the safety belt not being worn as the safety belt wearing state in response to the ratio being less than or equal to the preset ratio threshold value.

8. The seat belt wearing monitoring method according to claim 7, characterized by, The determining the safety belt wearing state according to the multiple determination results and the color similarity includes: determining the target determination result of wearing the safety belt as the determination result in response to the color similarity being greater than a preset second similarity threshold value and the ratio being greater than a preset ratio threshold value; determining a confidence degree of each of the target determination results to obtain a confidence degree set; determining the safety belt being worn as the safety belt wearing state in response to a maximum confidence degree in the confidence degree set being greater than or equal to a preset confidence degree threshold value; determining the safety belt being abnormally worn as the safety belt wearing state in response to the maximum confidence degree in the confidence degree set being less than the preset confidence degree threshold value.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the method of any one of claims 1 to 8 when executing the program.

10. A vehicle characterized by comprising: The electronic device of claim 9 is included. The electronic device of claim 9 is included.

Images