Road surface pothole recognition method, device and equipment, medium and vehicle

Abstract

The embodiment of the invention provides a road surface pothole recognition method, device and equipment, a medium and a vehicle, and relates to the technical field of road surface pothole recognition. The method comprises the following steps: acquiring behavior monitoring data of people in a vehicle in real time; and under the condition of determining that the personnel behavior monitoring data meets the preset behavior condition, obtaining a road surface pothole judgment result that the current road surface has a pothole. According to the embodiment of the invention, the behavior monitoring data of the driver and the passenger in the vehicle are acquired, and whether the current road surface has the pit is judged according to the preset behavior condition, so that the problem of inaccurate road condition recognition based on the image in front of the vehicle due to insufficient light or severe weather is avoided, and the accuracy of road surface pit recognition is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of road pothole recognition, and more specifically, to a road pothole recognition method, device, equipment, medium and vehicle. Background Art

[0002] During driving, road conditions have a significant impact on the vehicle's driving quality and safety, so real-time recognition of road conditions is an important function. For example, when potholes are detected, the suspension system is adjusted to adapt the vehicle to the potholes, thereby improving driving comfort.

[0003] The current mainstream solution is to use visual detection algorithms to identify images in front of the vehicle to achieve road condition detection. However, at night or in bad weather conditions, the accuracy of road condition recognition is not high because it is impossible to clearly obtain visual image information in front of the vehicle. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, equipment, medium and vehicle for identifying potholes in a road surface, so as to solve the problem of low accuracy in road condition identification in traditional solutions.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying potholes on a road surface, comprising: Obtain real-time monitoring data of people’s behavior in the vehicle; When it is determined that the personnel behavior monitoring data meets the preset behavior condition, a road pothole determination result is obtained indicating that there are potholes on the current road surface.

[0006] In an embodiment of the present application, by obtaining behavior monitoring data of the driver and passengers in the vehicle and judging whether there are potholes on the current road surface based on preset behavior conditions, the problem of inaccurate road condition recognition based on the image in front of the vehicle due to insufficient light or bad weather is avoided, thereby effectively improving the accuracy of pothole recognition.

[0007] In some possible embodiments, the personnel behavior monitoring data includes image data of the driver and passenger in the vehicle; When it is determined that the personnel behavior monitoring data meets the preset behavior condition, a road pothole determination result is obtained indicating that there are potholes on the current road surface, including: Performing behavior recognition based on the driver and passenger image data, and determining driving behavior information corresponding to the driver and passenger image data; If it is determined that the driving behavior information satisfies a preset driving behavior condition, a road pothole determination result indicating that there are potholes on the current road surface is obtained.

[0008] In an embodiment of the present application, behavior information is monitored by collecting image data of the driver and passengers in the vehicle, and when the driving behavior information meets preset conditions, it is determined that there are potholes on the current road surface.

[0009] In some possible embodiments, the driving behavior information includes driving behavior information and passenger behavior information; If it is determined that the driving behavior information satisfies the preset driving behavior condition, a road pothole determination result indicating that there are potholes on the current road surface is obtained, including: If it is determined that the driving behavior information satisfies the preset driving behavior condition, and it is determined that the passenger behavior information satisfies the preset passenger behavior condition, a road pothole determination result indicating that there are potholes on the current road surface is obtained.

[0010] In the embodiment of the present application, the accuracy of pothole recognition is further improved by respectively acquiring the driving behavior and the passenger behavior, and determining that there are potholes on the current road surface when the driving behavior and the passenger behavior respectively meet preset conditions.

[0011] In some possible embodiments, determining that the driving behavior information satisfies a preset driving behavior condition includes at least one of the following: When it is determined based on the driving behavior information that the driver is holding the steering wheel tightly, determining that the driving behavior information satisfies a preset driving behavior condition; Alternatively, when it is determined based on the driving behavior information that the driver's posture change information exceeds a preset first change threshold, determining that the driving behavior information satisfies a preset driving behavior condition; Alternatively, when it is determined based on the driving behavior information that the driver is gripping the steering wheel, and when it is determined that the driver's posture change information exceeds a preset second change threshold, it is determined that the driving behavior information meets the preset driving behavior condition.

[0012] In an embodiment of the present application, the accuracy of pothole recognition is further improved by identifying the driver's behavior of gripping the steering wheel and / or judging whether the driving behavior conditions are met based on posture changes.

[0013] In some possible embodiments, determining that the passenger behavior information satisfies a preset passenger behavior condition includes at least one of the following: When it is determined based on the passenger behavior information that the passenger is holding the handrail tightly, determining that the passenger behavior information meets a preset passenger behavior condition; Alternatively, when it is determined based on the passenger behavior information that the passenger posture change information exceeds a preset third change threshold, it is determined that the passenger behavior information satisfies a preset passenger behavior condition; Alternatively, when it is determined based on the passenger behavior information that the passenger is holding the handrail tightly, and when it is determined that the passenger posture change information exceeds a preset fourth change threshold, it is determined that the passenger behavior information meets the preset passenger behavior condition.

[0014] In an embodiment of the present application, the accuracy of pothole recognition is further improved by identifying the passenger's gripping handrail behavior and / or determining whether the passenger behavior conditions are met based on posture changes.

[0015] In some possible embodiments, when it is determined that the personnel behavior monitoring data satisfies the preset behavior condition, obtaining a road pothole determination result indicating that there are potholes on the current road surface includes: In the case where it is determined that the personnel behavior monitoring data meets the preset behavior conditions, determining a road pothole determination result; wherein the road pothole determination result indicates that there are potholes on the current road surface; Obtaining the current clarity of the visual field outside the vehicle, and determining whether the clarity of the visual field outside the vehicle is lower than a preset first clarity threshold; If yes, determining that there are potholes on the current road surface according to the road surface pothole determination result; If not, obtain a pothole prediction result, and combine the pothole prediction result with the pothole determination result to determine whether there are potholes on the current road surface; wherein the pothole prediction result includes a prediction result obtained based on recognition of an image in front of the vehicle.

[0016] In an embodiment of the present application, after a preliminary determination of potholes on the road surface is made based on personnel behavior monitoring data, whether pothole recognition should be performed in conjunction with the image in front of the vehicle is determined based on whether the field of view outside the vehicle is clear, thereby further improving the accuracy of pothole recognition.

[0017] In some possible embodiments, before obtaining the current clarity of the visual field outside the vehicle, the method further includes: Obtaining a first credibility corresponding to the road pothole determination result, and determining whether the first credibility is greater than a preset first credibility threshold; If yes, determining that there are potholes on the current road surface according to the road surface pothole determination result; If not, the current outside vision clarity is obtained.

[0018] In the embodiment of the present application, whether it is necessary to further identify potholes in combination with the image in front of the vehicle is determined based on the credibility of the preliminary determination of potholes, thereby further improving the efficiency and accuracy of pothole identification.

[0019] In some possible embodiments, combining the road pothole prediction result with the road pothole determination result to determine whether there is a pothole on the current road surface includes: If the road pothole prediction result indicates that there are potholes on the current road surface, it is determined that there are potholes on the current road surface; When the road pothole prediction result indicates that there is no pothole on the current road surface, obtaining a first credibility corresponding to the road pothole determination result, and obtaining a second credibility corresponding to the road pothole prediction result, and determining whether the first credibility is higher than the second credibility; If yes, determining that there are potholes on the current road surface according to the road surface pothole determination result; If not, it is determined that there are no potholes on the current road surface.

[0020] In an embodiment of the present application, when the pothole condition determined based on the behavior of the occupants is inconsistent with the pothole condition predicted based on the image in front of the vehicle, the accuracy of pothole recognition in the road surface is further improved by obtaining the credibility of the prediction results of both and using the one with higher credibility as the final judgment result.

[0021] In some possible embodiments, when the road pothole prediction result indicates that a pothole exists on the current road surface, determining that a pothole exists on the current road surface includes: When the road pothole prediction result indicates that a pothole exists on the current road surface, obtaining a first credibility corresponding to the road pothole determination result, and obtaining a second credibility corresponding to the road pothole prediction result; Determining a comprehensive credibility based on the first credibility and the second credibility, and judging whether the comprehensive credibility is greater than a preset comprehensive credibility threshold; If so, it is determined that there are potholes on the current road surface; If not, it is determined that there are no potholes on the current road surface.

[0022] In an embodiment of the present application, when the presence of a pothole is determined based on the behavior of the occupants of the vehicle, and the presence of a pothole is also predicted based on the image in front of the vehicle, the accuracy of pothole recognition is further improved by further combining the credibility of the prediction results of the two to determine whether a pothole is determined to exist.

[0023] In a second aspect, an embodiment of the present application provides a method for identifying potholes in a road surface, comprising: Collecting the image in front of the vehicle in real time, and obtaining the road pothole prediction result corresponding to the image in front of the vehicle; Get the current clarity of the outside view in real time; When the clarity of the outside view is higher than a preset second clarity threshold, judging whether there is a pothole on the current road surface based on the road pothole prediction result; When the clarity of the visual field outside the vehicle is not higher than a preset second clarity threshold, the personnel behavior monitoring data in the vehicle and the pothole prediction result are combined to determine whether there are potholes on the current road surface.

[0024] In an embodiment of the present application, the accuracy of pothole identification is further improved by first predicting the presence of potholes on the road based on the image in front of the vehicle, and determining whether to further combine the behavior monitoring data of occupants in the vehicle to comprehensively identify potholes based on the clarity of the field of view outside the vehicle.

[0025] In a third aspect, an embodiment of the present application provides a road pothole recognition device, comprising: A data acquisition module is used to acquire the behavior monitoring data of people in the vehicle in real time; The pothole recognition module is used to obtain a road pothole determination result indicating that there are potholes on the current road surface when it is determined that the personnel behavior monitoring data meets a preset behavior condition.

[0026] In a fourth aspect, an embodiment of the present application provides a road pothole recognition device, comprising: A result prediction module, used for collecting the image in front of the vehicle in real time and obtaining the road pothole prediction result corresponding to the image in front of the vehicle; The visual field acquisition module is used to obtain the current visual field clarity outside the vehicle in real time; A first determination module is used to determine whether there is a pothole on the current road surface based on the road pothole prediction result when the clarity of the outside view of the vehicle is higher than a preset second clarity threshold; The second determination module is used to determine whether there are potholes on the current road surface in combination with the personnel behavior monitoring data in the vehicle and the pothole prediction result when the clarity of the outside view of the vehicle is not higher than a preset second clarity threshold.

[0027] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the method described in any embodiment when executing the program.

[0028] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment can be implemented.

[0029] In a seventh aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, wherein the computer program can implement the method described in any embodiment when executed by a processor.

[0030] In an eighth aspect, an embodiment of the present application provides a vehicle, comprising the electronic device described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A schematic diagram of a process for identifying potholes in a road surface provided in an embodiment of the present application; Figure 2 A schematic diagram of a flow chart of another road pothole identification method provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a road pothole identification device provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of another road pothole identification device provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0034] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0035] It should be noted that the traditional solution mainly identifies potholes by collecting images in front of the vehicle. However, at night or in bad weather conditions, due to factors such as insufficient light or rain and fog, the road condition detection results based on visual images are not very accurate, and potholes are easily missed or misdetected.

[0036] In response to the problems existing in the above-mentioned prior art, an embodiment of the present application provides a method for identifying potholes in the road, which obtains behavior monitoring data of people in the vehicle and achieves the purpose of road condition identification based on whether the actions of people in the vehicle meet preset conditions. It avoids the problem of inaccurate road condition recognition based on the image in front of the vehicle due to insufficient light or bad weather, and effectively improves the accuracy of pothole identification.

[0037] like Figure 1 As shown, the first embodiment of the present application provides a method for identifying potholes in a road surface, which may include the following steps: S1. Obtaining real-time monitoring data of personnel behavior in the vehicle.

[0038] Specifically, the behavior data of the driving and passengers in the vehicle can be collected by cameras or sensors deployed on the vehicle to obtain the behavior monitoring data. Among them, the image of the driving and passengers in the vehicle can be collected by the camera, and the action behavior recognition is performed based on the image to obtain the behavior monitoring data. It can be understood that the driving and passengers include the driver and / or passengers (occupants) in the vehicle.

[0039] S2. When it is determined that the personnel behavior monitoring data meets the preset behavior conditions, a road pothole determination result is obtained indicating that there are potholes on the current road surface.

[0040] Specifically, specific behavioral conditions can be set in advance, for example, a threshold value for a certain monitoring data can be set, and when the personnel behavior monitoring data acquired in real time reaches the corresponding threshold value, it is determined that the preset behavioral condition is met; alternatively, a target action behavior for image recognition can be set, and when the action behavior identified according to the personnel behavior monitoring data (image data or video data) matches the target action behavior, it is determined that the preset behavioral condition is met.

[0041] For example, it is possible to identify based on images whether a person in the car has specific actions or behaviors (such as touching an armrest, holding a steering wheel, etc.); for another example, it is possible to collect information based on infrared sensors, pressure sensors, etc. whether a person in the car has specific actions or behaviors (such as holding an armrest, holding a certain position in the car with their hands, etc.).

[0042] Based on this, when it is determined that the real-time acquired personnel behavior monitoring data meets the preset behavior conditions, it is determined that there are potholes on the current road surface. Since the personnel behavior monitoring data (images or sensor data) inside the car will not be affected by insufficient light outside the car, bad weather, etc., it avoids the problem of inaccurate recognition results in the existing technology based on the image in front of the vehicle for road condition recognition, thereby effectively improving the accuracy of pothole recognition.

[0043] In some possible embodiments, the personnel behavior monitoring data includes image data of the driver and passengers in the vehicle; Step S2 may include: S201, performing behavior recognition based on the driver and passenger image data, and determining driving behavior information corresponding to the driver and passenger image data; S202: If it is determined that the driving behavior information satisfies a preset driving behavior condition, a road pothole determination result indicating that there are potholes on the current road surface is obtained.

[0044] It should be noted that, since image data is relatively easy to obtain and recognition technology is relatively mature, image data of drivers and passengers in the vehicle can be collected as personnel behavior monitoring data.

[0045] By placing cameras at appropriate locations in the vehicle, the corresponding image data of the driver and passengers can be collected. Then, the preset target detection algorithm is used to detect and identify the driving behavior information of the driver and passengers, and by judging whether the driving behavior information meets the preset driving behavior conditions, it is possible to identify whether there are potholes on the current road surface.

[0046] Further, the driving behavior information includes driving behavior information and passenger behavior information; Step S202, if it is determined that the driving behavior information meets the preset driving behavior condition, a road pothole determination result is obtained indicating that there are potholes on the current road surface, which may include: S2021. If it is determined that the driving behavior information satisfies a preset driving behavior condition, and it is determined that the passenger behavior information satisfies a preset passenger behavior condition, a road pothole determination result is obtained indicating that there are potholes on the current road surface.

[0047] It should be noted that the driving and passenger personnel include the driver and / or the passenger, so the driving and passenger behavior information may include driving behavior information and passenger behavior information. That is, the behavior information of the driver and the passenger is identified respectively, and the driving behavior information and the passenger behavior information are compared with the preset driving behavior conditions and the passenger behavior conditions respectively, and whether there is a pothole road condition is comprehensively judged through the comparison results.

[0048] Specifically, this embodiment determines that there are potholes on the current road surface only when the driving behavior information meets the preset driving behavior conditions and the passenger behavior information meets the preset passenger behavior conditions. Based on this, it can avoid the problem of low accuracy in identifying potholes on the road surface based on single personal behavior information. Exemplarily, based on the image data of the driver and passenger, it can be identified whether the driver has a specific behavior (such as holding the steering wheel tightly). If so, it is considered to meet the preset driving behavior conditions; at the same time, based on the image data of the driver and passenger, it can be identified whether the passenger has a specific behavior (such as holding the handrail tightly). If so, it is considered to meet the preset passenger behavior conditions; when both the driving behavior conditions and the passenger behavior conditions are met at the same time, it is determined that there are potholes on the current road surface.

[0049] In some possible embodiments, in step S2021, determining whether the driving behavior information satisfies a preset driving behavior condition may include at least one of the following: When it is determined based on the driving behavior information that the driver is holding the steering wheel tightly, determining that the driving behavior information satisfies a preset driving behavior condition; Alternatively, when it is determined based on the driving behavior information that the driver's posture change information exceeds a preset first change threshold, it is determined that the driving behavior information satisfies a preset driving behavior condition; Alternatively, when it is determined based on the driving behavior information that the driver is gripping the steering wheel, and when it is determined that the driver's posture change information exceeds a preset second change threshold, it is determined that the driving behavior information satisfies a preset driving behavior condition.

[0050] Specifically, whether the driving behavior information meets the preset driving behavior conditions can be determined in the following three ways: First, if it is identified that the driver has an action behavior of gripping the steering wheel, it is determined that the driving behavior information meets the preset driving behavior conditions; Second, if it is recognized that the posture change information exceeds a preset first change threshold, it is determined that the driving behavior information meets a preset driving behavior condition; The third type is that when it is identified that the driver has an action behavior of gripping the steering wheel, it is identified that the posture change information exceeds a preset second change threshold, and it is determined that the driving behavior information meets the preset driving behavior condition.

[0051] It should be noted that, for the first method mentioned above, the driver's target key points, such as finger key points, can be identified based on the driver and passenger image data; for the same hand, the finger key points may include the thumb tip position and the other four finger tip positions; after detecting the finger key points, first confirm that the driver's hand is on the steering wheel, and then calculate the distance between the thumb tip position and the other four finger tip positions respectively. When it is judged that the average value of these distances is less than the preset distance threshold, it is determined that the action of gripping the steering wheel has occurred.

[0052] For example, the distance between the thumb tip and the other four finger tips can be calculated according to the following formula: ,in,( ) represents the position coordinates of the thumb tip, ( ) represents the position coordinates of one of the remaining four fingertips.

[0053] It should be noted that, for the second method mentioned above, the target key points of the driver, such as the shoulder and hip key points, can be identified based on the image data of the driver and occupant; then, the shoulder inclination angle (used to characterize the horizontal posture of the driver and occupant) and the torso inclination angle (used to characterize the vertical posture of the driver and occupant) corresponding to each frame of the image are calculated based on the position coordinates of the shoulder and hip key points.

[0054] Exemplarily, the calculation formula of the shoulder tilt angle is as follows: ,in,( ) represents the position coordinates of the key points on the left shoulder of the target person, ( ) represents the position coordinates of the key points on the right shoulder of the target person.

[0055] Exemplarily, the calculation formula of the trunk tilt angle is as follows: ,in( ) represents the position coordinates of the target person’s shoulder key point (left or right), ( ) represents the position coordinates of the target person's hip key point (left or right). Generally, the torso inclination angle can be calculated using the shoulder key point coordinates and the hip key point coordinates on the same side.

[0056] Then, the shoulder tilt angle change and the trunk tilt angle change corresponding to the two frames of images are calculated, where the calculation formula of the shoulder tilt angle change value is as follows: ,in, Indicates the shoulder tilt angle corresponding to the current frame image, Indicates the shoulder tilt angle corresponding to the previous frame image; The calculation formula of the change value of the torso tilt angle is as follows: ,in, Indicates the torso tilt angle corresponding to the current frame image, Indicates the torso tilt angle corresponding to the previous frame image.

[0057] Next, the frequency and amplitude of the shoulder tilt angle change and the trunk tilt angle change within the preset time window are respectively obtained.

[0058] For example, the frequency of the tilt angle change can be represented by the number of times the tilt angle direction changes within a preset time window. For example, the change of the shoulder tilt angle from tilting to the left to tilting to the right is considered as one change, and the number of changes within the preset time window is the change frequency of the shoulder tilt angle. Alternatively, when the shoulder tilt angle change value of the previous and next two frames exceeds a set threshold, it is considered as one change, and the number of changes within the preset time window is the change frequency of the shoulder tilt angle.

[0059] Exemplarily, the magnitude of the tilt angle change may be represented by the tilt angle change value accumulated within a preset window time.

[0060] Then, judging that the driver's posture change information exceeds the preset first change threshold may include at least one of the following: 1. If the tilt angle change frequency within the set time is greater than the frequency threshold, it is considered that the driver's posture change information exceeds the preset first change threshold; 2. If the tilt angle change amplitude within the set time is greater than the amplitude threshold, it is considered that the driver's posture change information exceeds the preset first change threshold; 3. If the tilt angle change frequency within the set time is greater than the frequency threshold, and the tilt angle change amplitude is also greater than the amplitude threshold, it is considered that the driver posture change information exceeds the preset first change threshold.

[0061] Then, when it is determined based on the driving behavior information that the driver's posture change information exceeds a preset first change threshold, it is determined that the driving behavior information satisfies a preset driving behavior condition.

[0062] For the third method, the specific calculation process and judgment logic are the same as the first / second method. The difference lies in the difference between the second change threshold and the first change threshold, that is, the change frequency threshold and amplitude threshold of the tilt angle may be different from the corresponding thresholds of the second method.

[0063] In some possible embodiments, in step S2021, determining whether the passenger behavior information satisfies a preset passenger behavior condition may include at least one of the following: When it is determined based on the passenger behavior information that the passenger is holding the handrail tightly, determining that the passenger behavior information meets a preset passenger behavior condition; Alternatively, when it is determined based on the passenger behavior information that the passenger posture change information exceeds a preset third change threshold, it is determined that the passenger behavior information satisfies a preset passenger behavior condition; Alternatively, when it is determined based on the passenger behavior information that the passenger is holding the handrail tightly, and when it is determined that the passenger posture change information exceeds a preset fourth change threshold, it is determined that the passenger behavior information meets the preset passenger behavior condition.

[0064] Specifically, whether the passenger behavior information meets the preset passenger behavior conditions can be determined in the following three ways: First, if it is identified that the passenger has an action behavior of gripping the steering wheel, it is determined that the passenger behavior information meets the preset passenger behavior conditions; Second, if it is identified that the posture change information exceeds a preset third change threshold, it is determined that the passenger behavior information meets a preset passenger behavior condition; The third type is that when it is identified that the passenger has the action of gripping the steering wheel, it is identified that the posture change information exceeds the preset fourth change threshold, and it is determined that the passenger behavior information meets the preset passenger behavior condition.

[0065] It should be noted that, similar to driving behavior recognition, for the first method mentioned above, the passenger target key points, such as finger key points, can be identified based on the driver and passenger image data; for the same hand, the finger key points can include the thumb tip position and the remaining four finger tip positions; after detecting the finger key points, first confirm that the passenger's hand is located in the area of ​​interest (such as the armrest or seat belt area), and then calculate the distance between the thumb tip position and the remaining four finger tip positions respectively. When it is judged that the average value of these distances is less than the preset distance threshold, it is determined that the action behavior of gripping the armrest handle has occurred.

[0066] It is understandable that in the process of determining whether the passenger behavior information meets the passenger behavior conditions, the calculation process and judgment logic are the same as the process of driving behavior recognition, and will not be repeated here. Among them, the first change threshold, the second change threshold, the third change threshold and the fourth change threshold can be set to the same or different thresholds according to actual needs.

[0067] In some possible embodiments, step S2, when it is determined that the personnel behavior monitoring data meets the preset behavior condition, obtaining a road pothole determination result indicating that there are potholes on the current road surface, may include: S211, when it is determined that the personnel behavior monitoring data meets the preset behavior conditions, determining a road pothole determination result; wherein the road pothole determination result indicates that there are potholes on the current road surface; S212, obtaining the current clarity of the visual field outside the vehicle, and determining whether the clarity of the visual field outside the vehicle is lower than a preset first clarity threshold; S213: If yes, determine whether there are potholes on the current road surface according to the road surface pothole determination result; S214: If not, obtain a pothole prediction result, and combine the pothole prediction result with the pothole determination result to determine whether there are potholes on the current road surface; wherein the pothole prediction result includes a prediction result obtained based on recognition of the image in front of the vehicle.

[0068] Specifically, when it is judged based on the behavior monitoring data of the people in the vehicle that the preset behavior conditions are met, a preliminary determination result of whether potholes exist on the current road surface can be obtained; then it is further determined whether the current clarity of the outside view of the vehicle is clear enough. If the clarity of the outside view of the vehicle is lower than the preset first clarity threshold, it means that the outside view of the vehicle is not good (for example, at night or in bad weather conditions). At this time, the final determination result can be directly determined based on the result of the preliminary determination, that is, the road pothole determination result of whether potholes exist on the current road surface is obtained; if the clarity of the outside view of the vehicle is not lower than the preset first clarity threshold, it means that the outside view of the vehicle is good. At this time, the road pothole prediction result based on the image in front of the vehicle can be further obtained. By combining the results of the preliminary determination and the road pothole prediction results for comprehensive analysis and judgment, the final determination result of whether there are potholes on the road surface is obtained.

[0069] It should be noted that the clarity of the outside view of the vehicle can be obtained based on a variety of methods, which can exemplarily include the following methods: 1. Based on the current system time, and combined with seasonal patterns, daily sunrise and sunset times, and the weather forecast for the current location, determine whether the current time is day or night, whether the weather is good or bad, etc., and thus determine the current clarity of the outside view of the vehicle according to a preset correspondence; 2. By real-time acquisition of images outside the vehicle, determine the intensity of the light outside the vehicle based on information such as the brightness of the image, or determine the degree of blur of the outside view based on information such as the contrast of the image, and thus determine the corresponding clarity of the outside view of the vehicle; 3. Determine whether it is day or night, sunny or cloudy, and the specific light level through light sensors, and determine the corresponding clarity of the outside view of the vehicle according to a preset correspondence.

[0070] In some possible embodiments, before step S212, the following steps may also be included: S2121, obtaining a first credibility corresponding to the road pothole determination result, and determining whether the first credibility is greater than a preset first credibility threshold; S2122: If yes, determine whether there are potholes on the current road surface according to the road surface pothole determination result; S2123: If not, obtain the current clarity of the field of view outside the vehicle.

[0071] Specifically, when it is determined that the preset behavior conditions are met based on the behavior monitoring data of the people in the vehicle, a preliminary determination result of whether there are potholes on the current road surface can be obtained. Then, the credibility of the preliminary determination result can be obtained. If the credibility is higher than a threshold, the preliminary determination result can be directly used as the final determination result, that is, the road pothole determination result of whether there are potholes on the current road surface is obtained; if the credibility is not higher than the threshold, the current external field of view clarity is further obtained, and the subsequent judgment logic is executed (determining whether to perform comprehensive road pothole recognition in combination with external vehicle image recognition technology).

[0072] It should be noted that the first credibility corresponding to the pothole determination result can be calculated based on the following methods: 1. For example, when the driver behavior information meets the driving behavior condition but the passenger behavior information does not meet the passenger behavior condition, or when the passenger behavior information meets the passenger behavior condition but the driver behavior information does not meet the driving behavior condition, a preliminary pothole determination result indicating that there are potholes on the current road surface can be obtained, but the corresponding first credibility is low (the specific credibility value can be set according to actual needs); when the driver behavior information meets the driving behavior condition and the passenger behavior information also meets the passenger behavior condition, a preliminary pothole determination result indicating that there are potholes on the current road surface can also be obtained, and the corresponding first credibility is high. 2. For example, the first credibility corresponding to the preliminary determination result can be calculated based on the image quality information (such as brightness, whether there is occlusion, etc.) of the driver and passenger image data.

[0073] In some possible embodiments, in step S214, combining the road pothole prediction result and the road pothole determination result to determine whether there is a pothole on the current road surface may include: S2141, when the road pothole prediction result indicates that the current road surface has potholes, determining that the current road surface has potholes; S2142, when the road pothole prediction result indicates that there is no pothole on the current road surface, obtaining a first credibility corresponding to the road pothole determination result, and obtaining a second credibility corresponding to the road pothole prediction result, and determining whether the first credibility is higher than the second credibility; S2143: If yes, determine whether there are potholes on the current road surface according to the road surface pothole determination result; S2144: If not, determine that there are no potholes on the current road surface.

[0074] Specifically, the road pothole prediction result is a prediction result obtained by identifying the current image in front of the vehicle, and the prediction result includes two types: "there are potholes on the current road surface" and "there are no potholes on the current road surface".

[0075] It is understandable that when the prediction result indicates that "there are potholes on the current road surface", since the preliminary judgment result based on the personnel behavior monitoring data is also "there are potholes on the current road surface", "there are potholes on the current road surface" can be directly used as the final judgment result at this time.

[0076] When the prediction result indicates that "there are no potholes on the current road surface", the credibility of the prediction result and the preliminary judgment result is obtained respectively, and based on the credibility of the two, a judgment result (or prediction result) with higher credibility is determined to be adopted.

[0077] Specifically, referring to the method for obtaining the first credibility, the method for obtaining the second credibility can be calculated based on the following methods: 1. Determine the corresponding second credibility based on the clarity of the current outside view of the vehicle (the corresponding relationship can be set according to actual needs); 2. Calculate the corresponding second credibility based on relevant quality indicators of the collected image in front of the vehicle (such as brightness, whether there is any obstruction, etc.).

[0078] In some possible embodiments, step 2141, when the road pothole prediction result indicates that the current road surface has potholes, determining that the current road surface has potholes may include: S21411. When the road pothole prediction result indicates that there are potholes on the current road surface, obtain a first credibility corresponding to the road pothole determination result, and obtain a second credibility corresponding to the road pothole prediction result; S21412, determining a comprehensive credibility based on the first credibility and the second credibility, and judging whether the comprehensive credibility is greater than a preset comprehensive credibility threshold; S21413: If yes, determine that there are potholes on the current road surface; S21414: If not, determine that there are no potholes on the current road surface.

[0079] Specifically, when the prediction result obtained based on the current image in front of the vehicle also indicates that "there are potholes on the current road surface", although the prediction result is consistent with the result of the preliminary judgment based on the personnel behavior monitoring data, both are "there are potholes on the current road surface", but in order to ensure the reliability and accuracy of the final judgment result, the credibility of the two (the prediction result and the preliminary judgment result) can be further verified, and the credibility of the two can be comprehensively analyzed and the final judgment result can be output.

[0080] Exemplarily, based on the above embodiment, the first credibility of the preliminary judgment result and the second credibility of the prediction result can be obtained respectively; then, the first credibility and the second credibility can be weighted and summed according to the preset weight to obtain the comprehensive credibility; then, the comprehensive credibility is compared with the preset comprehensive credibility threshold. If the comprehensive credibility is greater than the threshold, "there are potholes on the current road surface" can be adopted as the final judgment result, otherwise the opposite judgment result is output (that is, it is determined that there are no potholes on the current road surface).

[0081] like Figure 2 As shown, the second embodiment of the present application provides a method for identifying potholes in a road surface, which may include the following steps: S2001, collecting an image in front of the vehicle in real time, and obtaining a road pothole prediction result corresponding to the image in front of the vehicle; S2002, obtaining the current clarity of the visual field outside the vehicle in real time; S2003: when the clarity of the outside view of the vehicle is higher than a preset second clarity threshold, judging whether there is a pothole on the current road surface based on the road pothole prediction result; S2004: When the clarity of the field of vision outside the vehicle is not higher than a preset second clarity threshold, determine whether there are potholes on the current road surface in combination with the occupant behavior monitoring data in the vehicle and the pothole prediction result.

[0082] It should be noted that the method of the embodiment of the present application is similar to the method of the first embodiment mentioned above. The difference is that the method of the embodiment of the present application first predicts potholes based on the real-time image in front of the vehicle, and at the same time judges the clarity of the outside view obtained in real time. When the outside view is clear enough, the pothole prediction result is directly used as the final basis for the pothole judgment. When the outside view is not clear enough, it is necessary to combine the pothole prediction result and the personnel behavior monitoring data in the car for comprehensive analysis and make the final pothole judgment.

[0083] It is understandable that the specific calculation process and judgment logic of the second embodiment are the same as those of the first embodiment, and detailed description is appropriately omitted here to avoid repetition. The second clarity threshold and the first clarity threshold can be set to the same or different thresholds.

[0084] Based on this, by first predicting the presence of potholes on the road based on the image in front of the vehicle, and then judging whether to further combine the behavior monitoring data of occupants in the vehicle to comprehensively identify potholes based on the clarity of the outside view, the problem of inaccurate road condition recognition based on the image in front of the vehicle due to insufficient light or bad weather is avoided, thereby effectively improving the accuracy of pothole recognition.

[0085] Please refer to Figure 3 , Figure 3 FIG. 1 is a block diagram showing the composition of a road pothole recognition device provided by some embodiments of the present application. It should be understood that the road pothole recognition device is similar to the above-mentioned Figure 1 Corresponding to the method embodiment, each step involved in the above method embodiment can be executed. The specific functions of the road pothole identification device can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0086] Figure 3 The road pothole recognition device includes at least one software function module that can be stored in a memory in the form of software or firmware or fixed in the road pothole recognition device, and the road pothole recognition device includes: The data acquisition module 310 is used to acquire the behavior monitoring data of people in the vehicle in real time; The pothole identification module 320 is used to obtain a road pothole determination result indicating that there are potholes on the current road surface when it is determined that the personnel behavior monitoring data meets a preset behavior condition.

[0087] Please refer to Figure 4 , Figure 4 FIG. 1 is a block diagram showing the composition of a road pothole recognition device provided by some embodiments of the present application. It should be understood that the road pothole recognition device is similar to the above-mentioned Figure 2 Corresponding to the method embodiment, each step involved in the above method embodiment can be executed. The specific functions of the road pothole identification device can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0088] Figure 4 The road pothole recognition device includes at least one software function module that can be stored in a memory in the form of software or firmware or fixed in the road pothole recognition device, and the road pothole recognition device includes: The result prediction module 410 is used to collect the image in front of the vehicle in real time and obtain the road pothole prediction result corresponding to the image in front of the vehicle; The visual field acquisition module 420 is used to acquire the current visual field clarity outside the vehicle in real time; A first determination module 430 is used to determine whether there is a pothole on the current road surface based on the road pothole prediction result when the clarity of the outside view of the vehicle is higher than a preset second clarity threshold; The second determination module 440 is used to determine whether there are potholes on the current road surface in combination with the personnel behavior monitoring data in the vehicle and the pothole prediction result when the clarity of the field of view outside the vehicle is not higher than a preset second clarity threshold.

[0089] It can be understood that the above-mentioned device item embodiment corresponds to the method item embodiment of the present invention. A road pothole recognition device provided by the embodiment of the present invention can implement the road pothole recognition method provided by any method item embodiment of the present invention.

[0090] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0091] like Figure 5 As shown, some embodiments of the present application provide an electronic device 500, which includes: a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 reads the program from the memory 510 through a bus 530 and executes the program to implement a method of any embodiment of the road pothole identification method as described above.

[0092] Processor 520 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 520 can be a microprocessor.

[0093] The memory 510 may be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 520 of the disclosed embodiment may be used to execute instructions in the memory 510 to implement the method shown above. The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.

[0094] Some embodiments of the present application also provide a vehicle, comprising the above-mentioned electronic device 500.

[0095] Some embodiments of the present application further provide a computer-readable storage medium having a computer program stored thereon. The computer program is executed by a processor to execute the method described in the method embodiment.

[0096] Some embodiments of the present application further provide a computer program product, which, when executed on a computer, enables the computer to execute the method described in the method embodiment.

[0097] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0098] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0100] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0101] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0102] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0103] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for identifying potholes on a road surface, characterized in that: include: Obtain real-time monitoring data of people’s behavior in the vehicle; When it is determined that the personnel behavior monitoring data meets the preset behavior condition, a road pothole determination result is obtained indicating that there are potholes on the current road surface; The personnel behavior monitoring data includes image data of the driver and passengers in the vehicle; When it is determined that the personnel behavior monitoring data meets the preset behavior condition, a road pothole determination result is obtained indicating that there are potholes on the current road surface, including: Performing behavior recognition based on the driver and passenger image data, and determining driving behavior information corresponding to the driver and passenger image data; If it is determined that the driving behavior information satisfies the preset driving behavior condition, a road pothole determination result indicating that there are potholes on the current road surface is obtained.

2. The road pothole identification method according to claim 1, characterized in that: The driving behavior information includes driving behavior information and passenger behavior information; If it is determined that the driving behavior information satisfies the preset driving behavior condition, a road pothole determination result indicating that there are potholes on the current road surface is obtained, including: If it is determined that the driving behavior information satisfies the preset driving behavior condition, and it is determined that the passenger behavior information satisfies the preset passenger behavior condition, a road pothole determination result indicating that there are potholes on the current road surface is obtained.

3. The road pothole identification method according to claim 2, characterized in that: The determining that the driving behavior information satisfies a preset driving behavior condition includes at least one of the following: When it is determined based on the driving behavior information that the driver is holding the steering wheel tightly, determining that the driving behavior information satisfies a preset driving behavior condition; Alternatively, when it is determined based on the driving behavior information that the driver's posture change information exceeds a preset first change threshold, determining that the driving behavior information satisfies a preset driving behavior condition; Alternatively, when it is determined based on the driving behavior information that the driver is gripping the steering wheel, and when it is determined that the driver's posture change information exceeds a preset second change threshold, it is determined that the driving behavior information meets the preset driving behavior condition.

4. The road pothole identification method according to claim 2, characterized in that: The determining that the passenger behavior information satisfies a preset passenger behavior condition includes at least one of the following: When it is determined based on the passenger behavior information that the passenger is holding the handrail tightly, determining that the passenger behavior information meets a preset passenger behavior condition; Alternatively, when it is determined based on the passenger behavior information that the passenger posture change information exceeds a preset third change threshold, it is determined that the passenger behavior information satisfies a preset passenger behavior condition; Alternatively, when it is determined based on the passenger behavior information that the passenger is holding the handrail tightly, and when it is determined that the passenger posture change information exceeds a preset fourth change threshold, it is determined that the passenger behavior information meets the preset passenger behavior condition.

5. The method for identifying potholes in a road surface according to claim 1, characterized in that: Also includes: Obtaining the current clarity of the visual field outside the vehicle, and determining whether the clarity of the visual field outside the vehicle is lower than a preset first clarity threshold; If yes, determining that there are potholes on the current road surface according to the road surface pothole determination result; If not, obtain a pothole prediction result, and combine the pothole prediction result with the pothole determination result to determine whether there are potholes on the current road surface; wherein the pothole prediction result includes a prediction result obtained based on recognition of an image in front of the vehicle.

6. The road pothole identification method according to claim 5, characterized in that: Before obtaining the current vehicle exterior visual clarity, the method further includes: Obtaining a first credibility corresponding to the road pothole determination result, and determining whether the first credibility is greater than a preset first credibility threshold; If yes, determining that there are potholes on the current road surface according to the road surface pothole determination result; If not, the current outside vision clarity is obtained.

7. The method for identifying potholes in a road surface according to claim 5, characterized in that: The combining the road pothole prediction result and the road pothole determination result to determine whether there is a pothole on the current road surface includes: If the road pothole prediction result indicates that there are potholes on the current road surface, it is determined that there are potholes on the current road surface; When the road pothole prediction result indicates that there is no pothole on the current road surface, obtaining a first credibility corresponding to the road pothole determination result, and obtaining a second credibility corresponding to the road pothole prediction result, and determining whether the first credibility is higher than the second credibility; If yes, determining that there are potholes on the current road surface according to the road surface pothole determination result; If not, it is determined that there are no potholes on the current road surface.

8. The road pothole identification method according to claim 7, characterized in that: When the road pothole prediction result indicates that the current road surface has potholes, determining that the current road surface has potholes includes: When the road pothole prediction result indicates that a pothole exists on the current road surface, obtaining a first credibility corresponding to the road pothole determination result, and obtaining a second credibility corresponding to the road pothole prediction result; Determining a comprehensive credibility based on the first credibility and the second credibility, and judging whether the comprehensive credibility is greater than a preset comprehensive credibility threshold; If so, it is determined that there are potholes on the current road surface; If not, it is determined that there are no potholes on the current road surface.

9. A method for identifying potholes on a road surface, characterized in that: include: Collecting the image in front of the vehicle in real time, and obtaining the road pothole prediction result corresponding to the image in front of the vehicle; Get the current clarity of the outside view in real time; When the clarity of the outside view is higher than a preset second clarity threshold, judging whether there is a pothole on the current road surface based on the road pothole prediction result; When the clarity of the visual field outside the vehicle is not higher than a preset second clarity threshold, combining the personnel behavior monitoring data in the vehicle and the road pothole prediction result, determining whether there is a pothole on the current road surface; The personnel behavior monitoring data includes image data of the driver and passengers in the vehicle; The combining of the occupant behavior monitoring data in the vehicle and the road pothole prediction result to determine whether there is a pothole on the current road surface includes: Performing behavior recognition based on the driver and passenger image data, and determining driving behavior information corresponding to the driver and passenger image data; If it is determined that the driving behavior information satisfies the preset driving behavior condition, a road pothole determination result indicating that there are potholes on the current road surface is obtained; The road pothole determination result and the road pothole prediction result are combined to determine whether there are potholes on the current road surface.

10. A road pothole identification device, characterized in that: include: A data acquisition module is used to acquire the behavior monitoring data of people in the vehicle in real time; A pothole recognition module, for obtaining a pothole determination result of a current road surface having potholes when it is determined that the personnel behavior monitoring data meets a preset behavior condition; The personnel behavior monitoring data includes image data of the driver and passengers in the vehicle; The pothole recognition module is specifically used for: Performing behavior recognition based on the driver and passenger image data, and determining driving behavior information corresponding to the driver and passenger image data; If it is determined that the driving behavior information satisfies the preset driving behavior condition, a road pothole determination result indicating that there are potholes on the current road surface is obtained.

11. A road pothole identification device, characterized in that: include: A result prediction module, used for collecting the image in front of the vehicle in real time and obtaining the road pothole prediction result corresponding to the image in front of the vehicle; The visual field acquisition module is used to obtain the current visual field clarity outside the vehicle in real time; A first determination module is used to determine whether there is a pothole on the current road surface based on the road pothole prediction result when the clarity of the outside view of the vehicle is higher than a preset second clarity threshold; A second determination module is used to determine whether there is a pothole on the current road surface in combination with the personnel behavior monitoring data in the vehicle and the road pothole prediction result when the clarity of the outside view of the vehicle is not higher than a preset second clarity threshold; The personnel behavior monitoring data includes image data of the driver and passengers in the vehicle; The second determination module is specifically used to: Performing behavior recognition based on the driver and passenger image data, and determining driving behavior information corresponding to the driver and passenger image data; If it is determined that the driving behavior information satisfies the preset driving behavior condition, a road pothole determination result indicating that there are potholes on the current road surface is obtained; The road pothole determination result and the road pothole prediction result are combined to determine whether there are potholes on the current road surface.

12. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the method for identifying potholes in a road surface according to any one of claims 1 to 9 can be implemented when the processor executes the program.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the road pothole identification method according to any one of claims 1 to 9 is executed.

14. A vehicle, characterized in that: Comprising the electronic device as claimed in claim 12.

Images