Sidewalk disease detection method, device, electronic equipment and storage medium

Abstract

An embodiment of the present invention provides a method for detecting sidewalk defects, the method comprising: when a patrol device scans a sidewalk, obtaining monitoring data of the sidewalk, the monitoring data including the sidewalk pavement and the feet of pedestrians; determining the support points of each of the pedestrian's feet on the sidewalk pavement according to the monitoring data; and determining whether the sidewalk pavement has defects according to the support points. The sidewalk pavement image and the position of the pedestrian's feet are obtained by the patrol device, the pedestrian's foothold is determined, and whether the sidewalk has defects is determined according to the foothold. Since the morphological distribution of each foothold is different, the principle that a load is generated on the road surface when the foothold is placed can be used, and the foothold is the real support point of the road surface, thereby improving the accuracy of sidewalk defect detection.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a sidewalk disease detection method, device, electronic equipment and storage medium. Background Art

[0002] Sidewalks are roads for pedestrians to pass through. The pavement conditions of sidewalks affect pedestrians' experience. Due to the long use of sidewalks, the inconsistency of pedestrian directions, the passing of non-motor vehicles, and other reasons, some sidewalk pavement defects that affect pedestrians' passage appear, such as sidewalk sinking, arching, and other sidewalk defects. The existing sidewalk defects are manually detected by relevant departments such as streets or transportation, which consumes a lot of manpower and has low efficiency. Summary of the invention

[0003] The embodiment of the present invention provides a method for detecting sidewalk defects, which aims to solve the problem that some defects that affect pedestrian traffic may occur on the sidewalk surface due to the long use time of the sidewalk, the inconsistency of pedestrian directions, the passing of non-motor vehicles, etc. The sidewalk image is obtained by the patrol equipment, and the sidewalk image includes the sidewalk surface and the feet of pedestrians. The footsteps of each pedestrian are determined according to the sidewalk image, and whether the sidewalk has defects is determined according to the footsteps. Since the morphological distribution of each footstep is different, the principle that the footstep will generate a load on the road surface when landing can be used. The footstep is the real support point of the road surface, thereby improving the accuracy of sidewalk defect detection.

[0004] In a first aspect, an embodiment of the present invention provides a method for detecting sidewalk damage, the method comprising:

[0005] Get monitoring data of sidewalks;

[0006] Determine the support point of each pedestrian's foot on the sidewalk according to the monitoring data;

[0007] Determine whether the sidewalk pavement has any damage based on the support points.

[0008] Optionally, determining the support point of each pedestrian's foot on the sidewalk according to the monitoring data includes:

[0009] Performing posture recognition processing on the monitoring data to obtain the foot posture of each of the pedestrians' feet on the sidewalk;

[0010] According to the foot posture, the support point of each pedestrian's foot on the sidewalk is determined.

[0011] Optionally, determining the support point of each pedestrian's foot on the sidewalk according to the foot posture includes:

[0012] When the foot posture is landing, extracting a first characteristic vector corresponding to the lowest point of the first foot, and determining a first support point according to the first characteristic vector;

[0013] When the foot posture is lifting the foot, a second characteristic vector corresponding to the lowest point of the second foot is extracted, and a second support point is determined according to the second characteristic vector.

[0014] Optionally, determining whether the sidewalk pavement is damaged according to the support point includes:

[0015] Determining the distribution type of the support points according to the positions of the support points;

[0016] Whether the sidewalk pavement is damaged is determined according to the distribution type of the support points.

[0017] Optionally, determining whether the sidewalk pavement is damaged according to the distribution type of the support points includes:

[0018] Determining whether the distribution type of the support points is a plane type;

[0019] If the distribution type of the support points is a plane type, it is determined that the sidewalk pavement has no disease;

[0020] If the distribution type of the support points is a non-planar type, it is determined that the sidewalk pavement has a defect.

[0021] Optionally, if the distribution type of the support points is a non-planar type, determining that the sidewalk pavement has a disease includes:

[0022] If the distribution type of the support points is a sinking curved surface type, it is determined that the sidewalk pavement has a sinking disease;

[0023] If the distribution type of the support points is a bulging curved surface type, it is determined that the sidewalk pavement has bulging damage.

[0024] Optionally, determining whether the sidewalk pavement is damaged according to the support point includes:

[0025] Determine a support point cloud of the sidewalk pavement according to the support points;

[0026] Determine, based on historical monitoring data, a scan point cloud of the sidewalk pavement when no one is walking on it;

[0027] Based on the support point cloud and the scanning point cloud, determine whether the sidewalk pavement has any damage.

[0028] In a second aspect, an embodiment of the present invention further provides a sidewalk disease detection device, the sidewalk disease detection device comprising:

[0029] An acquisition module, used to acquire monitoring data of the sidewalk, wherein the monitoring data includes the sidewalk pavement and the feet of pedestrians;

[0030] A processing module, used to determine the support point of each pedestrian's foot on the sidewalk according to the monitoring data;

[0031] The determination module is used to determine whether the sidewalk pavement has any damage according to the support points.

[0032] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the sidewalk disease detection method provided in the embodiment of the present invention when executing the computer program.

[0033] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the sidewalk disease detection method provided in the embodiment of the invention are implemented.

[0034] In an embodiment of the present invention, monitoring data of a sidewalk is obtained, and the monitoring data includes the sidewalk pavement and the feet of pedestrians; the supporting points of each of the pedestrian's feet on the sidewalk pavement are determined according to the monitoring data; and whether the sidewalk pavement is damaged is determined according to the supporting points. The supporting points of the sidewalk pavement for the pedestrian's feet are determined by the monitoring data of the sidewalk. Since the supporting points need to bear the load of the human body, the actual condition of the sidewalk pavement can be determined more accurately, thereby improving the accuracy of sidewalk damage detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 is a flow chart of a sidewalk disease detection method provided by an embodiment of the present invention;

[0037] Figure 2 is a schematic structural diagram of a sidewalk disease detection device provided in an embodiment of the present invention;

[0038] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] like Figure 1 As shown, Figure 1 1 is a flow chart of a sidewalk disease detection method provided by an embodiment of the present invention. The sidewalk disease detection method comprises the steps of:

[0041] 101. Obtain monitoring data of sidewalks.

[0042] In an embodiment of the present invention, the above-mentioned sidewalk disease detection method can be deployed on a server, an edge or a vehicle-mounted terminal. The above-mentioned server can be an electronic device such as a cloud server or a local server, the above-mentioned edge can be an electronic device arranged on the roadside, and the above-mentioned vehicle-mounted terminal can be an electronic device arranged on a patrol device.

[0043] The patrol equipment can be generally used to patrol, detect and monitor certain specific areas. The patrol equipment of the embodiment of the present invention is used to patrol and monitor the sidewalk area. The patrol equipment is provided with a photographing device, which can be used to photograph the sidewalk area.

[0044] The above monitoring data may be an image captured in real time by a shooting device on the patrol device and uploaded to an electronic device. Specifically, when the patrol device performs a road scanning task, the electronic camera on the patrol device captures the sidewalk in real time at a certain height and angle to obtain a captured image of the sidewalk, which includes the sidewalk pavement and the feet of pedestrians. After obtaining the captured image, the patrol device uploads the captured image to the electronic device in real time so that the electronic device obtains the monitoring data of the sidewalk. The above monitoring data may also be uploaded to the electronic device by the user himself, and the above pedestrian feet refer to the soles of the pedestrians.

[0045] The above monitoring data may be a sequence of multiple consecutive frames of images captured by the inspection equipment.

[0046] The above-mentioned shooting device can be a three-dimensional camera, and the above-mentioned three-dimensional camera can collect image depth information. The sidewalk is photographed by the three-dimensional camera to obtain a continuous multi-frame image sequence as monitoring data of the sidewalk.

[0047] The above-mentioned three-dimensional camera can also be called a 3D camera, which is a camera designed to capture depth information and conventional two-dimensional images. Specifically, the height and angle of the 3D camera are set so that the entire sidewalk and the feet of pedestrians can be captured. The height and angle need to be manually adjusted according to the shooting scene until the 3D camera's shooting picture can capture the sidewalk pavement and pedestrians' feet. The monitoring data of the sidewalk pavement and pedestrians' feet captured by the 3D camera is uploaded to the electronic device for image analysis. Through the monitoring data with depth information, a 3D model of pedestrians' feet and sidewalk pavement can be created.

[0048] 102. Determine the support point of each pedestrian's foot on the sidewalk based on the monitoring data.

[0049] In the embodiment of the present invention, the above-mentioned support point is the support point where the pedestrian's foot lands on the sidewalk, which can also be understood as the load point of the sidewalk. The above-mentioned support point of the pedestrian's foot on the sidewalk can also be the various foot support points and foot lifting support points when the pedestrian walks on the sidewalk. The above-mentioned monitoring data can be a continuous multi-frame image sequence captured by the inspection equipment.

[0050] After obtaining the monitoring data, the monitoring data can be identified and processed by the trained support point recognition model to obtain the support points of each pedestrian's foot on the sidewalk. The support point recognition model can be a recognition model based on a deep convolutional neural network. The support point recognition model can be trained by a large number of sample images with support point label data to obtain a trained support point recognition model. The sample images are obtained in the same way as the monitoring data. The support point label data is obtained by manual annotation by relevant personnel or by machine learning methods. Specifically, during the training process, the sample images are input into the support point recognition model to be trained to obtain the support point recognition results of the sample images. The support point recognition results of the sample images are calculated with the support point label data corresponding to the sample images to obtain the loss function. With the goal of minimizing the loss function, the parameters of the support point recognition model to be trained are adjusted by the back propagation algorithm. The parameter adjustment process is iterated until the support point recognition model to be trained converges at the minimum of the loss function or the number of iterations reaches a preset number. The training is stopped to obtain a trained support point recognition model. After the trained support point recognition model is obtained, the monitoring data is input into the trained support point recognition model for processing, and the support points of the pedestrian's feet on the sidewalk are obtained through the output of the trained support point recognition model.

[0051] 103. Determine whether there are any defects on the sidewalk pavement based on the support points.

[0052] In an embodiment of the present invention, the above-mentioned defects may refer to defects on a sidewalk, and the above-mentioned defects on the sidewalk pavement may be defects such as potholes on the sidewalk pavement, road depressions, road subsidence, road elevation, wavy pavement, hollows under bricks, etc. that affect pedestrian traffic. Specifically, the defects may include defect type, defect location, and defect degree. The defect type may include potholes on the road, road subsidence, wavy pavement, hollows under bricks, etc. The defect location may be the location of the defect in the target image, and the defect location may be determined by the support point of the pedestrian's foot. The degree of the defect may refer to the degree of damage to the road surface, which can be used to describe the flatness and safety of the road surface, and the above-mentioned degree of the defect may be determined based on the height difference between the support point and the sidewalk pavement.

[0053] After obtaining the support points of each pedestrian's foot on the sidewalk, the support points can be used to determine whether the sidewalk has any damage. The principle that footfall will generate load on the road surface can be used, and the footfall point is the real support point of the road surface. By judging whether the distribution of the support points conforms to the plane distribution, if the distribution of the support points conforms to the plane distribution, it is judged that there is no damage at the location of the sidewalk. If not, it can be determined that the sidewalk has a disease.

[0054] Furthermore, it can be determined whether the distribution of the support points conforms to the distribution of the sinking surface, or whether the distribution of the support points conforms to the distribution of the arching surface. If the distribution of the support points conforms to the distribution of the sinking surface or the arching surface, it is determined that the sidewalk pavement has a disease.

[0055] In the embodiment of the present invention, monitoring data of the sidewalk is obtained, and the monitoring data includes the sidewalk pavement and the feet of pedestrians; the support points of each pedestrian's feet on the sidewalk pavement are determined according to the monitoring data; and whether the sidewalk pavement has a disease is determined according to the support points. The support points of the sidewalk pavement for the pedestrian's feet are determined by the monitoring data of the sidewalk. Since the support points need to bear the load of the human body, the actual situation of the sidewalk pavement can be determined more accurately, thereby improving the accuracy of sidewalk disease detection.

[0056] Optionally, in the step of determining the support point of each pedestrian's foot on the sidewalk pavement based on the monitoring data, the monitoring data can be processed for posture recognition to obtain the foot posture of each pedestrian's foot on the sidewalk pavement; and based on the foot posture, the support point of each pedestrian's foot on the sidewalk pavement is determined.

[0057] In the embodiment of the present invention, the monitoring data may be a two-dimensional image sequence or a three-dimensional image sequence. The two-dimensional image sequence may be understood as an image sequence obtained by photographing a pedestrian path with a two-dimensional camera, and the three-dimensional image sequence may be understood as an image sequence obtained by photographing a pedestrian path with a three-dimensional camera. It should be noted that the difference between the two-dimensional image sequence and the three-dimensional image sequence is that the images in the two-dimensional image sequence do not have depth information, while the images in the three-dimensional image sequence have depth information.

[0058] The above-mentioned posture recognition processing can be human posture recognition, which refers to the recognition and analysis of human posture through computer vision technology. The human foot posture information can be parsed and obtained by analyzing the monitoring data or video captured and uploaded by the 3D camera, and the above-mentioned human foot posture information can be put into a trained foot posture recognition model to obtain the foot posture of each pedestrian's foot on the sidewalk. The foot posture can be the posture change of the pedestrian's foot when walking. Specifically, the foot posture can be divided into a starting posture and a landing posture, and the support point of each pedestrian's foot on the sidewalk is determined according to the starting posture and the landing posture.

[0059] In a possible embodiment, the monitoring data may be a two-dimensional image sequence, and the monitoring data may be subjected to support point prediction processing through a time series prediction model to obtain the support point of each pedestrian's foot on the sidewalk. The time series prediction model may be a time series model based on a recurrent neural network (RNN) or a long short-term memory (LSTM) network.

[0060] In another possible embodiment, the monitoring data may be a two-dimensional image sequence, and the support point of each pedestrian's foot on the sidewalk is determined based on the monitoring data. The pedestrian's foot in the monitoring data may be identified and located by target detection technology to obtain the position information of the pedestrian's foot. The pedestrian's foot posture information may be obtained by a two-dimensional posture estimation algorithm. The foot posture information includes the position and rotation angle of the pedestrian's foot. The support point of each pedestrian's foot on the sidewalk is determined by combining the pedestrian's foot position and the pedestrian's foot posture information. The two-dimensional posture estimation algorithm may be a posture recognition algorithm based on networks such as SIFT, SURF, ORB, HOG, and DPM.

[0061] In another possible embodiment, the monitoring data may be a three-dimensional image sequence, and the image information and depth information of the monitoring data collected by the 3D camera may be used to construct a 3D model of the sidewalk and the feet of pedestrians according to the image information and depth information of the monitoring data, and the support point position of the pedestrian's feet may be obtained according to the 3D modeling and spatial coordinates of the sidewalk and the feet of pedestrians. Specifically, after constructing the 3D model of the sidewalk and the feet of pedestrians, a three-dimensional posture recognition algorithm may be used to identify the feet of pedestrians and perform positioning processing on the feet of pedestrians. The three-dimensional posture recognition algorithm may determine the posture or movement of the human body in three-dimensional space by analyzing and calculating the information such as the joint angle, position, and direction of the human body. Furthermore, the three-dimensional posture recognition algorithm may be used to analyze and calculate the information such as the joint angle, position, and direction of the feet of pedestrians to determine the posture or movement of the feet of pedestrians in three-dimensional space. The above-mentioned three-dimensional posture recognition algorithm may be a posture recognition algorithm based on networks such as PointNet, PointCNN, DGCNN, and SO-Net.

[0062] In another possible embodiment, the monitoring data can be a three-dimensional image sequence, and the support point of each pedestrian's foot on the sidewalk can be determined based on the monitoring data. The image information and depth information of the sidewalk and the pedestrian's foot captured by the 3D camera can be synchronously uploaded to the electronic device for image analysis. The electronic device constructs a 3D model of the sidewalk and the pedestrian's foot through the image information and depth information. According to the relative spatial coordinates of the 3D model of the sidewalk and the 3D model of the pedestrian's foot, a judgment is made. When the 3D model of the sidewalk and the 3D model of the pedestrian's foot are in stable contact, the contact point of the 3D model of the sidewalk and the 3D model of the pedestrian's foot is determined as the pedestrian's landing point, and the support point of each pedestrian's foot on the sidewalk can be determined by the pedestrian's landing point. When the contact point position of the 3D model of the sidewalk and the 3D model of the pedestrian's foot no longer changes within a preset time after the contact, it can be determined that the contact between the 3D model of the sidewalk and the 3D model of the pedestrian's foot is stable contact.

[0063] Optionally, in the step of determining the support point of each pedestrian's foot on the sidewalk according to the foot posture, when the foot posture is landing, the first characteristic vector corresponding to the lowest point of the first foot can be extracted, and the first support point can be determined based on the first characteristic vector; when the foot posture is lifting, the second characteristic vector corresponding to the lowest point of the second foot can be extracted, and the second support point can be determined based on the second characteristic vector.

[0064] In an embodiment of the present invention, the above-mentioned foot posture includes landing and lifting off. Landing refers to the posture of the pedestrian's feet stepping on the sidewalk pavement, and lifting off refers to the posture of the pedestrian's feet leaving the sidewalk pavement. The support point can be determined by detecting the position where the pedestrian's feet overlap with the sidewalk pavement.

[0065] Specifically, after obtaining the monitoring data, the foot posture recognition model can be used to perform foot state recognition processing on the images in the monitoring data. The foot posture output by the foot posture recognition model includes foot landing and foot lift-off. The foot posture recognition model can be a posture recognition model based on networks such as SIFT, SURF, ORB, HOG, DPM, etc. The foot posture recognition model can be trained by a large number of foot image samples with foot posture labels.

[0066] When the foot posture is landing, the lowest point of the foot in each frame image in the image sequence is detected by the foot key point detection model to obtain the first lowest point of the foot corresponding to each pedestrian's foot. Each pedestrian's foot corresponds to a first lowest point of the foot when landing. The first lowest point of the foot in adjacent frame images or continuous frame images is used to determine the first feature vector corresponding to the first lowest point of the foot. The first feature vector includes direction and length. Specifically, for a pedestrian's foot, the nth frame image and the n+1th frame image are adjacent frame images, where the first lowest point of the foot (x n ,y n ), the n+1th frame image has the lowest point of the first foot (x n+1 ,y n+1 ), the first eigenvector is (x n+1 -x n ,y n+1 -y n ), the above (x n+1 -x n ,y n+1 -y n ) represents the lowest point of the first foot in the nth frame image (x n ,y n ) to the lowest point of the first foot in the n+1th frame image (x n+1 ,y n+1 ) vector.

[0067] After obtaining the first eigenvector, the modulus of the first eigenvector can be calculated. The modulus of the first eigenvector represents the lowest point (x n ,y n ) to the lowest point of the first foot in the n+1th frame image (x n+1 ,y n+1 ). When the modulus of the first eigenvector is equal to 0, the lowest point of the first foot (x n+1 ,y n+1 ) is determined as the first support point. The first support point refers to the support point where the pedestrian stably contacts the sidewalk after landing.

[0068] When the foot posture is to start, the lowest point of the foot in each frame image in the image sequence is detected by the foot key point detection model to obtain the second lowest point of the foot corresponding to each pedestrian's foot. Each pedestrian's foot corresponds to a second lowest point of the foot when landing. The second feature vector corresponding to the second lowest point of the foot is determined by the second lowest point of the foot in adjacent frame images or continuous frame images. The second feature vector includes direction and length. Specifically, for a pedestrian's foot, the mth frame image and the m+1th frame image are adjacent frame images, where the second lowest point of the foot (x m ,y m ), the m+1th frame image has the lowest point of the first foot (x m+1 ,y m+1 ), the first eigenvector is (x m+1 -x m ,y m+1 -y m ), the above (x m+1 -x m ,y m+1 -y m ) represents the lowest point of the second foot in the mth frame image (x m ,y m ) to the lowest point of the second foot in the m+1th frame image (x m+1 ,y m+1 ) vector.

[0069] After obtaining the second eigenvector, the modulus of the second eigenvector can be calculated. The modulus of the second eigenvector represents the lowest point (x m ,y m ) to the lowest point of the second foot in the m+1th frame image (x m+1 ,y m+1 ). When the modulus of the second eigenvector changes from 0 to greater than 0, the lowest point of the second foot (x m ,y m ) is determined as the second support point. The second support point refers to the support point where the pedestrian stably contacts the sidewalk when the pedestrian lifts his foot.

[0070] In a possible embodiment, the image information and depth information of the sidewalk and the pedestrian's feet captured by the 3D camera are synchronously uploaded to the electronic device for image analysis. The electronic device constructs a 3D model of the sidewalk and the 3D model of the pedestrian's feet in the 3D modeling space through the image information and depth information, and performs human posture recognition on the 3D model of the pedestrian's feet to determine the foot posture of each pedestrian's foot. When the foot posture is landing, the first foot lowest point corresponding to the adjacent frame image or continuous frame image is taken in the 3D modeling space, and the first feature vector is determined by the first foot lowest point corresponding to the adjacent frame image or continuous frame image, and the first support point is determined according to the first feature vector; when the foot posture is starting, the second foot lowest point corresponding to the adjacent frame image or continuous frame image is taken in the 3D modeling space, and the second feature vector is determined by the second foot lowest point corresponding to the adjacent frame image or continuous frame image, and the second support point is determined according to the second feature vector. In general, the support point of each pedestrian's foot on the sidewalk is determined by the lowest point of the current footstep posture, and the support point is the stable contact point between the pedestrian's foot and the sidewalk.

[0071] In a possible embodiment, the image information and depth information of the sidewalk pavement and the feet of pedestrians captured by a 3D camera are synchronously uploaded to an electronic device for image analysis. The electronic device constructs a 3D model of the sidewalk pavement and a 3D model of the pedestrian's feet in a 3D modeling space through the image information and depth information, performs motion simulation on the 3D model of the pedestrian's feet based on the three-dimensional image sequence, determines the posture of the pedestrian's feet based on the motion simulation results, and determines the first characteristic vector of each pedestrian's foot when landing and the second characteristic vector of each pedestrian's foot when taking off based on the motion simulation results.

[0072] Optionally, in the step of determining whether the sidewalk pavement is damaged based on the support points, the distribution type of the support points can be determined based on the positions of the support points; and determining whether the sidewalk pavement is damaged based on the distribution type of the support points.

[0073] In an embodiment of the present invention, the distribution type of the support points may be a plane distribution type or a curved surface distribution type. Specifically, the plane distribution type may be that all the pedestrian support points determined on the pedestrian road present a plane shape distribution in the image space of the current image. The curved surface distribution type may be that all the pedestrian support points determined on the pedestrian road present a curved surface shape distribution in the image space of the current image.

[0074] The above surface distribution type can be divided into a sinking surface type and an arching surface type. The sinking surface type can be that all pedestrian support points determined on the sidewalk generally present a downward concave shape in the image space of the current image; the arching surface type can be that all pedestrian support points determined on the sidewalk generally present a downward convex shape in the image space of the current image. Whether the sidewalk pavement has defects is determined by the distribution type of the support points. If the distribution type of the support points is a plane type, it is determined that the sidewalk pavement has no defects; if the distribution type of the support points is a non-plane type, it is determined that the sidewalk pavement has defects.

[0075] Optionally, in the step of determining whether the sidewalk pavement has defects based on the distribution type of support points, it can be determined whether the distribution type of support points is a plane type; if the distribution type of support points is a plane type, it is determined that the sidewalk pavement has no defects; if the distribution type of support points is a non-plane type, it is determined that the sidewalk pavement has defects.

[0076] In an embodiment of the present invention, whether the distribution type of the support points is a plane type can be determined by analyzing and judging by Ripley's K function or Clark-Evans index. Specifically, Ripley's K function is a spatial statistical analysis method for calculating the distribution of plane point patterns; Clark-Evans index is a spatial statistical tool for describing the degree of point pattern aggregation.

[0077] In a possible embodiment, Ripley's K is used to determine whether the distribution type of the support points is a plane type, which can be used to evaluate the degree of aggregation, randomness or uniformity of points in a region and can be compared with control data. The K function is defined as:

[0078] K(r)=(1 / n^2)*ΣΣI(||x_i-x_j||<=r)

[0079] Where n is the total number of points, x_i is the position of the i-th point, ||x_i-x_j|| represents the Euclidean distance between points i and j, r is the radius, and I(||x_i-x_j||<=r) is an indicator function, which is 1 when ||x_i-x_j||<=r, otherwise it is 0.

[0080] The K function reflects the average distance between the point pairs contained in a circle with a radius of r. If K(r) is less than r, it indicates that the point pattern is clustered within the range of distance r; if K(r) is greater than r, it indicates that the point pattern is dispersed within the range of distance r; if K(r) is equal to r, it indicates that the point pattern presents the characteristics of random distribution. In a plane point set, K(r) should be approximately a constant, while in a non-plane point set, K(r) will increase as r increases. Specifically, if K(r) is linearly related to r, it indicates that the point set has random distribution characteristics; if K(r) presents a shape similar to a normal distribution curve, it indicates that the point set is clustered; and if K(r) presents a single-peak or multi-peak shape, there may be a periodic structure or local density change. Therefore, we can judge whether the support point has a planar structure by comparing the relationship between K(r) and r.

[0081] In a possible embodiment, the Clark-Evans index can be used to determine whether the distribution type of the support points is a plane type. Clark-Evans is based on the nearest neighbor distance and can be used to determine whether a point pattern presents a random, uniform or clustered distribution. The Clark-Evans index is defined as follows: r_m is the average of the nearest neighbor distances, and bar(d) is the average of the distances between all points in the point pattern. When C>0, it indicates that the point pattern presents a clustered distribution; when C<0, it indicates that the point pattern presents a dispersed distribution; and when C=0, it indicates that the point pattern presents a random distribution.

[0082] C=(r_m / 2*bar(d))–1

[0083] To determine whether the distribution type of support points is a plane type by using the Clark-Evans index, it is necessary to calculate the distances between all support points and sort these distances from small to large. For each support point, calculate the average of the distances between it and its k nearest neighbors, where k is a fixed integer, usually 6 or 8. Put the average distances of all support points into a histogram and count the number of values ​​n with the most distances in the histogram. If n is greater than or equal to 5, the distribution type of the support point set is considered to be a plane type; otherwise, it is not a plane type.

[0084] Optionally, in the step of determining whether the sidewalk pavement has a disease if the distribution type of the support points is a non-planar type, if the distribution type of the support points is a sinking curved surface type, it can be determined that the sidewalk pavement has a sinking disease; if the distribution type of the support points is a arched curved surface type, it is determined that the sidewalk pavement has an arching disease.

[0085] In an embodiment of the present invention, the above-mentioned disease may refer to a disease of a sidewalk. The above-mentioned disease of the sidewalk pavement may be a disease that affects pedestrian traffic, such as potholes, road depressions, road subsidence, road elevation, wavy road surface, and hollows under bricks. Among them, road depressions and road subsidence correspond to sinking diseases. If the distribution type of the support points is a sinking surface type, it can be determined that the sidewalk pavement is a road depression or a road subsidence. Road elevation corresponds to an arching disease. If the distribution type of the support points is an arching surface type, it can be determined that the sidewalk pavement is an elevated pavement. Road potholes and wavy road surfaces correspond to concave-convex surfaces. If the distribution type of the support points is a concave-convex surface type, it can be determined that the disease of the sidewalk pavement is a road pothole or a wavy road surface. The above-mentioned concave-convex surfaces can be understood as both sinking surfaces and concave-convex surfaces.

[0086] After determining that the sidewalk has a disease, the type of disease, the location of the disease, and the degree of the disease can be further determined. The type of disease can include sinking diseases such as potholes, road depressions, and road settlement, or arching diseases such as wavy road surfaces and road bulges. The location of the disease can be the location of the disease in the target image, and the location of the disease can be determined by the support point of the pedestrian's foot. The degree of the disease can refer to the degree of damage to the road surface, which can be used to describe the flatness and safety of the road surface. Specifically, the sinking disease can be manifested as uneven road surface, depressions, or road sinking. The long-term existence of sinking diseases may also cause serious damage to the road structure, and even dangerous situations such as road collapse. Arching disease refers to the phenomenon of bulging or bulging on the road surface, which usually occurs on the base layer or subbase layer under the asphalt concrete layer of the road surface. This disease will cause the road surface to be uneven, and vehicles will feel bumpy when driving, affecting driving safety and comfort.

[0087] In a possible embodiment, the surface distribution corresponding to each disease type of the sidewalk can be determined in advance, and each disease type corresponds to a surface distribution. After obtaining the support points of each pedestrian's foot on the sidewalk, the distribution of the support points is determined, and the distribution of the support points is compared with the surface distribution corresponding to each disease type. If the comparison is a hit, it is determined that the disease type corresponding to the hit surface distribution is the disease type that appears on the sidewalk. If the comparison is not a hit, it is determined that no disease appears on the sidewalk. Further, each disease type corresponds to multiple surface distributions. For a disease type, each surface distribution corresponds to a different degree of disease. After obtaining the support points of each pedestrian's foot on the sidewalk, the distribution of the support points is determined, and the distribution of the support points is compared with the surface distribution of each degree of disease. If the comparison is a hit, it is determined that the disease type corresponding to the hit surface distribution is the disease type that appears on the sidewalk, and the degree of disease corresponding to the hit surface distribution is the degree of disease of the disease type that appears on the sidewalk.

[0088] Optionally, in the step of determining whether the sidewalk pavement has any damage based on the support points, the support point cloud of the sidewalk pavement can be determined based on the support points; the scanning point cloud of the sidewalk pavement when no one is walking can be determined based on historical monitoring data; and based on the support point cloud and the scanning point cloud, it can be determined whether the sidewalk pavement has any damage.

[0089] In the embodiment of the present invention, the monitoring data is a three-dimensional image sequence, and the support points determined by the three-dimensional image sequence also include depth information. Therefore, the support point cloud of the sidewalk pavement can be determined based on the depth information of the support points.

[0090] The above-mentioned support point cloud can be a collection of support points at different positions of the sidewalk, and the above-mentioned scan cloud can be a collection of scan points at different positions of the sidewalk when no one is walking. Specifically, multiple point cloud files can be loaded into the same scene through point cloud processing software, and registered and fused to create a large point cloud. For example, PCL (Point Cloud Library) or Open3D can be used to realize the collection of points at different positions of the sidewalk into support point clouds or scan point clouds.

[0091] In a possible embodiment, support points at different positions on the sidewalk can be gathered into a cloud based on PCL (Point Cloud Library). Specifically, PCL is an open source, general 3D point cloud processing library that provides a series of algorithms and tools that can be used for processing, filtering, reconstruction, segmentation, registration, visualization and other aspects of 3D point cloud data. PCL can process 3D point data obtained from various sensors, and the point data can be support point data or scanning point data converted from image information captured by a 3D camera. The support point or scanning point data is converted into a point cloud object through the point cloud database in PCL, and the point cloud objects at different positions are merged into a support point cloud object or a scanning point cloud object in an additive manner through the point cloud object. Based on the support point cloud and the scanning point cloud, determine whether the sidewalk pavement is damaged.

[0092] In a possible embodiment, the collection of points at different positions on the sidewalk pavement into a cloud can be implemented based on Open3D. Open3D is an open source point cloud and mesh processing library that supports rapid development and processing of 3D data for processing three-dimensional data. Specifically, before using Open3D to collect points at different positions into a cloud, the coordinate data of the points needs to be converted into the type in Open3D. Generally speaking, the support point data or scan point data converted from the image information captured by the 3D camera can be used to store the point data as a Numpy array, and the open3d.geometry.PointCloud class can be used to create a point cloud object. Different point cloud objects are merged into support point cloud objects or scan point cloud objects by adding point cloud objects. Based on the support point cloud and the scan point cloud, determine whether the sidewalk pavement is damaged.

[0093] After determining the support point cloud and the scan point cloud, the support point cloud and the scan point cloud can be aligned in three-dimensional space. After alignment, each support point corresponds to a spatial coordinate (x i ,y i , z i ), each scanning point also corresponds to a spatial coordinate (x j ,y j , z j ), determine whether the z coordinate of the support point is the same as the z coordinate of the scanning point under the same horizontal plane coordinate. If the z coordinate of the support point is the same as the z coordinate of the scanning point under the same horizontal plane coordinate, it means that the support point is in the same spatial position as the scanning point, and the support point is a normal support point. If the z coordinate of the support point is different from the z coordinate of the scanning point under the same horizontal plane coordinate, it means that the support point is different from the spatial position of the scanning point, and the support point is an abnormal support point. The area where the abnormal support point is located can be determined as the diseased area of ​​the sidewalk pavement. Further, it can be determined whether the support point adjacent to the abnormal support point is an abnormal support point. If the support point adjacent to the abnormal support point is also an abnormal support point, the area where the abnormal support point and its adjacent abnormal support point are located is determined to be the diseased area of ​​the sidewalk pavement.

[0094] After determining the abnormal support point, it is further determined whether the abnormal support point is higher than the corresponding scanning point or lower than the corresponding scanning point. If the abnormal support point is higher than the corresponding scanning point, the abnormal support point is determined to be an arched support point. The area where the arched support point is located can be determined as the arched area of ​​the sidewalk pavement. Further, it can be determined whether the support point adjacent to the arched support point is an arched support point. If the support point adjacent to the arched support point is also an arched support point, the area where the arched support point and its adjacent arched support point are located is determined to be the arched area of ​​the sidewalk pavement. If the abnormal support point is lower than the corresponding scanning point, the abnormal support point is determined to be a sinking support point. The area where the sinking support point is located can be determined as the sinking area of ​​the sidewalk pavement. Further, it can be determined whether the support point adjacent to the sinking support point is a sinking support point. If the support point adjacent to the sinking support point is also a sinking support point, the area where the sinking support point and its adjacent sinking support point are located is determined to be the sinking area of ​​the sidewalk pavement. Furthermore, if the support points adjacent to the sinking support point include both sinking support points and arching support points, the area where the sinking support point and its adjacent abnormal support points are located is determined to be the concave-convex area of ​​the sidewalk pavement.

[0095] In a possible embodiment, the above-mentioned determination of whether the sidewalk pavement has defects based on the support point cloud and the scanning point cloud can be that the support point cloud and the scanning point cloud are placed into a trained plane distribution confirmation model to obtain the distribution type of the support points of the current sidewalk pavement. If the distribution type of the support points of the current sidewalk pavement is a plane type, it is determined that the sidewalk pavement has no defects; if the distribution type of the support points of the current sidewalk pavement is a sinking surface type, it is determined that the sidewalk pavement has a sinking defect; if the distribution type of the support points of the current sidewalk pavement is a arched surface type, it is determined that the sidewalk pavement has an arching defect.

[0096] It should be noted that the sidewalk disease detection method provided in the embodiment of the present invention can be applied to devices such as photographing equipment, smart phones, computers, servers, etc. that can perform sidewalk disease detection.

[0097] like Figure 2 As shown, an embodiment of the present invention provides a sidewalk disease detection device, the sidewalk disease detection device comprising:

[0098] An acquisition module 201 is used to acquire monitoring data of the sidewalk;

[0099] A processing module 202 is used to determine the support point of each pedestrian's foot on the sidewalk according to the monitoring data;

[0100] The determination module 203 is used to determine whether the sidewalk pavement has any damage according to the support points.

[0101] Optionally, the processing module 202 includes:

[0102] A processing submodule, used for performing posture recognition processing on the monitoring data to obtain the foot posture of each of the pedestrians' feet on the sidewalk;

[0103] The first determination submodule is used to determine the support point of each pedestrian's foot on the sidewalk according to the foot posture.

[0104] Optionally, the first determining submodule includes:

[0105] A first extraction unit, configured to extract a first characteristic vector corresponding to the lowest point of the first foot when the foot posture is landing, and determine a first support point according to the first characteristic vector;

[0106] The second extraction unit is used to extract the second characteristic vector corresponding to the lowest point of the second foot when the foot posture is lifting the foot, and determine the second support point according to the second characteristic vector.

[0107] Optionally, the determining module 203 includes:

[0108] A second determining submodule, configured to determine a distribution type of the supporting points according to the positions of the supporting points;

[0109] The third determination submodule is used to determine whether the sidewalk pavement has any damage according to the distribution type of the support points.

[0110] Optionally, the third determination submodule includes:

[0111] A judging unit, used to judge whether the distribution type of the supporting points is a plane type;

[0112] A first determination unit is used to determine that the sidewalk pavement has no damage if the distribution type of the support points is a plane type;

[0113] The second determining unit is used to determine that the sidewalk pavement has a disease if the distribution type of the support points is a non-planar type.

[0114] Optionally, the second determining unit includes:

[0115] A first determination subunit is used to determine that the sidewalk pavement has a sinking disease if the distribution type of the support points is a sinking curved surface type;

[0116] The second determining subunit is used to determine that the sidewalk pavement has an arching defect if the distribution type of the support points is an arched curved surface type.

[0117] Optionally, the determination module 203 includes:

[0118] A fourth determination submodule, configured to determine a support point cloud of the sidewalk pavement according to the support points;

[0119] A fifth determination submodule is used to determine the scan point cloud of the sidewalk pavement when no one is walking on it according to historical monitoring data;

[0120] The sixth determination submodule is used to determine whether the sidewalk pavement has any damage based on the support point cloud and the scanning point cloud.

[0121] It should be noted that the sidewalk disease detection device provided in the embodiment of the present invention can be applied to devices such as photographing equipment, smart phones, computers, servers, etc. that can perform sidewalk disease detection.

[0122] The sidewalk disease detection device provided in the embodiment of the present invention can implement each process implemented by the sidewalk disease detection method in the above method embodiment, and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0123] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, it includes: a memory 302, a processor 301, and a computer program of a sidewalk disease detection method stored in the memory 302 and executable on the processor 301, wherein:

[0124] Used to obtain monitoring data of sidewalks;

[0125] Used to determine the support point of each pedestrian's foot on the sidewalk according to monitoring data;

[0126] Used to determine whether the sidewalk pavement is damaged based on the support points.

[0127] Optionally, the processor 301 determines the support point of each pedestrian's foot on the sidewalk according to the monitoring data, including:

[0128] Performing posture recognition processing on the monitoring data to obtain the foot posture of each of the pedestrians' feet on the sidewalk;

[0129] According to the foot posture, the support point of each pedestrian's foot on the sidewalk is determined.

[0130] Optionally, the processor 301 determines the support point of each pedestrian's foot on the sidewalk according to the foot posture, including:

[0131] When the foot posture is landing, extracting a first characteristic vector corresponding to the lowest point of the first foot, and determining a first support point according to the first characteristic vector;

[0132] When the foot posture is lifting the foot, a second characteristic vector corresponding to the lowest point of the second foot is extracted, and a second support point is determined according to the second characteristic vector.

[0133] Optionally, the processor 301 determines whether the sidewalk pavement is damaged according to the support point, including:

[0134] Determining the distribution type of the support points according to the positions of the support points;

[0135] Whether the sidewalk pavement is damaged is determined according to the distribution type of the support points.

[0136] Optionally, the processor 301 determines whether the sidewalk pavement is damaged according to the distribution type of the support points, including:

[0137] Determining whether the distribution type of the support points is a plane type;

[0138] If the distribution type of the support points is a plane type, it is determined that the sidewalk pavement has no disease;

[0139] If the distribution type of the support points is a non-planar type, it is determined that the sidewalk pavement has a defect.

[0140] Optionally, if the distribution type of the support points is a non-planar type, determining that the sidewalk pavement has a disease, performed by the processor 301, includes:

[0141] If the distribution type of the support points is a sinking curved surface type, it is determined that the sidewalk pavement has a sinking disease;

[0142] If the distribution type of the support points is a bulging curved surface type, it is determined that the sidewalk pavement has bulging damage.

[0143] Optionally, the processor 301 determines whether the sidewalk pavement is damaged according to the support point, including:

[0144] Determine a support point cloud of the sidewalk pavement according to the support points;

[0145] Determine, based on historical monitoring data, a scan point cloud of the sidewalk pavement when no one is walking on it;

[0146] Based on the support point cloud and the scanning point cloud, determine whether the sidewalk pavement has any damage.

[0147] It should be noted that the electronic device provided in the embodiment of the present invention can be applied to smart phones, computers, servers and other devices that can perform the sidewalk disease detection method.

[0148] The electronic device provided in the embodiment of the present invention can implement each process implemented by the sidewalk disease detection method in the above method embodiment, and can achieve the same beneficial effect. To avoid repetition, it will not be described here.

[0149] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the various processes of the sidewalk disease detection method or the application-end sidewalk disease detection method provided in the embodiment of the present invention, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0150] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0151] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for detecting sidewalk defects, characterized in that: The method comprises the following steps: Acquire monitoring data of a sidewalk, wherein the monitoring data includes the sidewalk pavement and the feet of pedestrians; Performing posture recognition processing on the monitoring data to obtain the foot posture of each of the pedestrians' feet on the sidewalk; According to the foot posture, the support point of each pedestrian's foot on the sidewalk is determined; when the foot posture is landing, the first feature vector corresponding to the lowest point of the first foot is extracted, and the first support point is determined according to the first feature vector; when the foot posture is starting, the second feature vector corresponding to the lowest point of the second foot is extracted, and the second support point is determined according to the second feature vector; when the foot posture is landing, the lowest point of the foot of each frame image in the image sequence is detected by a foot key point detection model to obtain the first lowest point of the foot corresponding to each pedestrian's foot, each pedestrian's foot corresponds to a first lowest point of the foot when landing, and the vector from the lowest point of the first foot in the nth frame image to the lowest point of the first foot in the n+1th frame image is determined as the first feature vector corresponding to the lowest point of the first foot, and the first feature vector includes The first feature vector includes direction and length; the modulus length of the first feature vector is calculated. When the modulus length of the first feature vector is equal to 0, the lowest point of the first foot in the n+1th frame image is determined as the first support point; when the foot posture is the starting foot, the lowest point of the foot of each frame image in the image sequence is detected by the foot key point detection model to obtain the second lowest point of the foot corresponding to each pedestrian's foot. Each pedestrian's foot corresponds to a second lowest point of the foot when landing. The vector from the lowest point of the second foot in the mth frame image to the lowest point of the second foot in the m+1th frame image is determined as the second feature vector corresponding to the second lowest point of the foot, and the second feature vector includes direction and length; the modulus length of the second feature vector is calculated. When the modulus length of the second feature vector changes from 0 to greater than 0, the lowest point of the second foot in the mth frame image is determined as the second support point; Determine whether the sidewalk pavement has any defects based on the support points; determine whether the distribution of the support points conforms to the plane distribution. If the distribution of the support points conforms to the plane distribution, it is determined that there is no defect at the sidewalk pavement location; if not, it is determined that the sidewalk pavement has any defects.

2. The sidewalk disease detection method according to claim 1, characterized in that: The determining whether the sidewalk pavement is damaged according to the supporting point includes: Determining the distribution type of the support points according to the positions of the support points; Whether the sidewalk pavement is damaged is determined according to the distribution type of the support points.

3. The sidewalk disease detection method according to claim 2, characterized in that: The determining whether the sidewalk pavement is damaged according to the distribution type of the support points includes: Determining whether the distribution type of the support points is a plane type; If the distribution type of the support points is a plane type, it is determined that the sidewalk pavement has no disease; If the distribution type of the support points is a non-planar type, it is determined that the sidewalk pavement has a defect.

4. The sidewalk disease detection method according to claim 3, characterized in that: If the distribution type of the support points is a non-planar type, determining that the sidewalk pavement has a disease includes: If the distribution type of the support points is a sinking curved surface type, it is determined that the sidewalk pavement has a sinking disease; If the distribution type of the support points is a bulging curved surface type, it is determined that the sidewalk pavement has bulging damage.

5. The sidewalk disease detection method according to claim 1, characterized in that: The determining whether the sidewalk pavement is damaged according to the supporting point includes: Determine a support point cloud of the sidewalk pavement according to the support points; Determine, based on historical monitoring data, a scan point cloud of the sidewalk pavement when no one is walking on it; Based on the support point cloud and the scanning point cloud, determine whether the sidewalk pavement has any damage.

6. A sidewalk disease detection device, characterized in that: The sidewalk disease detection device comprises: An acquisition module, used to acquire monitoring data of the sidewalk, wherein the monitoring data includes the sidewalk pavement and pedestrians' feet; A processing module is used to perform posture recognition processing on the monitoring data to obtain the foot posture of each pedestrian's foot on the sidewalk; determine the support point of each pedestrian's foot on the sidewalk according to the foot posture; when the foot posture is landing, extract the first feature vector corresponding to the lowest point of the first foot, and determine the first support point according to the first feature vector; when the foot posture is starting, extract the second feature vector corresponding to the lowest point of the second foot, and determine the second support point according to the second feature vector; when the foot posture is landing, detect the lowest point of the foot of each frame image in the image sequence through the foot key point detection model to obtain the first lowest point of the foot corresponding to each pedestrian's foot, each pedestrian's foot corresponds to a first lowest point of the foot when landing, and determine the vector from the first lowest point of the foot in the nth frame image to the first lowest point of the foot in the n+1th frame image as the first A first feature vector corresponding to the lowest point of the foot, wherein the first feature vector includes a direction and a length; the modulus of the first feature vector is calculated, and when the modulus of the first feature vector is equal to 0, the lowest point of the first foot in the n+1th frame image is determined as the first support point; when the foot posture is a kick-off posture, the lowest point of the foot of each frame image in the image sequence is detected by a foot key point detection model to obtain a second lowest point of the foot corresponding to each pedestrian's foot, and each pedestrian's foot corresponds to a second lowest point of the foot when landing, and a vector from the lowest point of the second foot in the mth frame image to the lowest point of the second foot in the m+1th frame image is determined as the second feature vector corresponding to the second lowest point of the foot, wherein the second feature vector includes a direction and a length; the modulus of the second feature vector is calculated, and when the modulus of the second feature vector changes from 0 to greater than 0, the lowest point of the second foot in the mth frame image is determined as the second support point; A determination module is used to determine whether the sidewalk pavement has any defects based on the support points; by judging whether the distribution of the support points conforms to the plane distribution, if the distribution of the support points conforms to the plane distribution, it is judged that there is no defect at the sidewalk pavement position; if not, it is determined that the sidewalk pavement has any defects.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the sidewalk disease detection method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the sidewalk disease detection method according to any one of claims 1 to 5 are implemented.

Images