Method for checking a tactile paving problem

Abstract

A blind path problem checking method, comprising: an information collection end, a blind path detection processing system and a detection result output end; characterized in that: the information collection end comprises an information collection device, the image collected by the information collection device contains geographic position coordinate information, and the obtained information is transmitted to the blind path detection processing system; the blind path detection processing system detects the corresponding image, judges and classifies the blind path condition in the image, checks out the problem blind path, integrates the obtained blind path construction and use condition classification result and the geographic position coordinate information of the corresponding image to generate a blind path construction and use condition feedback report, and outputs the report through the detection result output end. The method can judge and classify the blind path condition in the image, check out the blind path problem, and output the blind path construction and use condition feedback report through the detection result output end; assist the municipal management department and the blind path rectification construction unit to grasp the overall construction situation of the regional blind path, and provide technical support for targeted rectification. Through intelligent checking, identification and classification, the professional threshold of the staff can be reduced.

Description

Technical Field

[0001] This invention relates to a method for inspecting problems with tactile paving. Background Technology

[0002] Currently, there are over 17 million visually impaired people in China, and tactile paving systems are a crucial component of barrier-free infrastructure development, serving as an essential means of transportation for them. However, many urban tactile paving systems suffer from problems such as missing paving stones, improper installation, and obstruction by obstacles. The absence, substandard installation, or obstruction of parts of the tactile paving prevents visually impaired individuals from effectively using the entire path, thus affecting their normal travel. Furthermore, tactile paving construction is an indispensable part of municipal road construction; constructing substandard or improperly used tactile paving systems represents a significant waste of human, material, and financial resources for municipal engineering projects.

[0003] Currently, there are some devices to assist blind people in traveling. For example, Chinese Patent CN113029154A discloses a navigation method and device for the blind. In this technical solution, convolutional neural networks and long short-term memory neural networks are used to detect obstacles, and graph neural networks are used to predict the direction of walking of the user. The outputs of the two are combined to provide prompts for obstacles in front of the blind person during walking. However, there are still the following shortcomings: (1) It is limited to the detection of obstacles on the blind path and cannot identify and record the problems of non-standard blind path laying and missing blind path bricks; (2) It only realizes the detection of objects and does not propose identification and classification methods for different blind path problems, nor can it efficiently sort out the blind path situation and generate a feedback report on the construction and use of blind paths in the area; (3) It only provides obstacle reminders and navigation for the blind person and cannot provide technical support for the systematic rectification of municipal blind paths, and cannot fundamentally solve the pain points of "low construction qualification rate and timely generation of reports when problems are found during use".

[0004] Currently, some cities and regions have begun to conduct a comprehensive investigation of tactile paving issues, but relying on accessibility professionals to conduct on-site inspections, record data, and provide feedback results in low work efficiency and high professional thresholds. To improve the efficiency of tactile paving issue investigation, some systems using computer technology to assist in the collection of tactile paving issues have been developed. For example, Chinese Patent CN112345001A discloses a method, device, and storage medium for collecting and storing basic data on accessibility facilities. In this technical solution, the location, classification, and size information of the facilities are collected according to the assigned task information, and then relevant markings are made on the map based on the collected information, solving the problem of low efficiency in manual on-site collection. However, the following shortcomings still exist: (1) It is limited to the collection of objective data on accessibility facilities and does not intelligently identify and classify whether there are problems; (2) It still requires manual judgment of the collected facility information, and accessibility professionals need to manually write reports on the status and problems of accessibility facilities, resulting in low identification efficiency and high professional threshold requirements for staff.

[0005] For example, Chinese Patent CN111860468A discloses a method, system, device and storage medium for identifying tactile paving based on machine vision. In this technical solution, by establishing a database of tactile paving bricks, and combining the tactile paving brick patterns and geographical coordinates collected in real time, relatively complete corresponding tactile paving information is obtained; on this basis, machine vision technology is used to compare and detect the images with the tactile paving images in the database, obtain the geographical coordinates of the tactile paving in the images, and broadcast the tactile paving information. The computer has realized intelligent identification of tactile paving types, but there are still the following shortcomings: (1) Professional personnel are still required to collect and upload tactile paving information on-site, and the establishment of the tactile paving database also requires manual identification of the collected tactile paving patterns and information; (2) Only the tactile paving itself is classified, without in-depth exploration of the combination features of problematic tactile paving to classify the problems existing in the tactile paving; (3) The comparison and detection of tactile paving images is only used to determine the geographical location of the tactile paving, and does not involve the identification and classification of tactile paving problems based on machine vision. Summary of the Invention

[0006] The purpose of this invention is to provide a method for inspecting tactile paving problems based on deep learning technology, enabling machines to replace manual labor in the intelligent investigation, identification, and classification of tactile paving problems, and automatically generating feedback reports on the construction and usage of the inspected tactile paving.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution: A method for inspecting tactile paving problems includes three parts: an information acquisition terminal, a tactile paving detection and processing system, and a detection result output terminal. The information acquisition terminal includes an information acquisition device that acquires images containing geographic location coordinate information. This information is transmitted to the tactile paving detection and processing system, which detects the corresponding images, judges and classifies the tactile paving conditions in the images, identifies problematic tactile paving, and integrates the obtained tactile paving construction and usage status classification results with the corresponding geographic location coordinate information to generate a tactile paving construction and usage feedback report, which is then output through the detection result output terminal.

[0008] The advantages of this invention are as follows: The tactile paving detection and processing system detects and classifies the tactile paving conditions in corresponding images, identifies tactile paving problems, and outputs a feedback report on the construction and use of tactile paving through the output terminal. This assists municipal management departments and tactile paving renovation construction units in understanding the overall construction status of tactile paving in a region, providing technical support for targeted rectification. Furthermore, this method uses machines to replace manual labor for intelligent inspection, identification, and classification of problematic tactile paving, reducing the professional threshold for staff.

[0009] In this invention, the geographic location coordinates obtained by the information collection terminal are matched with an electronic map to determine the actual location of the corresponding collected image on the electronic map. This allows for the intuitive presentation of road information for problematic tactile paving in the feedback report on the construction and use of tactile paving.

[0010] In this invention, the information collection device is a device capable of timed shooting, real-time positioning, and recording of shooting time. In use, the automatic shooting interval is first set, then the information collection device is installed on a carrier. As the carrier travels along a tactile paving section, the information collection device automatically takes pictures of the tactile paving, simultaneously acquiring the geographic coordinates and shooting time of each photo. The geographic coordinates and shooting time are then bound to the images, and the bound information is transmitted to the tactile paving detection and processing system. This allows machines to replace manual shooting and recording of tactile paving information, improving information collection efficiency. The information collection device can be installed on one of the following: a mobile phone, a white cane, a motor vehicle, a non-motor vehicle, a surveillance camera, a drone, or an unmanned vehicle. The aforementioned carrier carries the collection device and travels along a specific path, allowing the collection device to shoot and record surrounding tactile paving information.

[0011] In this invention, the tactile paving detection and processing system includes a tactile paving problem judgment module, employing one of the following: graph neural network, logical judgment, SVM, KNN, or CNN. It can detect tactile paving and related elements in acquired images, and based on the detection, uses a graph neural network to model the relationships between elements in the image, thereby expressing the relative positional relationships as an abstract graph structure. Then, through inference learning on the graph structure, the computer can intelligently identify and classify the condition of tactile paving in the image.

[0012] In this invention, the tactile paving problem judgment module includes an image input section, an element detection section, a direction information extraction section, a graph structure transformation section, a tactile paving node label prediction section based on graph neural networks, a problem classification and output section, and a feedback report generation section. The image is input from the information collection terminal. First, image processing algorithms are used to detect elements in the image and extract the tactile paving travel direction information. Based on the element detection results, the relationships between elements in the image are modeled, converting the real-world image into an abstract graph structure. Then, based on graph neural network algorithms, the abstract graph structure is inferred and learned to obtain the probability vector of each type of tactile paving condition in the image. The obtained probability vectors are mapped one-to-one with the names of tactile paving conditions; the tactile paving condition name corresponding to the component with the highest probability value is the detection result, thereby enabling the computer to intelligently identify and classify tactile paving conditions in the image. Finally, the tactile paving condition classification results are integrated with the information obtained from the information collection terminal to generate a tactile paving construction and usage feedback report file, allowing users to obtain a more intuitive and comprehensive understanding of the tactile paving construction and usage status in the investigated area.

[0013] In this invention, the image input section inputs the image acquired by the information acquisition terminal into the tactile paving detection and processing system. This image is then used for subsequent detection of tactile paving and related elements, identification of tactile paving conditions, and classification.

[0014] In this invention, the element detection part uses image processing methods to detect elements in an image, including the type of the element, the coordinates of the center point of the detection box, and the length and width information of the detection box.

[0015] In this invention, the objects of element detection include one or all of the following: tactile paving for pedestrians, tactile paving with correct end prompts, tactile paving with incorrect end prompts, tactile paving at correct intersections, tactile paving at incorrect intersections, non-motorized vehicles, motorized vehicles, manhole covers, and cylindrical objects. The detected elements are common elements involved in problematic tactile paving during construction and use. This allows the conversion of real-world images into abstract graph structures to retain and highlight key element information, thereby enabling precise modeling of the relationships between elements in the image based on the element detection results.

[0016] In this invention, the image processing methods include one or a combination of Faster R-CNN, YOLO, color clustering analysis, Radon transform, Gaussian mixture model, Hough transform, H hue thresholding segmentation, Sobel edge detection, Canny edge detection, Lab color model, fuzzy C-means (KFCM) algorithm, single-channel color space thresholding segmentation, K-Means, and FCM segmentation algorithms. Based on traditional image processing methods or deep learning image processing methods, the detection of elements in an image is achieved, including the element type, the coordinates of the center point of the detection box, and the length and width information of the detection box.

[0017] In this invention, the element detection part is selected after the element detection in the image is implemented, and the target detection box is precisely adjusted according to the outer contour of the object by the SOLOv2 algorithm to accurately reflect the position and size of the element in the image.

[0018] In this invention, the direction information extraction part utilizes the periodicity of the tactile paving texture arrangement and its prior knowledge of the tactile paving direction to calculate the correlation through the calculated gray-level co-occurrence matrix. The similarity of the gray levels of the image texture in different directions is measured; a larger value indicates a stronger correlation between the tactile paving texture in that direction, meaning the closer the texture recurrences, the closer it is to the tactile paving direction. This direction information is then used for secondary determination of edges in the subsequent abstract graph structure, thereby achieving accurate modeling of the relative position of obstacles and tactile paving, used to determine whether obstacles occupy the tactile paving direction or are adjacent to either side of the tactile paving.

[0019] In this invention, the graph structure conversion part models the relationships between elements in the image based on the element detection results obtained by the element detection part, converting the real-world scene image into an abstract graph structure composed of points and edges. Nodes are generated from the detected elements and their attributes, with each node's initial feature being a one-hot encoding of its node type. Then, the corresponding Interchange of Units (IOU) index is calculated using the different positional relationships between objects, thus deriving two types of edges: "overlapping" and "proximity." After obtaining the preliminary abstract graph structure, the tactile paving direction information is input into the secondary judgment of the edges in the abstract graph structure. Specifically, "proximity" edges connecting tactile paving and non-tactile paving element nodes, where the non-tactile paving node is not in the direction of travel along the tactile paving, are deleted, resulting in the final abstract graph structure.

[0020] In this invention, the "overlapping" edge type refers to the representation of the relationship between elements when there is an overlap between the detection boxes of different detected elements.

[0021] In this invention, the "adjacent" edge type refers to the relationship between elements when there is no overlap between the detection boxes of different detected elements, but there is overlap after the detection boxes are expanded to 105% of their original size.

[0022] In this invention, the tactile paving node label prediction part based on graph neural network performs inference learning on abstract graph structure based on graph neural network, and realizes the computer's intelligent recognition and classification of tactile paving conditions in images through node label prediction.

[0023] In this invention, the node label prediction employs a heterogeneous graph structure and applies a multi-head attention graph convolutional network (GATS). Different DGLNN modules are used to process each type of relationship according to different edge types, allowing information from the source node to be transmitted to the target node along different relationships. Then, for the same target node, information from different relationships is aggregated for feature updating. After updating the node information, since tactile paving condition classification often relies only on neighboring nodes, and the graph structure abstracted from each image is relatively simple, a single-layer attention graph convolutional network is adopted. After obtaining normalized attention coefficients through the single-layer attention graph convolutional network, the linear combination of corresponding features is calculated using these coefficients as the final output feature of each node. The node-level output based on the graph neural network is extracted and fed into a fully connected neural network to transform the final tactile paving features into the tactile paving condition feature space for classification, thereby obtaining the probability vectors of each type of tactile paving condition in the image.

[0024] In this invention, the problem classification and output section maps the probability vector of tactile paving problems to the names of tactile paving conditions one-to-one, and outputs the name of the tactile paving condition corresponding to the component with the highest probability value as the classification result.

[0025] In this invention, the names of tactile paving conditions include one or all of the following: correct tactile paving, tactile paving occupied by non-motorized vehicles, tactile paving occupied by motorized vehicles, tactile paving occupied by manhole covers, tactile paving occupied by cylindrical objects, incorrect tactile paving at intersections, and incorrect tactile paving at the ends. The categorized tactile paving conditions are all common and readily apparent, allowing users to more intuitively understand the construction and usage problems of the tactile paving. By combining the geographical location and road information of the problematic tactile paving, the underlying causes of the problem can be analyzed, thereby enabling macro-level control of the tactile paving condition and assisting in deriving a systematic renovation strategy for tactile paving.

[0026] In this invention, the term "correct tactile paving" refers to a tactile paving situation where the type of tactile paving bricks and the laying sequence comply with the relevant specifications for tactile paving in the "GB50763-2012 Barrier-Free Design Code," and there are no obstacles in the direction of travel along the tactile paving. The term "non-motorized vehicle occupying tactile paving" refers to a tactile paving situation where a non-motorized vehicle is occupying the direction of travel along the tactile paving. The term "motorized vehicle occupying tactile paving" refers to a tactile paving situation where a motorized vehicle is occupying the direction of travel along the tactile paving. The term "manhole cover occupying tactile paving" refers to a tactile paving situation where a manhole cover is occupying the direction of travel along the tactile paving. The term "pillar-shaped object occupying tactile paving" refers to a tactile paving situation where a pillar-shaped object is occupying the direction of travel along the tactile paving, including one of the following: utility pole, bollard, or tree trunk. The term "incorrect tactile paving at intersections" refers to a situation where the tactile paving at intersections in the image does not comply with the relevant specifications for tactile paving at intersections as outlined in GB50763-2012 "Accessibility Design Standard" and 12J926 "Accessibility Design Atlas". The term "incorrect tactile paving at the ends" refers to a situation where the tactile paving at the ends (including starting and ending points) in the image does not comply with the relevant specifications for end tactile paving as outlined in GB50763-2012 "Accessibility Design Standard" and 12J926 "Accessibility Design Atlas".

[0027] In this invention, the problem classification and output section, if it is determined that the obtained tactile paving condition is not "correct tactile paving", classifies it into two levels, "serious" and "minor", based on the absolute magnitude of the maximum probability value in the tactile paving problem probability vector. This reflects the severity of the tactile paving problem involved in the image and achieves a more accurate description of the problematic tactile paving in construction and use.

[0028] In this invention, the "severe" level refers to tactile paving nodes whose absolute value of the maximum probability in the probability vector of tactile paving problems is greater than 0.75.

[0029] In this invention, the term "minor" refers to tactile paving nodes where the absolute value of the maximum probability in the probability vector of tactile paving problems is less than or equal to 0.75.

[0030] In this invention, the report file generation section integrates the tactile paving condition classification results and the information obtained by the information collection terminal, and matches the obtained geographical location information with the electronic map to obtain the actual location of the corresponding collected image in the electronic map, and generates a tactile paving construction and usage feedback report file which is stored locally in the system or uploaded to a network platform.

[0031] In this invention, the feedback report on the construction and use of tactile paving includes the number of tactile paving conditions of each type during the current tactile paving information collection, the classification results of tactile paving conditions in each photo, the severity, the shooting time, and the actual geographical location information of the image, and displays the location of each tactile paving condition on an electronic map, so as to achieve comprehensive and intuitive feedback on the tactile paving conditions in the investigated area.

[0032] In this invention, the output terminal of the detection result outputs the classification result of the blind path condition in the image and the feedback report on the construction and use of the blind path.

[0033] In this invention, the output device for the detection results includes one of a handheld printer, a remote wireless printer, or an electronic screen. This facilitates the presentation of the final tactile paving condition classification results and feedback reports on the construction and use of tactile paving. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the device structure of the present invention. Figure 2 This is a schematic diagram of the device in Embodiment 1 of the present invention. Figure 3 This is a schematic diagram of the tactile paving detection and processing system of the present invention. Figure 4 This is a schematic diagram of the element detection section of Embodiment 1 of the present invention. Figure 5 This is a schematic diagram of the diagram structure transformation part of Embodiment 1 of the present invention. Figure 6 This is a schematic diagram of the tactile paving node label prediction part based on graph neural networks in Embodiment 1 of the present invention. Figure 7 This is a schematic diagram of the blind path problem judgment process based on graph neural network in Embodiment 1 of the present invention. Figure 8 This is a schematic diagram of the problem classification and output section of Embodiment 1 of the present invention. Figure 9 This is a schematic diagram of the problem classification and output process in Embodiment 1 of the present invention. Figure 10 This is a schematic diagram of the report file generation part of Embodiment 1 of the present invention. Figure 11 This is a schematic diagram of the element detection part in Embodiment 3 of the present invention. Detailed Implementation Specific Implementation Example 1 Figures 1-10 Example 1 of the present invention is given.

[0036] like Figure 1As shown: The tactile paving problem inspection method in this embodiment includes an information acquisition terminal 1, a tactile paving detection and processing system 2, and a detection result output terminal 3. The information acquisition terminal includes an information acquisition device. The images acquired by the information acquisition device contain geographic location coordinate information, and the acquired information is transmitted to the tactile paving detection and processing system. The tactile paving detection and processing system detects the corresponding images, judges and classifies the tactile paving conditions in the images, identifies problematic tactile paving, and integrates the obtained tactile paving construction and usage status classification results with the corresponding geographic location coordinate information of the images to generate a tactile paving construction and usage status feedback report, which is output through the detection result output terminal.

[0037] In this embodiment, the geographic location coordinates obtained by the information collection terminal are matched with the electronic map to determine the actual location of the corresponding collected image on the electronic map, which makes it easier to intuitively provide road information of problematic blind paths in the feedback report on the construction and use of blind paths.

[0038] like Figure 2 As shown: The information collection terminal includes an information collection device. In this embodiment, the information collection device includes a pan-tilt unit 4 and a digital camera 5, which can be mounted on the handlebars of a non-motorized vehicle. The information collection terminal can be wirelessly connected to the network and has functions such as timed shutter activation, real-time positioning, and recording of shooting time. The real-time positioning function of the information collection terminal is activated to obtain GPS signals and the geographical coordinates of each photo; the shooting time recording function is activated to record the shooting time of each photo; the timed shutter activation function is activated, setting the automatic shooting interval to 1 second / photo. The information collection terminal is installed on the handlebars of a non-motorized vehicle. Users only need to ride on blind paths, and the information collection device will automatically take photos of the blind path, simultaneously obtaining the geographical coordinates and shooting time of each photo; the geographical coordinates and shooting time are bound to the image, and the bound information is uploaded to the blind path detection and processing system 3.

[0039] like Figure 3 As shown, the tactile paving detection and processing system includes a tactile paving problem judgment module 6. In this embodiment, the tactile paving problem judgment module includes: an image input section 6-1, an element detection section 6-2, a direction information extraction section 6-3, a graph structure transformation section 6-4, a tactile paving node label prediction section based on graph neural networks 6-5, a problem classification and output section 6-6, and a report file generation section 6-7. Using a graph neural network algorithm, based on image detection, a graph neural network is used to model the relationships between elements in the image, thereby expressing the relative positional relationships as an abstract graph structure. Then, through inference learning on the graph structure, the computer can intelligently identify and classify the tactile paving conditions in the image.

[0040] In this embodiment, the image input section 6-1 inputs the image acquired by the information acquisition terminal into the tactile paving detection and processing system for subsequent tactile paving and related element detection, tactile paving condition identification and classification.

[0041] Combination Figure 3 , Figure 4 As shown: In this embodiment, the element detection part 6-2 includes: image input 7, an element detection module based on Faster R-CNN 8, and detection result output 9. The Faster R-CNN-based element detection module integrates feature extraction, target extraction, candidate boxes, and a classifier into a single network. It replaces the shared convolutional layers, activation functions, and pooling layers in the original Faster R-CNN model with FPN and a residual network, thereby achieving the ability to extract image features at different scales and avoiding gradient vanishing and gradient exploding. Feature maps are obtained through FPN and the residual network, and their feature information is simultaneously transmitted to the ROI pooling layer and the RPN layer to infer the position of candidate regions, obtaining preliminary detection results. The target classification layer uses the output candidate regions to calculate the category of each candidate region through fully connected layers, outputting a probability vector predicting it as an element of a certain class. Simultaneously, through candidate region adjustment, bounding box regression is used to obtain the positional offset of each selected region. Regression calculations are used to obtain a more accurate size and position of the target detection box, resulting in the element detection result.

[0042] In this embodiment, the objects of element detection include tactile paving for pedestrians, tactile paving with correct end prompts, tactile paving with incorrect end prompts, tactile paving at correct intersections, tactile paving at incorrect intersections, non-motorized vehicles, motorized vehicles, manhole covers, and cylindrical objects. The detected elements are common elements involved in problematic tactile paving during construction and use, allowing the conversion of the real-world image into an abstract graph structure to retain and highlight key element information. This, combined with the element detection results, enables accurate modeling of the relationships between elements in the image.

[0043] In this embodiment, the direction information extraction part 6-3 includes: utilizing the periodicity of the arrangement of the tactile paving texture and the prior knowledge of the texture's direction of travel on the tactile paving, the correlation is calculated using the calculated gray-level co-occurrence matrix, which measures the similarity of the gray levels of the image texture in different directions. The larger the value, the stronger the correlation of the tactile paving texture in that direction, i.e., the closer it is to the repeated occurrence of the texture, and the closer it is to the direction of travel on the tactile paving. The obtained tactile paving direction information is used for the secondary determination of edges in the abstract graph structure of the graph structure transformation part 6-4, i.e., deleting the "neighboring" edges that connect tactile paving and non-tactile paving element nodes respectively, and where the non-tactile paving node is not in the direction of travel on the tactile paving, thereby achieving accurate modeling of the relative position of the obstacle and the tactile paving. In the tactile paving node label prediction part 6-5 based on the graph neural network, it is used to determine whether the obstacle occupies the direction of travel on the tactile paving or is adjacent to both sides of the tactile paving.

[0044] like Figure 5As shown: In this embodiment, the graph structure transformation part 6-4 includes: detection result output 9, IOU index calculation 10, preliminary abstract graph structure 11, and final abstract graph structure 12. Based on the element detection results obtained by the element detection module, a graph neural network is used to model the relationships between elements in the image, converting the real-world scene image into an abstract graph structure composed of points and edges. Specifically, nodes are generated from the detected elements and their attributes, with each node's initial feature being a one-hot encoding of the node type. Then, the corresponding IOU index is calculated using the different positional relationships between objects, thus deriving two types of edges: "overlapping" and "proximity". When there is overlap between the detection boxes of different detected elements, the relationship between elements is represented as an "overlapping" edge. When there is no overlap between the detection boxes of different detected elements, but overlap occurs after the detection boxes are expanded to 105% of their original size, the relationship between elements is represented as a "proximity" edge. This yields a preliminary abstract graph structure. The directional information of the tactile paving is then input into the secondary judgment of the edges in the abstract graph structure. This involves deleting the "neighboring" edges that connect the tactile paving and non-tactile paving element nodes, and where the non-tactile paving node is not in the direction of travel of the tactile paving, thus obtaining the final abstract graph structure.

[0045] Combination Figure 6 , Figure 7 As shown: In this embodiment, the tactile paving node label prediction part 4-5 based on graph neural network includes: final abstract graph structure 12, multi-head attention graph convolutional network 13, node information update 14, single-layer attention graph convolutional network 15, node final features 16, fully connected neural network 17, and tactile paving condition probability vector output 18. The abstract graph structure is constructed using a heterogeneous graph structure, that is, each type of element in the element detection part 6-2 is used as a type of node, so that there are multiple different types of nodes in the abstract graph. After the abstract graph structure is input, the multi-head attention graph convolutional network 13 is used for inference learning: according to different types of edges, different DGLNN modules are used to process each relationship, that is, the information of the source node is passed to the target node along different relationships; then for the same target node, the information from different relationships is aggregated to update the node features 14. After the node information update is completed, since the tactile paving condition classification often only depends on neighboring nodes and the graph structure abstracted from each image is relatively simple, a single-layer attention graph convolutional network 15 is adopted. After obtaining normalized attention coefficients through a single-layer attention map convolutional network, linear combinations of corresponding features are calculated using these coefficients, serving as the final features for each node.16 The node-level outputs based on the graph neural network are extracted and fed into a fully connected neural network.17 The tactile paving features are transformed into a tactile paving condition feature space for classification, resulting in tactile paving scene outputs as probability vectors for each type of tactile paving condition in the image.18 This achieves inference learning based on the graph neural network on the abstract graph structure, enabling intelligent recognition and classification of tactile paving problems in images by the computer through node label prediction.

[0046] Combination Figure 8 , Figure 9 As shown, in this embodiment, the problem classification and output section 6-6 includes: a probability vector output of tactile paving conditions 18, a component with the highest probability value 19, a classification result output of tactile paving conditions 20, and a severity output of tactile paving problems 21. The probability vector of tactile paving conditions is mapped one-to-one with the names of tactile paving conditions. The name of the tactile paving condition corresponding to the component with the highest probability value is output as the classification result of the tactile paving conditions. It is determined whether the tactile paving condition corresponding to the component with the highest probability value is a tactile paving problem. If so, the severity of the tactile paving problem involved in the image is determined based on the absolute value of the highest probability value, divided into two levels: "severe" and "mild." When the absolute value is greater than 0.75, it is "severe," and when it is less than or equal to 0.75, it is "mild." For example, if the tactile paving condition corresponding to the first component is "non-motorized vehicles occupying the tactile paving," and if the first component in the output tactile paving problem probability vector has the highest probability value and an absolute value of 0.85, then the problem classification and output result is "severe" level "non-motorized vehicles occupying the tactile paving."

[0047] In this embodiment, the tactile paving conditions classified include: correct tactile paving, tactile paving occupied by non-motorized vehicles, tactile paving occupied by motorized vehicles, tactile paving occupied by manhole covers, tactile paving occupied by cylindrical objects, incorrect tactile paving at intersections, and incorrect tactile paving at the ends. These categorized tactile paving conditions are all common and readily apparent, allowing users to more intuitively understand the construction and usage problems of the tactile paving. By combining the geographical location and road information of the problematic tactile paving, the underlying causes of the problem can be analyzed, thereby enabling macro-level control of the tactile paving condition and assisting in deriving a systematic renovation strategy for tactile paving.

[0048] like Figure 10 As shown: In this embodiment, the report file generation section 6-7 includes: detection results and information input 22, information integration 23, and report file generation 24. The tactile paving condition classification results and the information obtained from the information collection terminal are integrated, and the obtained geographical location information is matched with the electronic map to determine the actual location of the corresponding collected image on the electronic map. A feedback report file on the construction and use of tactile paving is then generated and stored locally on the system or uploaded to a network platform.

[0049] In this embodiment, the feedback report on the construction and use of tactile paving includes the number of tactile paving conditions of each type during this tactile paving information collection, the classification results of tactile paving conditions in each photo, the severity, the shooting time, and the actual geographical location information of the image. The location of each tactile paving condition is displayed on an electronic map, achieving comprehensive and intuitive feedback on the tactile paving conditions in the investigated area.

[0050] like Figure 2As shown: In this embodiment, the output of the detection results includes: a handheld printer 25 and a feedback report on the construction and use of tactile paving 26. The device outputs the classification results of the tactile paving condition in the image and the feedback report on the construction and use of tactile paving. Finally, the detection results are transmitted to the handheld printer 25 via wireless network. The printer prints out the feedback report on the construction and use of tactile paving 26, giving the user an intuitive understanding and macroscopic analysis of the tactile paving condition in the inspected area, providing technical support for targeted rectification. Specific Implementation Example 2 In this embodiment, the data acquisition device at the information acquisition end can replace the digital camera in Embodiment 1 with an optical camera, infrared camera, etc., or can replace the gimbal on the handlebar of the non-motorized vehicle in Embodiment 1 with a handheld shooting device such as a mobile phone or a white cane, a vehicle-mounted device on a motor vehicle or motorcycle, or an autonomous driving device such as a drone or unmanned vehicle. The acquisition device must have the functions of timed shooting, real-time positioning, and recording shooting time; the carrier of the acquisition device travels along a certain path so that the acquisition device can shoot and record the surrounding tactile paving information. The rest is the same as in Embodiment 1. Specific Implementation Example 3 like Figure 11 As shown: In this embodiment, the element detection section 6-2 of the tactile paving problem judgment module 6 can optionally add a detection result refinement module 27 after obtaining the element detection results in the image based on image recognition technology. This module uses the SOLOv2 algorithm to make more precise adjustments to the target detection box according to the outer contour of the object, and outputs a refined detection result 28. The rest is the same as in embodiment 1. Specific Implementation Example 4 In this embodiment, the output device for the detection results can be replaced by a remote wireless printer, an electronic screen, etc., instead of the handheld printer in Specific Embodiment 1, to present a feedback report on the construction and use of tactile paving. The rest is the same as in Embodiment 1.

[0054] The above embodiments are merely examples of the present invention. As can be seen from the above embodiments, any substitutions made in accordance with the spirit of the present invention, including the selection of specific image detection algorithms, the selection of specific node label prediction algorithms based on graph neural networks, the selection of acquisition and output terminal devices, and the specific names of tactile paving conditions; any device that uses a computer to replace manual tactile paving detection, realizes the identification and classification of tactile paving conditions in images, identifies problematic tactile paving, and generates feedback reports on the construction and use of tactile paving, should be understood as not departing from the protection scope of the present invention.

Claims

1. A method for inspecting problems with tactile paving, comprising: The system comprises three parts: an information acquisition terminal, a tactile paving detection and processing system, and a detection result output terminal. Its key feature is that the information acquisition terminal includes an information acquisition device. The image acquired by the information acquisition device contains geographic location coordinate information, and the acquired information is transmitted to the tactile paving detection and processing system. The tactile paving detection and processing system detects the corresponding image, judges and classifies the tactile paving condition in the image, identifies problematic tactile paving, and integrates the obtained tactile paving construction and usage status classification results with the corresponding geographic location coordinate information to generate a tactile paving construction and usage feedback report, which is then output through the detection result output terminal. The conditions of the tactile paving mentioned include one or all of the following: correct tactile paving, tactile paving occupied by non-motorized vehicles, tactile paving occupied by motorized vehicles, tactile paving occupied by manhole covers, tactile paving occupied by cylindrical objects, incorrect tactile paving at intersections, and incorrect tactile paving at the ends. The tactile paving detection and processing system includes a tactile paving problem judgment module, which includes an image input part, an element detection part, a direction information extraction part, a graph structure transformation part, a tactile paving node label prediction part based on graph neural network, a problem classification and output part, and a feedback report generation part. The image input section inputs the images acquired by the information acquisition terminal into the tactile paving detection and processing system; The element detection section uses image processing methods to detect elements in the image, including the type of the element, the coordinates of the center point of the detection box, and the length and width information of the detection box. The element detection includes one or all of the following: tactile paving for walking, tactile paving with correct end prompts, tactile paving with incorrect end prompts, tactile paving with correct intersection prompts, tactile paving with incorrect intersection prompts, non-motorized vehicles, motorized vehicles, manhole covers, and cylindrical objects. In the direction information extraction section, the periodicity of the arrangement of the tactile paving texture and its prior relationship with the direction of travel of the tactile paving are used to obtain the direction information of travel of the tactile paving. The correlation is calculated by the gray-level co-occurrence matrix, which measures the similarity of the gray levels of the image texture in different directions. The larger the value obtained, the stronger the correlation of the tactile paving texture in that direction, that is, the closer the texture is to repeated occurrence, the closer it is to the direction of travel of the tactile paving. This direction information is then used for the secondary determination of edges in the subsequent abstract graph structure. That is, the "neighboring" edges that connect the tactile paving and non-tactile paving element nodes respectively, and the non-tactile paving nodes are not in the direction of travel of the tactile paving are deleted, so as to realize the accurate modeling of the relative position of obstacles and tactile paving, which is used to determine whether the obstacle occupies the direction of travel of the tactile paving or is adjacent to both sides of the tactile paving. In the graph structure transformation section, based on the element detection results obtained from the element detection section, the relationships between elements in the image are modeled, transforming the real-world scene image into an abstract graph structure composed of points and edges. Nodes are generated from the detected elements and their attributes, with each node's initial feature being a one-hot encoding of its node type. Then, the corresponding IOU index is calculated using the different positional relationships between objects, leading to the identification of "overlapping" and "proximity" edges. After obtaining the preliminary abstract graph structure, the tactile paving direction information is input into the secondary judgment of the edges in the abstract graph structure. That is, "proximity" edges that connect tactile paving and non-tactile paving element nodes, and where the non-tactile paving node is not in the direction of travel on the tactile paving, are deleted, resulting in the final abstract graph structure. The tactile paving node label prediction part, based on graph neural networks, uses inference learning on abstract graph structures to achieve intelligent recognition and classification of tactile paving conditions in images through node label prediction. The node label prediction part employs a heterogeneous graph structure and applies a multi-head attention graph convolutional network (GATS). Different DGLNN modules are used to process each type of relationship according to edge type, allowing information from the source node to be passed to the target node along different relationships. Then, for the same target node, information from different relationships is aggregated for feature update. After updating the node information, since tactile paving condition classification often only relies on neighboring nodes... Furthermore, the abstract graph structure of each image is relatively simple, employing a single-layer attention graph convolutional network. After obtaining normalized attention coefficients through the single-layer attention graph convolutional network, these coefficients are used to calculate a linear combination of corresponding features, serving as the final output feature of each node. The node-level output based on the graph neural network is extracted and fed into a fully connected neural network, transforming the final features of the tactile paving into the tactile paving condition feature space for classification. This yields probability vectors for each type of tactile paving condition in the image, thus enabling inference learning based on the abstract graph structure using graph neural networks. Node label prediction is then used to achieve intelligent recognition and classification of tactile paving problems in images by the computer.

2. The method for inspecting tactile paving problems according to claim 1, characterized in that: In the problem classification and output section, the probability vector of tactile paving problems is mapped one-to-one with the name of the tactile paving condition. The name of the tactile paving condition corresponding to the component with the highest probability value is output as the classification result. If the obtained tactile paving condition is not "correct tactile paving", the severity of the tactile paving problem involved in the image is judged according to the absolute size of the maximum probability value in the tactile paving problem probability vector, and divided into two levels: "severe" and "mild". The report generation section integrates the tactile paving condition classification results with the information obtained from the data collection terminal, and matches the obtained geographical location information with the electronic map to determine the actual location of the corresponding collected image on the electronic map. This generates a feedback report on the construction and use of tactile paving, which is stored locally on the system or uploaded to a network platform. The feedback report includes the number of tactile paving conditions of each type in this data collection, the tactile paving condition classification results, severity, shooting time, and actual geographical location information of each photo, and displays the location of each tactile paving condition on the electronic map.

3. The method for inspecting tactile paving problems according to claim 1, characterized in that: The geographical location coordinates obtained by the information collection terminal are matched with the electronic map to determine the actual location of the corresponding collected image on the electronic map; it can intuitively provide road information of problematic blind paths in the feedback report on the construction and use of blind paths.

4. The method for inspecting tactile paving problems according to claim 1, characterized in that: The information collection device is a device that can capture images at set times, locate them in real time, and record the capture time. The information collection device can be installed on one of the following: a mobile phone, a white cane, a motor vehicle, a non-motor vehicle, a surveillance camera, a drone, or an unmanned vehicle.

5. The method for inspecting tactile paving problems according to claim 1, characterized in that: The tactile paving problem detection module uses one of the following: graph neural network, logical judgment, SVM, KNN, or CNN.

6. The method for inspecting tactile paving problems according to claim 1, characterized in that: The image processing methods mentioned include one of the following: Faster R-CNN, YOLO, color clustering analysis, Radon transform, Gaussian mixture model, Hough transform, H hue thresholding segmentation, Sobel edge detection, Canny edge detection, Lab color model, fuzzy C-means (KFCM) algorithm, single-channel color space thresholding segmentation, K-Means, and FCM segmentation algorithm.

7. The method for inspecting tactile paving problems according to claim 1, characterized in that: The element detection section can be configured to precisely adjust the target detection box based on the outer contour of the object using the SOLOv2 algorithm after element detection in the image is performed.

8. The method for inspecting tactile paving problems according to claim 1, characterized in that: The "overlapping" edge type refers to the relationship between elements when there is an overlap between the detection boxes of different detected elements; the "adjacent" edge type refers to the relationship between elements when there is no overlap between the detection boxes of different detected elements, but there is overlap after the detection boxes are expanded to 105% of their original size; the "severe" level refers to tactile paving nodes with an absolute value of the maximum probability greater than 0.75 in the probability vector of tactile paving problems; the "mild" level refers to tactile paving nodes with an absolute value of the maximum probability less than or equal to 0.75 in the probability vector of tactile paving problems.

9. The method for inspecting tactile paving problems according to claim 1, characterized in that: The names of the conditions of the tactile paving include one or all of the following: correct tactile paving, tactile paving occupied by non-motorized vehicles, tactile paving occupied by motorized vehicles, tactile paving occupied by manhole covers, tactile paving occupied by cylindrical objects, incorrect tactile paving at intersections, and incorrect tactile paving at the ends.

10. The method for inspecting tactile paving problems according to claim 1, characterized in that: The detection result output terminal outputs the classification results of the tactile paving condition in the image obtained by the tactile paving detection and processing system, as well as a feedback report on the construction and use of the tactile paving; the detection result output terminal device includes one of a handheld printer, a remote wireless printer, or an electronic screen.

Images