Method, device and equipment for surveying non-public road and storage medium

Abstract

The invention discloses a method, a device and equipment for surveying a non-public road and a storage medium. The method comprises the following steps: respectively acquiring a point cloud image and an RGB image of a current reconnaissance road section in a non-public road scene; the RGB curb and the point cloud curb of the current investigation road section are identified respectively; based on the confidence evaluation index, respectively performing confidence evaluation on the RGB curb and the point cloud curb; if the confidence evaluation results of the point cloud curb and the RGB curb are both greater than a set first threshold value, determining that the survey is reliable, and determining the road information of the current survey road section based on the point cloud curb and the RGB curb; and if at least one of the two is not greater than the set first threshold value, determining that the survey is unreliable, and performing the survey again. Therefore, manual surveying specialty is not needed, automatic intelligent surveying is realized, surveying cost and time consumption are reduced, and surveying accuracy is high.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to technical fields such as autonomous driving control and road survey and evaluation, and in particular to a method, device, equipment and storage medium for surveying non-public roads. Background Art

[0002] Currently, there are an increasing number of devices equipped with autonomous driving systems, particularly those used in non-municipal and non-public roads, such as parks, industrial parks, and scenic spots, which have become the primary operating environments for functional autonomous driving equipment. However, compared to municipal non-public roads, these non-public roads in these scenarios suffer from incomplete data, inconsistent design standards, and complex infrastructure.

[0003] Therefore, to assess whether the non-public roads within a scenario meet the requirements for autonomous driving equipment operation, a common approach to obtaining a relatively accurate assessment is to arrange for experienced engineers to conduct on-site surveys of the non-public roads where autonomous driving equipment will be deployed. This involves simply measuring basic data on the non-public roads, such as subsidence height, length, width, height, slope, obstacles, and obstructions. Engineers can also combine relevant basic information about the autonomous driving equipment and its autonomous driving capabilities, such as length, width, height, gradeability, turning radius, and autonomous driving system performance, to comprehensively evaluate the feasibility of using the autonomous driving equipment on non-public roads within the scenario.

[0004] However, the above manual survey solution is heavily dependent on experienced engineers, has high survey costs and a long survey cycle, and the professionalism and accuracy of the survey cannot be guaranteed. Summary of the Invention

[0005] The present application provides a method, apparatus, device and storage medium for surveying non-public roads to solve the problems of existing road survey solutions that rely on manual survey expertise, have high survey costs, are time-consuming, and have low survey accuracy.

[0006] The technical solution is as follows:

[0007] In a first aspect, a method for surveying a non-public road is provided, comprising:

[0008] In non-public road scenarios, obtain the point cloud image and RGB image of the current survey section and perform image preprocessing operations;

[0009] Based on the curb recognition model trained by the Faster-RCNN algorithm, the RGB curb of the current surveyed road section is identified from the RGB image after preprocessing operation;

[0010] Extracting a point cloud curb of the current surveyed road section from the point cloud image based on a road window corresponding to a set single-directional scanning line;

[0011] Based on the confidence evaluation index, respectively performing confidence evaluation on the RGB curb and the point cloud curb;

[0012] If the confidence evaluation results of the RGB curb and the confidence evaluation results of the point cloud curb are both greater than the set first threshold, the survey is determined to be reliable, and the road information of the current surveyed section is determined based on the point cloud curb and the RGB curb;

[0013] If at least one of the confidence evaluation result of the RGB curb and the confidence evaluation result of the point cloud curb is not greater than the set first threshold, it is determined that the current survey is unreliable and the above survey is performed again.

[0014] In one possible implementation, the curb recognition model trained using the Faster-RCNN algorithm is determined as follows:

[0015] Obtaining historical RGB images of other non-public road scenes as image samples, and inputting the image samples into the initialized Faster-RCNN model to train the RPN network; each historical RGB image is marked with one or more curbs;

[0016] The candidate boxes generated by the trained RPN network are input into the Faster-RCNN network for training;

[0017] Initialize a new RPN network using the network parameters trained by the Faster-RCNN network; wherein the new RPN network shares network parameters and all common convolutional layers with the Faster-RCNN network;

[0018] The Faster-RCNN network is fine-tuned to obtain a trained curb recognition model.

[0019] In a possible implementation, based on a set road window corresponding to a single-directional scanning line, extracting a point cloud curb of the current surveyed road section from the point cloud image specifically includes:

[0020] Converting the point cloud image into a linear scan point cloud map comprising a series of ordered two-dimensional scan line data sets; wherein the two-dimensional scan lines are perpendicular to the survey travel direction;

[0021] Extracting a scan line cross-section corresponding to each two-dimensional scan line from the linear scan point cloud map as a road window;

[0022] Estimate the curb boundaries on the left and right sides based on the discrete point clouds on the left and right sides of each road window, respectively, wherein the curb boundaries include the curb boundary position and height;

[0023] The curb boundaries of each road window are summarized to obtain the point cloud curb of the current survey section.

[0024] In a possible implementation, determining the road information of the current surveyed road section based on the point cloud curb and the RGB curb specifically includes:

[0025] Cutting the RGB image based on the RGB curb, wherein the RGB image retained after cutting includes a curb image area and a road surface image area between the curbs;

[0026] Cutting the point cloud image based on the point cloud curb, wherein the point cloud image retained after cutting includes a curb image area and a road surface image area between the curbs;

[0027] The RGB image and the point cloud image retained after cutting are used as the road information of the current survey section.

[0028] In a possible implementation, cutting the point cloud image based on the point cloud curb specifically includes:

[0029] The image area outside the point cloud curb in the point cloud image is cut off and removed, and the image area in the point cloud image whose distance from the road surface is greater than a set second threshold is cut off and removed.

[0030] In a possible implementation, after determining the road information of the current surveyed road section based on the point cloud curb and the RGB curb, the method further includes:

[0031] Acquire key parameters of the target intelligent body, wherein the key parameters include at least: size and device performance of the target intelligent body;

[0032] Determining road characterization parameters based on image information related to the point cloud in the road information, wherein the road characterization parameters include at least: the slope, length, width, height, and obstacles of the current surveyed road section;

[0033] Perform parameter matching evaluation on the road characterization parameters of the current surveyed road section and the key parameters of the target intelligent agent;

[0034] Verifying the matching evaluation result based on the RGB-related image information in the road information;

[0035] Based on the verification results, the suitability and deployment risk of the target intelligent agent for use in the current survey section are evaluated.

[0036] In a second aspect, a survey device for a non-public road is provided, comprising:

[0037] The acquisition module is used to obtain the point cloud image and RGB image of the current survey section in the non-public road scene and perform image preprocessing operations;

[0038] A recognition module is used to identify the RGB curb information of the current surveyed road section from the RGB image after preprocessing based on the curb recognition model trained using the Faster-RCNN algorithm;

[0039] An extraction module, configured to extract point cloud curb information of a currently surveyed road section from the point cloud image based on a road window corresponding to a set single-directional scanning line;

[0040] An evaluation module, configured to perform confidence evaluation on the RGB curb information and the point cloud curb information based on a confidence evaluation index;

[0041] a determination module, configured to determine that the survey is reliable if the confidence evaluation results of the RGB curb information and the confidence evaluation results of the point cloud curb information are both greater than a set threshold, and determine the road information of the current surveyed section based on the point cloud curb information and the RGB curb information;

[0042] And, if at least one of the confidence evaluation results of the RGB curb information and the confidence evaluation results of the point cloud curb information is not greater than a set threshold, it is determined that the current survey is unreliable and the survey is repeated.

[0043] According to a third aspect, an electronic device is provided, including:

[0044] at least one processor; and

[0045] a memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0047] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the method of the above-mentioned aspect and any possible implementation manner.

[0048] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the above-mentioned aspects and any possible implementation method.

[0049] The beneficial effects of the technical solution provided by this application include at least:

[0050] As can be seen from the above technical solution, the embodiment of the present application obtains the point cloud image and RGB image of the current survey section in a non-public road scene; and respectively identifies the RGB curb and point cloud curb of the current survey section; based on the confidence evaluation index, the confidence evaluation is performed on the RGB curb and point cloud curb respectively; if the confidence evaluation results of both are greater than the set first threshold, the survey is determined to be reliable, and the road information of the current survey section is determined based on the point cloud curb and the RGB curb; if at least one of the two is not greater than the set first threshold, the survey is determined to be unreliable, and the above survey is repeated. Therefore, there is no need to rely on the professionalism of manual survey, and automatic intelligent survey is achieved, which reduces the cost and time of survey and has high survey accuracy.

[0051] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 This is a schematic diagram of the steps of the survey method for non-public roads provided in the embodiment of the present application. Figure 1 .

[0054] Figure 2 This is a simplified schematic diagram of the smart handheld device provided in an embodiment of the present application.

[0055] Figure 3a This is a linear scanning point cloud image provided by an embodiment of the present application.

[0056] Figure 3b This is a scan line cross-sectional view provided by an embodiment of the present application.

[0057] Figure 4a This is a schematic diagram of the survey process of a non-public road provided in an embodiment of the present application.

[0058] Figure 4b This is a schematic diagram of the survey and evaluation process of non-public roads provided in an embodiment of the present application.

[0059] Figure 5This is a schematic diagram of the steps of the survey method for non-public roads provided in the embodiment of the present application. Figure 2 .

[0060] Figure 6 This is a structural block diagram of a survey device for non-public roads provided in one embodiment of the present application.

[0061] Figure 7 This is a schematic diagram of the structure of a survey and evaluation system for non-public roads provided in one embodiment of the present application.

[0062] Figure 8 This is a block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following description of exemplary embodiments of the present application is provided in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0064] Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0065] It should be noted that the terminal devices involved in the embodiments of the present application may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0066] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0067] In view of the fact that manual survey solutions rely heavily on experienced engineers, have high survey costs and long survey cycles, and cannot guarantee survey professionalism and accuracy, this application proposes a survey solution for non-public roads. The main inventive concept is: in a non-public road scenario, respectively obtain point cloud images and RGB images of the current survey section; and respectively identify the RGB curbs and point cloud curbs of the current survey section; based on the confidence evaluation index, respectively perform confidence evaluation on the RGB curbs and point cloud curbs; if the confidence evaluation results of both are greater than a set first threshold, then the survey is determined to be reliable, and the road information of the current survey section is determined based on the point cloud curbs and the RGB curbs; if at least one of the two is not greater than the set first threshold, then the survey is determined to be unreliable, and the above survey is repeated. Therefore, there is no need to rely on the professionalism of manual surveys, automatic intelligent surveys are achieved, survey costs and time consumption are reduced, and survey accuracy is high.

[0068] Reference Figure 1 The following is a schematic diagram of the steps of a method for surveying non-public roads provided in an embodiment of the present application. Figure 1 It should be understood that the execution subject of the survey method can be a survey device, which can be a software module or hardware device with computer data processing, storage and other functions as well as data acquisition functions, such as a computer with an acquisition module, a tablet computer, a smart phone or a smart wearable device. Specifically, the survey device can be a smart handheld device, such as Figure 2 As shown, the system includes a handheld pole 201 and an intelligent host 202 located above the handheld pole 201. A solid-state laser radar 203 and a camera 204 are mounted on the outside of the intelligent host 202. The solid-state laser radar 203 is used to collect point cloud images of the surveyed road section, and the camera 204 is used to collect RGB images of the surveyed road section. The intelligent host 202 is used to receive the point cloud images collected by the solid-state laser radar 203 and the RGB images collected by the camera 204, and perform image processing to obtain the required road information.

[0069] The survey method may include the following steps:

[0070] Step 102: In a non-public road scenario, a point cloud image and an RGB image of the current surveyed road section are obtained respectively, and image preprocessing operations are performed.

[0071] In this application, a solid-state LiDAR with 64 or more laser lines can be used to collect point cloud images, while a camera can be used to collect RGB images. Image preprocessing operations such as classification and noise reduction are then performed on the point cloud images, while also on the RGB images, to optimize image data and improve image reliability.

[0072] Step 104: Based on the curb recognition model trained using the Faster-RCNN algorithm, the RGB curb of the current surveyed road section is identified from the RGB image after the preprocessing operation.

[0073] Optionally, the curb recognition model trained using the Faster-RCNN algorithm is determined by:

[0074] The first step is to obtain historical RGB images of other non-public road scenes as image samples, and input the image samples into the initialized Faster-RCNN model to train the RPN network; each historical RGB image is marked with one or more curbs.

[0075] Specifically, by obtaining curb data from other non-public roads and a large amount of RGB image data collected by cameras, the feature extraction network is initialized in the ImageNe pre-trained model, and the RPN network is trained.

[0076] In the second step, the candidate boxes generated by the trained RPN network are input into the Faster-RCNN network for training.

[0077] Use the model pre-trained on ImageNet to initialize the Fast-RCNN feature extraction network, use the candidate boxes generated by the RPN network trained in the first step as input, and train a Fast-RCNN network. At this time, the parameters of each layer of the two networks are completely unshared.

[0078] The third step is to initialize a new RPN network using the network parameters trained by the Faster-RCNN network; wherein the new RPN network shares network parameters and all common convolutional layers with the Faster-RCNN network.

[0079] Initialize a new RPN network using the Fast-RCNN network parameters from step 2. At the same time, set the learning rate of the network parameters of the feature extraction network shared by the RPN and Fast-RCNN to 0. This means that the parameters unique to the RPN network are learned, while the feature extraction network is fixed. At this point, the two networks share all common convolutional layers.

[0080] The fourth step is to fine-tune the Faster-RCNN network to obtain a trained curb recognition model.

[0081] The shared network layers are fixed, the Fast-RCNN network's unique network layers are taken into account, and training continues. Based on the curb training results, the Fast-RCNN network's unique network layers are fine-tuned, so that the RPN and Fast-RCNN networks fully share parameters. The trained Fast-RCNN model can then simultaneously perform candidate box extraction and non-public road curb feature recognition. It should be understood that in this application solution, the Fast-RCNN model can include both the RPN network and the Faster-RCNN network.

[0082] The advantage of the Faster-RCNN model over other deep learning algorithms is that it introduces RPN, which directly generates candidate regions. In this application scheme, a CNN model is first used to receive the RGB image and extract the feature map. This feature map is then divided into 5×5 sliding windows, where each sliding window position maps a low-dimensional feature. These low-dimensional features are then fed into two fully connected layers, one for classification prediction and the other for regression. In the actual recognition process, 25 prior boxes of different proportions containing non-public road curb information can be set for each sliding window position, and 25 candidate regions are predicted for each sliding window position. For the classification layer, its output size can be 50, indicating the probability value of each candidate region for the possibility of containing a curb in these 50 regions, while the regression layer outputs 100 coordinate values, indicating the position of each candidate region (relative to each prior box). For each sliding window position, these two fully connected layers are shared. Therefore, RPN can be implemented using convolutional layers: first, a 5×5 convolution is performed to obtain low-dimensional features, followed by two 1×1 convolutions, used for classification and regression respectively. It should be noted that the setting of 25 prior boxes and the setting of the classification layer output size to 50 are just examples and do not limit this solution.

[0083] Therefore, the RGB image after the preprocessing operation can be input into the curb recognition model trained in the above manner to achieve curb recognition and positioning of the survey section of the non-public road.

[0084] Step 106: extracting the point cloud curb of the current surveyed road section from the point cloud image based on the road window corresponding to the set single-directional scanning line.

[0085] Optionally, in the solution of the present application, based on the road window corresponding to the set single-directional scanning line, extracting the point cloud curb of the current surveyed road section from the point cloud image may specifically include the following steps:

[0086] Step 1: Convert the point cloud image into a linear scan point cloud image comprising a series of ordered two-dimensional scan line data sets; wherein the two-dimensional scan lines are perpendicular to the survey travel direction.

[0087] Solid-state laser radar mainly performs linear scanning, and the scan points obtained are arranged according to the scan lines. In the road environment, the top is mostly the sky, and the data obtained by the vehicle-mounted laser scanning system is distributed in a strip-shaped, non-closed manner. The vehicle-mounted laser scanner mostly uses a panoramic scan with a 360-degree field of view. The arrangement of the scan lines extracted according to the GPS time or scanning angle of the point cloud is basically along the direction of travel of the vehicle. Each scan line is equivalent to a cross section of the road. The discretely distributed point cloud in the point cloud image is converted into a series of ordered two-dimensional scan line data sets, that is, Figure 3a The linear scan point cloud shown.

[0088] Step 2: extract the scanning line cross-section corresponding to each two-dimensional scanning line from the linear scanning point cloud image as a road window.

[0089] From the linear scan point cloud, extract the scan line cross-section corresponding to each two-dimensional scan line as the road window, and refer to Figure 3b As shown in the figure, different targets (here referring to targets such as roads, curbs, and obstructions in the point cloud image) show different spatial distributions and geometric shapes on the same scan line. At the same time, adjacent scan lines have similar spatial distributions. From the spatial distribution law of the point cloud scan line, the ground is relatively flat and the elevation value is relatively small and does not change much. It is generally distributed horizontally on the scan line and can be approximately regarded as a smooth and continuous horizontal straight line.

[0090] In step 3, based on the discrete point clouds on the left and right sides of each road window, the curb boundaries on the left and right sides are estimated, where the curb boundaries include the curb boundary position and height.

[0091] Still refer to Figure 3b As shown, for each road window, the discrete point cloud on the left is counted and clustered to obtain the left curb boundary corresponding to the left point cloud. The discrete point cloud on the right is counted and clustered to obtain the right curb boundary corresponding to the right point cloud. This boundary does not need to be precise; it only needs to be estimated based on the discrete distance calculation between the point clouds to form a virtual center point of the curb boundary. Through coordinate calculation and matching, fuzzy calculation is performed to cluster the point clouds within 10 cm to the left and right of the center point and 50 cm in height. This determines the location and height of the curb boundary, resulting in the left and right curb boundaries.

[0092] In step 4, the curb boundaries of each road window are summarized to obtain the point cloud curb of the current survey section.

[0093] Step 108: Based on the confidence evaluation index, perform confidence evaluation on the RGB curb and the point cloud curb.

[0094] For RGB curbs and point cloud curbs, the confidence evaluation method can be used to select the same or different confidence evaluation indexes to perform confidence evaluation separately.

[0095] Step 110: If the confidence evaluation results of the RGB curb and the confidence evaluation results of the point cloud curb are both greater than the set first threshold, it is determined that the survey is reliable, and the road information of the current survey section is determined based on the point cloud curb and the RGB curb.

[0096] In the present application, the first threshold may be flexibly set based on empirical data or business requirements. For example, the first threshold may be 80%, 85%, 90%, or 95%.

[0097] Optionally, when determining the road information of the current survey section based on the point cloud curb and the RGB curb, the RGB image can be cut based on the RGB curb, wherein the RGB image retained after cutting includes the curb image area and the road surface image area between the curbs; and, the point cloud image can be cut based on the point cloud curb, wherein the point cloud image retained after cutting includes the curb image area and the road surface image area between the curbs; thereafter, the RGB image retained after cutting and the point cloud image retained after cutting are used as the road information of the current survey section.

[0098] When the point cloud image is cut based on the point cloud curb, the image area outside the point cloud curb in the point cloud image is cut and removed, and the image area in the point cloud image whose distance from the road surface is greater than a set second threshold is cut and removed. Figure 3b As shown, for example, if the second threshold is set to 5m, the area below the red dotted box is cut out and the required road information is retained.

[0099] Step 112: If at least one of the confidence evaluation result of the RGB curb and the confidence evaluation result of the point cloud curb is not greater than a set first threshold, it is determined that the current survey is unreliable and the above survey is repeated.

[0100] Taking the confidence evaluation index of 95% as an example, if the confidence evaluation result of the RGB curb is greater than 95%, but the confidence evaluation result of the point cloud curb is less than 95%; or, the confidence evaluation result of the RGB curb is less than 95%, but the confidence evaluation result of the point cloud curb is greater than 95%; or, the confidence evaluation result of the RGB curb is less than 95%, but the confidence evaluation result of the point cloud curb is less than 95%, then it is determined that the survey data is insufficient and unreliable, and the above survey needs to be re-executed.

[0101] It should be noted that based on Figure 2In the handheld device shown, a corresponding radar indicator light is located below the solid-state laser radar 203, and a corresponding camera indicator light is located below the camera 204. When the confidence assessment results indicate that the survey is reliable, both the radar and camera indicators on the handheld device illuminate green. When the confidence assessment results indicate that the survey is unreliable, the corresponding indicators illuminate red. These indicators provide a timely and convenient reminder to the handheld user regarding the reliability of the survey results, allowing for prompt re-survey or continued survey of the next road section.

[0102] Reference Figure 4a , which is a schematic diagram of the survey process of a non-public road provided in an embodiment of the present application.

[0103] The user turns on the survey switch of the handheld device, and the solid-state lidar and camera simultaneously collect and process images.

[0104] For the solid-state lidar side: acquire point cloud images through the lidar, perform image classification and noise reduction preprocessing on the point cloud images, determine the curbs in the point cloud, and perform a confidence evaluation on the determined curbs. If the confidence level is greater than 90%, segment and extract the point cloud curbs; otherwise, jump back to the step of acquiring the point cloud images.

[0105] On the camera side: RGB images are acquired through the camera, image classification and noise reduction are performed on the RGB images, curbs in the images are identified and located, and the confidence level of the identified curbs is evaluated. If the confidence level is greater than 95%, the curbs in the image are segmented and extracted. Otherwise, the process jumps back to the step of acquiring the RGB image.

[0106] This technical solution, through RGB and point cloud image recognition and segmentation, can obtain information related only to roads and curbs, enabling rapid offline processing with low equipment requirements and directly filtering out other interfering images and point clouds. At the same time, the introduction of a confidence evaluation index enables offline evaluation of the acquisition operation, ensuring the reliability of the data source required for the acquisition.

[0107] Furthermore, in the present application scheme, after surveying non-public roads and obtaining road information of the surveyed sections, the road information can also be used, combined with the product parameter information of the intelligent body, to automatically and intelligently evaluate the feasibility of deploying different intelligent bodies on the surveyed non-public roads, to determine which intelligent bodies are suitable for deployment on the surveyed non-public roads, which are not suitable for deployment, and the risks of deployment, etc., thereby avoiding the dependence and uncertainty of manual evaluation and improving evaluation efficiency and accuracy.

[0108] Reference Figure 5 The following is a schematic diagram of the steps of the survey method for non-public roads provided in the embodiment of the present application. Figure 2 Specifically, after determining the road information of the current surveyed road section based on the point cloud curb and the RGB curb, the following steps may also be included:

[0109] Step 114: Acquire key parameters of the target intelligent body, wherein the key parameters include at least the size and device performance of the target intelligent body.

[0110] The target agent can be an intelligent driving device that is about to be deployed, such as a cleaning device, a passenger-carrying device, or a delivery device with autonomous driving capabilities. The dimensions of the target agent include its length, width, and height. Equipment performance includes gradeability, turning radius, and system performance. Furthermore, the target agent's model, batch, and manufacturer information may also be included.

[0111] Step 116: Determine road characterization parameters based on the image information related to the point cloud in the road information, where the road characterization parameters include at least: the slope, length, width, height, and obstacles of the current surveyed road section.

[0112] In addition, parameters such as road settlement height and obstructions can also be included.

[0113] Step 118: Perform parameter matching evaluation on the road characterization parameters of the current surveyed road section and the key parameters of the target intelligent agent.

[0114] Specifically, a digital twin model of the current surveyed road section and the target agent is constructed to map the current surveyed road section and the target agent into a virtual environment. Based on the road characterization parameters of the current surveyed road section and the key parameters of the target agent, a variety of driving behaviors are simulated. Based on the simulation results, a matching evaluation of the different parameters is performed. For example, in the digital twin model, the agent is simulated driving at different speeds on a road section with a curvature radius of r, and its steering control and driving stability are observed to obtain matching evaluation results for speed and road curvature.

[0115] Step 120: Verify the matching evaluation result based on the RGB-related image information in the road information.

[0116] Specifically, for unmatched parameter items in the matching evaluation results, or fuzzy matching parameter items, the RGB image of the corresponding road section is searched from the image database established based on RGB-related image information; based on the concrete content in the found RGB image, the matching result is analyzed and verified.

[0117] Step 122: Evaluate the suitability and deployment risk of the target intelligent agent for use in the current survey section based on the verification results.

[0118] Finally, the verification and matching assessment results are summarized to comprehensively assess the suitability and deployment risk of the target agent for the current surveyed road section. Specifically, a table listing different parameter items can be used to summarize the suitability and deployment risk based on a scoring method.

[0119] Reference Figure 4b , which is a schematic diagram of the survey and evaluation process of a non-public road provided in an embodiment of the present application.

[0120] Turn on the handheld device's survey function to extract road characterization parameters based on the point cloud image. Simultaneously, establish an image database based on the RGB image. Turn on the product addition function for the product adding device (this product addition device can be a third-party communication device, such as a mobile phone, or the handheld device itself, meaning it has product addition functionality) to obtain key product parameters. These road characterization parameters and key product parameters are then evaluated in the matching evaluation backend. The evaluation results are then output and verified against the RGB images in the image database.

[0121] The evaluation scheme for this part does not require field deployment tests of target intelligent agents. It only needs to summarize in advance or input in real time the key parameters of different target intelligent agents, which can meet the rapid switching evaluation of different intelligent agent products. At the same time, it can realize the rapid evaluation of the adaptability and deployment risk of roads and intelligent agent products through comparative evaluation of characterization parameters.

[0122] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0123] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0124] Figure 6 FIG. 1 shows a structural block diagram of a survey device for non-public roads provided by an embodiment of the present application. Figure 6As shown. The survey device 600 for non-public roads in this embodiment may include an acquisition module 601, an identification module 602, an extraction module 603, an evaluation module 604 and a determination module 605. Among them, the acquisition module 601 is used to respectively acquire the point cloud image and RGB image of the current survey section in the non-public road scene, and perform image preprocessing operations. The identification module 602 is used to identify the RGB curb information of the current survey section from the RGB image after the preprocessing operation based on the curb recognition model trained using the Faster-RCNN algorithm. The extraction module 603 is used to extract the point cloud curb information of the current survey section from the point cloud image based on the road window corresponding to the set single-directional scanning line. The evaluation module 604 is used to perform confidence evaluation on the RGB curb information and the point cloud curb information using different confidence evaluation indices. Determination module 605 is used to determine that the current survey is reliable if both the confidence evaluation results of the RGB curb information and the confidence evaluation results of the point cloud curb information are greater than a set threshold, and to determine the road information of the current surveyed section based on the point cloud curb information and the RGB curb information; and to determine that the current survey is unreliable and to re-survey if at least one of the confidence evaluation results of the RGB curb information and the confidence evaluation results of the point cloud curb information is not greater than a set threshold.

[0125] It should be noted that part or all of the survey device for non-public roads in this embodiment can be an application located in the local terminal, or can also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or can also be a processing engine located in the network side server, or can also be a distributed system located on the network side. This embodiment does not specifically limit this.

[0126] It is understandable that the application may be a native program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0127] Optionally, in a possible implementation of this embodiment, the curb recognition model obtained by training using the Faster-RCNN algorithm is determined by:

[0128] Historical RGB images of other non-public road scenes are obtained as image samples, and the image samples are input into the initialized Faster-RCNN model to train the RPN network; each historical RGB image is marked with one or more curbs; the candidate boxes generated by the trained RPN network are input into the Faster-RCNN network for training; the network parameters of the trained Faster-RCNN network are used to initialize a new RPN network; wherein the new RPN network shares network parameters and all common convolutional layers with the Faster-RCNN network; the Faster-RCNN network is fine-tuned to obtain a trained curb recognition model.

[0129] Optionally, in a possible implementation of this embodiment, when the extraction module 603 extracts the point cloud curb of the current surveyed road section from the point cloud image based on the road window corresponding to the set single-directional scan line, it is specifically configured to:

[0130] The point cloud image is converted into a linear scan point cloud map comprising a series of ordered two-dimensional scan line data sets; wherein the two-dimensional scan lines are perpendicular to the survey travel direction; a scan line cross-section corresponding to each two-dimensional scan line is extracted from the linear scan point cloud map as a road window; and the curb boundaries on the left and right sides are estimated based on the discrete point clouds on the left and right sides of each road window, wherein the curb boundaries include the curb boundary position and height; and the curb boundaries of each road window are aggregated to obtain a point cloud curb of the current survey section.

[0131] Optionally, in a possible implementation of this embodiment, when determining the road information of the current surveyed road section based on the point cloud curb and the RGB curb, the determination module 605 is specifically configured to:

[0132] The RGB image is cut based on the RGB curb, wherein the RGB image retained after cutting includes the curb image area and the road surface image area between the curbs; the point cloud image is cut based on the point cloud curb, wherein the point cloud image retained after cutting includes the curb image area and the road surface image area between the curbs; and the RGB image retained after cutting and the point cloud image retained after cutting are used as road information of the current survey section.

[0133] Optionally, in a possible implementation of this embodiment, when the determination module 605 cuts the point cloud image based on the point cloud curb, it is specifically configured to:

[0134] The image area outside the point cloud curb in the point cloud image is cut off and removed, and the image area in the point cloud image whose distance from the road surface is greater than a set second threshold is cut off and removed.

[0135] Optionally, in a possible implementation of this embodiment, the survey device further includes: a matching module, an evaluation module and a verification module; after the determination module determines the road information of the current survey section based on the point cloud curb and the RGB curb, the acquisition module is further used to obtain the key parameters of the target intelligent body, and the key parameters include at least: the size of the target intelligent body and the equipment performance; the determination module is further used to determine the road characterization parameters based on the point cloud-related image information in the road information, and the road characterization parameters include at least: the slope, length, width, height and obstacles of the current survey section; the matching module is used to perform parameter matching evaluation on the road characterization parameters of the current survey section and the key parameters of the target intelligent body; the verification module is used to verify the matching evaluation results based on the RGB-related image information in the road information; the evaluation module is used to evaluate the adaptability and deployment risk of the target intelligent body in the current survey section based on the verification results.

[0136] Reference Figure 7 As shown, the present application further provides a survey and evaluation system 700 for non-public roads, including: a survey device 701 and an evaluation device 702 .

[0137] Among them, the survey device 701 can be Figure 2 The handheld device shown is used to perform road section surveys, specifically: in a non-public road scenario, a point cloud image and an RGB image of the current surveyed road section are respectively obtained, and image preprocessing operations are performed; based on a curb recognition model trained using the Faster-RCNN algorithm, the RGB curb of the current surveyed road section is identified from the RGB image after the preprocessing operation; based on a road window corresponding to a set single-directional scan line, the point cloud curb of the current surveyed road section is extracted from the point cloud image; based on a confidence evaluation index, the confidence of the RGB curb and the point cloud curb are respectively evaluated; if the confidence evaluation results of the RGB curb and the confidence evaluation results of the point cloud curb are both greater than a set first threshold, the current survey is determined to be reliable, and road information of the current surveyed road section is determined based on the point cloud curb and the RGB curb; if at least one of the confidence evaluation results of the RGB curb and the confidence evaluation results of the point cloud curb is not greater than the set first threshold, the current survey is determined to be unreliable, and the above-mentioned survey is performed again.

[0138] The evaluation device 702 can be a background server, which is used to perform a matching evaluation between the surveyed road and the target intelligent body. Specifically, the key parameters of the target intelligent body are obtained, and the key parameters include at least: the size and equipment performance of the target intelligent body; the road characterization parameters are determined based on the image information related to the point cloud in the road information, and the road characterization parameters include at least: the slope, length, width, height and obstacles of the current surveyed road section; the road characterization parameters of the current surveyed road section are evaluated for parameter matching with the key parameters of the target intelligent body; the matching evaluation results are verified based on the RGB-related image information in the road information; and the adaptability and deployment risk of the target intelligent body for use in the current surveyed road section are evaluated based on the verification results.

[0139] Among them, the background server can be a cloud background or a distributed background, or an ordinary server, and this application does not limit this.

[0140] In this embodiment, in a non-public road scenario, a point cloud image and an RGB image of the currently surveyed road section can be obtained, respectively; the RGB curbs and point cloud curbs of the currently surveyed road section can be identified, respectively; based on the confidence evaluation index, the confidence of the RGB curbs and the point cloud curbs can be evaluated; if both confidence evaluation results are greater than a set first threshold, the survey is determined to be reliable, and the road information of the currently surveyed road section is determined based on the point cloud curbs and the RGB curbs; if at least one of the two is less than the set first threshold, the survey is determined to be unreliable, and the survey is repeated. This eliminates the need for manual survey expertise and enables automated intelligent surveying, reducing survey costs and time consumption, and achieving high survey accuracy.

[0141] One embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the blind spot detection method as described above.

[0142] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the blind spot detection method as described above.

[0143] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0144] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0145] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0146] Multiple components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0147] The computing unit 801 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the blind spot detection method. For example, in some embodiments, the blind spot detection method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the blind spot detection method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the blind spot detection method by any other appropriate means (e.g., by means of firmware).

[0148] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0149] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0152] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0153] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0154] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved. This is not a limitation herein.

[0155] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A method for surveying a non-public road, characterized in that: include: In non-public road scenarios, obtain the point cloud image and RGB image of the current survey section and perform image preprocessing operations; Based on the curb recognition model trained by the Faster-RCNN algorithm, the RGB curb of the current surveyed road section is identified from the RGB image after preprocessing operation; Extracting a point cloud curb of the current surveyed road section from the point cloud image based on a road window corresponding to a set single-directional scanning line; Based on the confidence evaluation index, respectively performing confidence evaluation on the RGB curb and the point cloud curb; If the confidence evaluation results of the RGB curb and the confidence evaluation results of the point cloud curb are both greater than the set first threshold, the survey is determined to be reliable, and the road information of the current surveyed section is determined based on the point cloud curb and the RGB curb; If at least one of the confidence evaluation result of the RGB curb and the confidence evaluation result of the point cloud curb is not greater than the set first threshold, it is determined that the current survey is unreliable and the above survey is performed again.

2. The method according to claim 1, wherein The curb recognition model trained using the Faster-RCNN algorithm is determined in the following way: Obtaining historical RGB images of other non-public road scenes as image samples, and inputting the image samples into the initialized Faster-RCNN model to train the RPN network; each historical RGB image is marked with one or more curbs; The candidate boxes generated by the trained RPN network are input into the Faster-RCNN network for training; Initialize a new RPN network using the network parameters trained by the Faster-RCNN network; wherein the new RPN network shares network parameters and all common convolutional layers with the Faster-RCNN network; The Faster-RCNN network is fine-tuned to obtain a trained curb recognition model.

3. The method according to claim 1, wherein Extracting the point cloud curb of the current surveyed road section from the point cloud image based on the road window corresponding to the set single-directional scanning line, specifically including: Converting the point cloud image into a linear scan point cloud map comprising a series of ordered two-dimensional scan line data sets; wherein the two-dimensional scan lines are perpendicular to the survey travel direction; Extracting a scan line cross-section corresponding to each two-dimensional scan line from the linear scan point cloud map as a road window; Estimate the curb boundaries on the left and right sides based on the discrete point clouds on the left and right sides of each road window, respectively, wherein the curb boundaries include the curb boundary position and height; The curb boundaries of each road window are summarized to obtain the point cloud curb of the current survey section.

4. The method according to claim 1, wherein Determining the road information of the current surveyed road section based on the point cloud curb and the RGB curb specifically includes: Cutting the RGB image based on the RGB curb, wherein the RGB image retained after cutting includes a curb image area and a road surface image area between the curbs; Cutting the point cloud image based on the point cloud curb, wherein the point cloud image retained after cutting includes a curb image area and a road surface image area between the curbs; The RGB image and the point cloud image retained after cutting are used as the road information of the current survey section.

5. The method according to claim 4, wherein Cutting the point cloud image based on the point cloud curb specifically includes: The image area outside the point cloud curb in the point cloud image is cut off and removed, and the image area in the point cloud image whose distance from the road surface is greater than a set second threshold is cut off and removed.

6. The method according to any one of claims 1 to 5, wherein: After determining the road information of the current surveyed road section based on the point cloud curb and the RGB curb, the method further includes: Acquire key parameters of the target intelligent body, wherein the key parameters include at least: size and device performance of the target intelligent body; Determining road characterization parameters based on image information related to the point cloud in the road information, wherein the road characterization parameters include at least: the slope, length, width, height, and obstacles of the current surveyed road section; Perform parameter matching evaluation on the road characterization parameters of the current surveyed road section and the key parameters of the target intelligent agent; Verifying the matching evaluation result based on the RGB-related image information in the road information; Based on the verification results, the suitability and deployment risk of the target intelligent agent for use in the current survey section are evaluated.

7. A survey device for non-public roads, characterized in that: include: The acquisition module is used to obtain the point cloud image and RGB image of the current survey section in the non-public road scene and perform image preprocessing operations; A recognition module is used to identify the RGB curb information of the current surveyed road section from the RGB image after preprocessing based on the curb recognition model trained using the Faster-RCNN algorithm; An extraction module, configured to extract point cloud curb information of a currently surveyed road section from the point cloud image based on a road window corresponding to a set single-directional scanning line; An evaluation module, configured to perform confidence evaluation on the RGB curb information and the point cloud curb information based on a confidence evaluation index; a determination module, configured to determine that the survey is reliable if the confidence evaluation results of the RGB curb information and the confidence evaluation results of the point cloud curb information are both greater than a set threshold, and determine the road information of the current surveyed section based on the point cloud curb information and the RGB curb information; And, if at least one of the confidence evaluation results of the RGB curb information and the confidence evaluation results of the point cloud curb information is not greater than a set threshold, it is determined that the current survey is unreliable and the survey is repeated.

8. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images