A vision-based traversable area detection and classification method and system

Abstract

The present disclosure provides a vision-based passable area detection and classification method and system, which relates to the field of computer vision. The method includes: dividing multiple target areas based on campus map data, obtaining first image data of the target area; performing first image processing on the first image data to obtain first training data corresponding to the first image data; performing second image processing on the first image data to obtain first test data; training a target neural network model based on the first training data and the first test data, optimizing parameters using optimization processing to obtain a first neural network model; performing semantic segmentation on images from the target area through the first neural network model to generate semantic segmentation images corresponding to the images of the target area; and determining the passable route of the target area based on the semantic segmentation images. The load of the neural network model on devices with limited computing power is saved, and the idle computing power can be used for obstacle avoidance walking and path planning.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a vision-based traversable area detection and classification method and system. Background Art

[0002] Nowadays, intelligent robots can be seen everywhere in our lives, and an example around us is the navigation robot in major libraries, which can avoid obstacles and do borrowing-related tasks in indoor environments. However, the use of intelligent robots on campus is not limited to indoor scenes. Intelligent robots should go out of the gate to complete more tasks, such as ensuring campus safety through automatic patrols, delivering medical supplies and medicines to solve emergencies on campus, and serving as an information terminal to help visiting tourists.

[0003] To achieve the above scenario, related technologies usually require the purchase of high-computing-power intelligent robots at high cost, and directly deploy the neural network model used for vehicle intelligent driving on the high-computing-power intelligent robots. Such high costs and computing power requirements limit the number of robots on campus and reduce the coverage of intelligent robots on campus. In addition, intelligent robots have limited power and can only operate in a limited area. They need to frequently return to designated locations for charging, wasting unnecessary costs and time.

[0004] For example, a Chinese patent with the authorization announcement number CN109840471B discloses a feasible road segmentation method based on an improved Unet network model, which includes: modifying the convolutional layer in the Unet training model to a ResNet residual module, using the cross entropy loss function and the Lovász hinge loss function for back propagation training respectively; using the trained model to perform feasible area segmentation on road scenes without segmentation labels; performing scale, contrast, and normalization processing on the training images and the corresponding segmentation mask annotations, and performing enhancement processing on the training samples, using the average histogram of the training data as a template, performing histogram matching processing on the test data, and then testing, and outputting the results after road segmentation. Without increasing the complexity of the model, the accuracy of road segmentation is greatly improved, and the real-time and accurate segmentation of the feasible area of ​​the road can basically be achieved.

[0005] At the same time, for example, a Chinese patent with authorization announcement number CN113569724B discloses a road extraction method and system based on attention mechanism and dilated convolution, including extracting road information of the target area based on remote sensing image data of the target area and a trained A&D-UNet aggregation network model; the provided trained A&D-UNet aggregation network model uses residual learning units to alleviate performance degradation problems caused by deep convolutional networks, uses dilated convolution units to integrate detailed information of road features, and uses convolutional attention modules to enhance the utilization of shallow information, thereby improving the accuracy of road information extraction in complex scenes by the trained A&D-UNet aggregation network model, and the degree of automation is high. The composite loss function combining the BCE loss function and the Dice loss function is used to train the A&D-UNet aggregation network model, which effectively alleviates the model instability problem caused by the imbalance of sample quantity and improves the model classification prediction ability.

[0006] The above methods all have the problems raised by this background technology. As mobile terminals, intelligent robots used in outdoor environments on campus have limited battery capacity, which also limits the computing power of the equipment. It is difficult to run network models with large parameters. In addition, the area of ​​universities is large, and dozens or hundreds of intelligent robots are needed to complete the comprehensive deployment on campus. How to achieve high-precision semantic segmentation and obstacle avoidance walking on campus while saving costs and limited computing power has become a problem to be solved. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention proposes a vision-based traversable area detection and classification method and system, which is used to realize road classification and obstacle avoidance walking in campus areas at a low cost on devices with limited computing power.

[0008] In a first aspect, an embodiment of the present disclosure provides a vision-based traversable area detection and classification method, comprising:

[0009] S1. Based on campus map data, divide a plurality of target areas and obtain first image data of the target areas; wherein the first image data is composed of a plurality of sub-image data;

[0010] S2. Performing first image processing on the first image data to obtain first training data corresponding to the first image data; wherein the first image processing includes: image annotation;

[0011] S3, performing second image processing on the first image data to obtain first test data;

[0012] S4, training a target neural network model based on the first training data and the first test data, optimizing parameters by using optimization processing, and obtaining a first neural network model;

[0013] S5. Performing semantic segmentation on the image from the target area by using the first neural network model to generate a semantic segmentation image corresponding to the image of the target area;

[0014] S6. Determine a route through the target area based on the semantic segmentation image.

[0015] As an optional implementation manner, the sub-image data includes: an RGB image of the target area obtained by shooting; performing first image processing on the first image data to obtain first training data corresponding to the first image data, including:

[0016] S21, determining target area label information required for marking the first image data;

[0017] S22, based on the label information of the target area, performing pixel-level image annotation on the first image data, so that each pixel in each sub-image data in the first image data has corresponding label information;

[0018] S23. Based on the label information corresponding to each pixel of each sub-image data in the first image data, determine the training set data and the annotation data corresponding to each image in the training set data to obtain the first training data corresponding to the first image data.

[0019] As an optional implementation manner, the label information of the target area includes nine types of labels: road, path, grass, building, people, sky, margin, tree, and background.

[0020] As an optional implementation manner, performing second image processing on the first image data to obtain first test data includes:

[0021] S31. Crop, rotate, mirror, intercept, add noise, and sharpen the first image data to obtain first test data, wherein the first test data has the same size format as the first training data.

[0022] As an optional implementation, the target neural network model is composed of a Unet neural network model and a MobileNetV3 neural network model; the target neural network model is trained based on the first training data and the first test data, and the parameters are optimized by optimization processing to obtain a first neural network model, including:

[0023] S41. Based on the first training data, perform semantic segmentation capability training of the target area through a Unet neural network model;

[0024] S42. During the training process of the Unet neural network model, the parameters of the MobileNetV3 neural network model are adjusted to generate a target neural network model. The function expression used to reduce operations in the MobileNetV3 neural network model is:

[0025]

[0026] in, is the activation function about x, where h stands for hard, which means converting the smooth, continuous, nonlinear form of the original swish function into a simple, piecewise, linear form; For each element value in the feature map of the first training data generated by the convolution layer of the target neural network model, ReLU6() is a maximum suppression function, and the maximum suppression function expression is:

[0027]

[0028] For each element value x in the feature map of the first training data, ReLU transforms x according to the above formula. When x<0, x becomes 0 after activation; when 0≤x≤6, x remains unchanged after activation; when x>6, x is limited to 6 after activation;

[0029] S43. Optimize the parameters of the target neural network model using optimization processing to obtain a first neural network model.

[0030] As an optional implementation, the optimization process is used to optimize the parameters of the target neural network model to obtain a first neural network model; wherein the function used in the optimization process includes: a first loss function, a second loss function and an mIOU indicator function, including:

[0031] S431, estimating the consistency between the predicted value of the target neural network model and the true value based on the first loss function; wherein the first loss function is a cross entropy loss function, and the expression is:

[0032]

[0033] in, represents the cross entropy loss function value, N represents the number of training images in the first training data used, Indicates The labeled data value of the pixel point corresponding to the training image, at least one type of label in the label information of the target area will be regarded as the first target label, and the first target label for calculating the cross entropy loss function value is equal to 1, the remaining labels in the label information of the target area is equal to 0; Indicates that the target neural network model is The predicted value of the pixel corresponding to the image; when the cross entropy loss function value When it is greater than a first threshold, a gradient descent method is used to reduce the loss value of the target neural network model;

[0034] S432. Calculate the sum of squares of the errors between the predicted value and the true value of the pixel point corresponding to the target neural network model based on the second loss function, wherein the second loss function is a mean square error loss function, and the expression is:

[0035]

[0036] in, represents the second loss function; when When it is greater than the second threshold, the gradient descent method is used to reduce the loss value of the target neural network model.

[0037] As an optional implementation, the function used in the optimization process further includes: an mIOU indicator function; the optimization process is used to optimize the parameters of the target neural network model to obtain the first neural network model, and further includes:

[0038] S433, based on the mIOU index, the target neural network model is evaluated as a whole, and the semantic segmentation accuracy corresponding to the label information of the target regions is generated, and the parameters of the target neural network model are adjusted. The mIOU index function is:

[0039]

[0040] Wherein, mIOU represents the evaluation result of the mIOU indicator function, k is the number of labels of the target area, k+1 is the total number of labels of the target area plus the background label, and the label types corresponding to i and j are at least one of the label information of the target area respectively; Indicates the probability of predicting label type i as label type j; represents the probability that the target neural network model predicts label type i, and the true label type of the first test data is also i; represents the probability that the target neural network model predicts label type j, while the true label type of the first test data is i; represents the difference between the predicted value of the target neural network model and the true value of the first test data;

[0041] S434. Based on the function values ​​of the first loss function and the second loss function, and based on the evaluation result of the mIOU index function, the parameters of the target neural network model are optimized to obtain the first neural network model.

[0042] As an optional implementation manner, performing semantic segmentation on the image from the target area by using the first neural network model to generate a semantic segmentation image corresponding to the image of the target area includes:

[0043] S51, acquiring an image of the target area;

[0044] S52. Perform semantic segmentation on the image of the target area through the first neural network model to generate a semantic segmentation image corresponding to the image of the target area.

[0045] As an optional implementation, the semantic segmentation image corresponding to the image of the target area includes: nine categories of semantic information, corresponding to nine categories of labels: road, path, grass, building, people, sky, margin, tree, and background; wherein the nine categories of semantic information are distinguished by nine categories of colors.

[0046] As an optional implementation manner, determining a route through the target area based on the semantic segmentation image includes:

[0047] S61, acquiring a semantic segmentation image of the target area in real time;

[0048] S62, based on the semantic segmentation of the target area, taking the image having semantic information of at least one of road, path, and grass as the first area, and taking the image having semantic information of at least one of building, people, sky, margin, tree, and background as the second area;

[0049] S63: Generate a travel route to the target area based on the first area and the second area.

[0050] As an optional implementation manner, the generating a route to the target area based on the first area and the second area further includes:

[0051] S631, determining a robot-traversable area based on the first area and the second area;

[0052] S632: Calculate the shortest path to the target area based on the robot's traversable area, and use the shortest path to the target area as the traversable route to the target area.

[0053] In a second aspect, the embodiments of the present disclosure further provide a vision-based traversable area detection and classification system including: an acquisition module, a processing module, a construction module, and a generation module; including: an acquisition module, a processing module, a construction module, and a generation module;

[0054] The acquisition module is used to divide a plurality of target areas based on the campus map data, and acquire first image data of the target areas; wherein the first image data is composed of a plurality of sub-image data;

[0055] The processing module is used to perform a first image processing on the first image data to obtain first training data corresponding to the first image data; wherein the first image processing includes: image annotation; performing a second image processing on the first image data to obtain first test data;

[0056] The construction module is used to train the target neural network model based on the first training data and the first test data, and optimize the parameters by using optimization processing to obtain the first neural network model;

[0057] The generation module is used to perform semantic segmentation on the image from the target area through the first neural network model to generate a semantic segmentation image corresponding to the image of the target area; based on the semantic segmentation image, determine the passage route of the target area.

[0058] Compared with the prior art, the beneficial effects of the present invention are: by analyzing the semantic information of the campus scene, the types of labels required for annotation are reduced, a pixel-level campus data set is self-made and enhanced by copying, cropping, mirroring, etc., so that low-complexity networks can also perform image classification of campus road scenes, reducing the load on the mobile terminal. By adding the attention mechanism MobileNetV3, the network segmentation effect is optimized, the information overload problem caused by the weak computing power of the mobile platform is alleviated, and a variety of lightweight loss functions and mIOU indicator evaluation functions are used to optimize the semantic segmentation effect of the neural network model, so that the detection and classification of the passable area is achieved at low cost and limited computing power. The above method saves the load of the neural network model on the device with limited computing power, and the idle computing power can be used for obstacle avoidance walking and path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0060] Figure 1 A schematic diagram of a flow chart of a vision-based traversable area detection and classification method provided in an embodiment of the present disclosure is shown;

[0061] Figure 2 A specific example of the evaluation result of an mIOU indicator function provided by an embodiment of the present disclosure is shown;

[0062] Figure 3 A schematic diagram of the structure of a vision-based traversable area detection and classification system provided in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0064] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0065] See also Figure 1-Figure 3 The present invention provides a disclosed embodiment: a vision-based traversable area detection and classification method, the method comprising steps S1 to S6, wherein:

[0066] S1. Based on campus map data, divide a plurality of target areas and obtain first image data of the target areas; wherein the first image data is composed of a plurality of sub-image data;

[0067] In a specific implementation, there may be many ways to divide the target area. In the embodiment of the present disclosure, according to the task type of the mobile intelligent robot used in the embodiment of the present disclosure, based on the map data of the campus, the target area is divided into the living area, teaching area, and landscape area of ​​the campus.

[0068] In a specific implementation, at least one of the above-mentioned different categories of regions can be regarded as a target region.

[0069] It should be noted that the target area can be divided in many ways, and can also be divided within a certain area of ​​the campus, for example: the target area can be divided into a river area, a vegetation area, an animal area, etc. within the landscape area of ​​the campus.

[0070] In an optional implementation, the sub-image data is an RGB image of the target area obtained by shooting, where "RGB" stands for red, green and blue, and these three colors of light are combined with different intensities to produce various colors.

[0071] S2. Performing first image processing on the first image data to obtain first training data corresponding to the first image data; wherein the first image processing includes: image annotation;

[0072] In this embodiment, the specific steps of S2 are as follows:

[0073] S21. Determine target area label information required for labeling the first image data according to the type of task to be performed in the embodiment of the present disclosure;

[0074] In a specific implementation, the types of labels required for the annotation of image data can be determined by the task being performed, and the optimization of the types of labels can reduce the complexity of the neural network model and the load occupied by the deployment on the device.

[0075] Exemplarily, the present disclosure takes a campus scene as an example, and labels can be set according to the main semantic information and rare semantic information in the campus. For example, if there is no semantic information such as billboards and plaques in the campus, these labels can be excluded when setting labels; the trash cans in the campus are all of the same category, that is, each trash can has the same semantic information, and the semantic type with such fixed features can be set in the semantic information of other categories in the label setting, such as classifying trash cans as buildings. Based on the above method and the scene type of the application, the type of label required for annotation can be determined.

[0076] In an optional implementation, the label information of the target area includes nine types of labels: road, path, grass, building, people, sky, margin, tree, and background;

[0077] In the specific implementation, the selection of labels will determine the applicability of the neural network, the training effect of the neural network model, and the task execution effect of the neural network model in the target scenario.

[0078] The disclosed embodiment provides a label type setting for annotating campus scene image data. In a campus scene, roads are usually divided into paved, wider roads for vehicles and pedestrians; narrower, unpaved or incompletely paved roads for pedestrians; and grass roads mainly composed of green plants. The characteristics of these three types of roads are quite different. Therefore, the disclosed embodiment marks these three as road, path, and grass labels respectively during annotation.

[0079] For stationary or slow-moving objects, the most common ones in campus scenes are the sky (including clouds), plants, buildings, road edges, and backgrounds. Among them, for the sky, the dataset can include a variety of sky states such as sunny, cloudy, rainy, snowy, and night; for plants, since the plant species and distribution on campus are more regular and the feature information is similar, compared to complex scenes such as cities, different types of trees, weeds, and flowers can be uniformly labeled as tree tags; for buildings, in target scenes similar to campus scenes, the styles of buildings are less different, the features are similar, and compared to urban scenes, they have lower light and shadow information, which reduces the interference of light and shadow, so all buildings on campus (including benches, trash cans, etc.) can be uniformly labeled as "building" tags; for road edges, this embodiment performs a separate classification because campus Most of the roads in the image are of the "road" type. The edges of this type of paved roads have stone bricks or railings with highly similar features, and usually the road edges and roads are connected, which is also very helpful for the training of the neural network model and the improvement of the recognition accuracy of the passable area. Therefore, the disclosed embodiment marks this type of road edge as "margin"; for the background, since there are still a few semantic information with complex features but only appearing occasionally on campus, this type of semantic information is marked as "background" in the disclosed embodiment. The advantage of this approach is that it avoids wasting computing power on rare information. At the same time, when this rare semantic information appears, it can also be regarded as an obstacle and avoided. For moving objects in the scene, such as vehicles and pedestrians, the embodiments of the present disclosure uniformly label them as "people". Since vehicles and pedestrians on campus are quite different from regularly and orderly arranged buildings, sky, plants, roads and other features, even if the two are not distinguished, the neural network model can identify them with high accuracy and avoid them as obstacles. Most vehicles on campus travel at low speeds, so it is feasible to avoid them even when they are near the terminal device where the neural network model is deployed. This labeling method can effectively reduce the load of the neural network model and save costs.

[0080] S22, based on the label information of the target area, performing pixel-level image annotation on the first image data, so that each pixel in each sub-image data in the first image data has corresponding label information;

[0081] For example, taking campus scenes as an example, labelme, a labeling software, can be used to annotate semantic information for each pixel in each picture in the data set composed of campus scene images. Irregular rectangles can be used to annotate the sky or grass in the picture with the required labels. For the labeling of irregular areas, multiple smaller segments can be added near each corner of the rectangle to gradually simulate the effect of rounded corners. This is actually simulating the process of curve interpolation. By subdividing more points near the corners and connecting these points with straight line segments, the corners can be made to look more "rounded" visually. Since unlabeled pixels will not participate in the subsequent neural network model training process, it is only necessary to annotate the semantic information of the pixels of each image data as much as possible through the above means.

[0082] S23. Based on the label information corresponding to each pixel of each sub-image data in the first image data, 70% of these data are used as training set data to obtain first training data corresponding to the first image data.

[0083] In the disclosed embodiment, after each image data is annotated, a corresponding JSON (JavaScript Object Notation) file will be generated. This is a lightweight data exchange format. This article does not limit the format of the annotation file, and the JSON file is just one of them. In the production of the first training data, part of the image data in the annotated campus image data set and the corresponding annotation file can be used as the first training data, i.e., the training set.

[0084] S3, performing second image processing on the first image data to obtain first test data;

[0085] In this embodiment, the specific steps of S3 are as follows: S31. 70% of the first image data is used as a training set, and the remaining 30% of the data will be cropped, rotated, mirrored, intercepted, noised, and sharpened to obtain the first test data. In order to keep the size format of the first test data the same as that of the first training data, for the case of inconsistent sizes, the embodiment of the present disclosure adopts a pixel filling method to fill the pixels at these positions with black, which corresponds to the "background" in the label type.

[0086] The embodiments of the present disclosure have expanded the test set data. Since the data set formed by the scenarios to which the embodiments of the present disclosure are applicable may be small in number, for example, the number of images in the data set is 5,000, most of which are usually used as training sets. However, since the data set itself is small, the data set formed by the remaining images may not be able to fully test the training effect of the neural network model. In this case, data expansion of the test set data can effectively increase the number of data in the test set.

[0087] S4, training a target neural network model based on the first training data and the first test data, optimizing parameters by using optimization processing, and obtaining a first neural network model;

[0088] In this embodiment, the specific steps of S4 are as follows:

[0089] S41. After the first training data for training, that is, the production of the training set, is completed, the training set for training can be called in the training of the UNet neural network model through a script file.

[0090] Among them, the UNet neural network is a deep learning model designed specifically for medical image segmentation. This network structure is named for its unique "U" shape. It is designed to effectively handle biomedical image segmentation tasks, especially when the number of samples is small. It can still achieve high-precision segmentation effects. The number of medical images is much smaller than the number of data sets used for other tasks, and it is also very difficult to obtain clinically significant labels. Therefore, the network model structure used in this field should not be too complex, and the fewer parameters, the better. The scenarios to which the disclosed embodiments are applicable exist in which the data sets are small and the UNet neural network has good results on small data sets.

[0091] S42. During the training process of the Unet neural network model, the parameters of the MobileNetV3 neural network model are adjusted to generate a target neural network model. The function expression used to reduce operations in the MobileNetV3 neural network model is:

[0092]

[0093] in, is the activation function about x, where h stands for hard, meaning that this version of the swish activation function is a simplified version that converts the smooth, continuous, nonlinear form of the original swish function into a simple, piecewise, linear form; For each element value in the feature map of the first training data generated by the convolution layer of the target neural network model, ReLU6() is a maximum suppression function, and the maximum suppression function expression is:

[0094]

[0095] For each element value x in the feature map of the first training data, ReLU6 can transform x according to the above formula. When x<0, x becomes 0 after activation; when 0≤x≤6, x remains unchanged after activation; when x>6, x is limited to 6 after activation; using When the element value in the feature map of the first training data is too large, it is limited to keep the target neural network model stable; when the element value in the feature map of the first training data is negative, it is set to 0 to reduce the number of activated neurons, reduce the load of the target neural network model, reduce the sensitivity of the target neural network model to the noise of the first training data, and reduce the overfitting rate.

[0096] In the disclosed embodiment, during the training process of the target neural network model, by performing the convolution operation included in the UNet neural network on the image data in the input training set data, a feature map corresponding to the image data can be obtained. For the element value x in the feature map, the element value that is too large will be limited to 6, the element value with a negative value will be set to 0, and the element value between 0 and 6 will remain unchanged. Among them, each element value in the feature map is the result of the input data (such as an image) being processed by a specific convolution kernel (or filter) and an activation function. The element value in the feature map represents the response intensity of the feature detected by the convolution kernel. The element value with a large numerical value represents that the input data and the convolution kernel match well at the corresponding position, that is, the feature at the position is similar to the feature represented by the convolution kernel.

[0097] Through the above method, the number of activated neurons in the fusion neural network of UNet and MobileNetV3 will be reduced. This is because the neurons will be activated regardless of whether the element value is a negative value less than 0 or a positive value greater than 0, while the neurons will not be activated when the element value is 0. This effectively reduces the number of activated neurons and reduces the complexity of the neural network model. The reduction in the complexity of the neural network model allows the neural network model to reduce the load of the device when it is deployed on an intelligent robot device with limited computing power to which the disclosed embodiment is applicable. The reduction in the complexity of the neural network model also limits the dependence of the neural network model on the noise features in the training set, alleviating the overfitting phenomenon.

[0098] S43. Optimize the parameters of the target neural network model using optimization processing to obtain a first neural network model.

[0099] In this embodiment, S43 can also be divided into the following steps:

[0100] S431, estimating the consistency between the predicted value of the target neural network model and the true value based on the first loss function; wherein the first loss function is a cross entropy loss function, and the expression is:

[0101]

[0102] in, represents the cross entropy loss function value, N represents the number of training images in the first training data used, Indicates The labeled data value of the pixel point corresponding to the training image, at least one type of label in the label information of the target area will be regarded as the first target label, and the first target label for calculating the cross entropy loss function value is equal to 1, the remaining labels in the label information of the target area is equal to 0; Indicates that the target neural network model is The predicted value of the pixel point corresponding to the image is a real value between 0 and 1. The closer the predicted value is to 1, the greater the probability that the target neural network model predicts that the current training image belongs to the first target label. When it is greater than a first threshold, a gradient descent method is used to reduce the loss value of the target neural network model;

[0103] In the specific implementation, the number of image data used for training is 70% of the total dataset. In the training process, "road", "path", and "grass" can be regarded as the first target label. is equal to 1, while the other tags is equal to 0. Indicates that the target neural network model is For example, if the neural network being trained predicts a pixel in the ith image as "road" = 0.9, then =0.9, =1, Substitute the number of N used for training into the function expression to find the cross entropy loss function value of the "road" label in this training , if you want to calculate the cross entropy loss function value of the "tree" label that does not belong to the first target label You can substitute = 0 and the probability that the neural network being trained predicts a pixel in the i-th image as "tree". The cross entropy loss function threshold set after comparison, for example 0.23, if the cross entropy loss value If the cross entropy loss value is greater than 0.23, the gradient descent method is used to reduce the loss function value. If it is less than or equal to 0.23, no further action is required.

[0104] S432. Calculate the sum of squares of the errors between the predicted value and the true value of the pixel point corresponding to the target neural network model based on the second loss function, wherein the second loss function is a mean square error loss function, and the expression is:

[0105]

[0106] in, represents the second loss function, N represents the number of training images in the first training data used, Indicates the first training data The annotation value of the pixel corresponding to the training image, Indicates that the target neural network model is The predicted value of the pixel corresponding to the image. and The difference is squared and the average is calculated. When it is greater than a second threshold, a gradient descent method is used to reduce the loss value of the target neural network model;

[0107] For example, the output of semantic segmentation is considered as a continuous value of the classification probability of each pixel. For each pixel, the model predicts a vector, and each element of the vector represents the probability (or score) of the pixel belonging to each category. The true label needs to be processed into the same format as the predicted output, that is, the category label of each pixel is converted into a probability distribution vector. For each pixel in the image, calculate its predicted probability vector With the true probability vector The mean square error loss function value between , and average the losses of all pixels to get the mean square error loss function value of the entire image . After getting the mean square error loss function value The mean square error loss function threshold set after comparison, for example 0.13, if the cross entropy loss value If the cross entropy loss value is greater than 0.13, the gradient descent method is used to reduce the loss function value. If it is less than or equal to 0.13, no further action is required.

[0108] S433, based on the mIOU index, the target neural network model is evaluated as a whole, and the semantic segmentation accuracy corresponding to the label information of the target regions is generated, and the parameters of the target neural network model are adjusted. The mIOU index function is:

[0109]

[0110] Wherein, mIOU represents the evaluation result of the mIOU indicator function, k is the number of labels of the target area, k+1 is the total number of labels of the target area plus the background label, and the label types corresponding to i and j are at least one of the label information of the target area respectively; Indicates the probability of predicting label type i as label type j; represents the probability that the target neural network model predicts label type i, and the true label type of the first test data is also i; represents the probability that the target neural network model predicts label type j, while the true label type of the first test data is i; Represents the difference between the predicted value of the target neural network model and the true value of the first test data; generates the prediction accuracy of the target neural network model for each type of label in the target area through the evaluation result of the mIOU indicator function;

[0111] For example, see Figure 2 , Figure 2 The figure shows a specific example of the evaluation result of a mIOU indicator function provided by the embodiment of the present disclosure. In the embodiment of the present disclosure, the mIOU indicator is usually calculated by accumulating the IOU of each category and then averaging it. The result obtained is the evaluation based on the overall model. The results of this experiment are shown in Figure 2. Figure 2 As shown in the figure, the accuracy of the people tag is slightly lower because there are fewer pedestrians in the entire dataset and the number of samples obtained is insufficient. The high recognition rate of the margin tag can help us use it as the area boundary to plan the robot's walking path in the subsequent traversable area planning. Class4: road, Class6: grass, and Class7: path are the traversable areas. Figure 2 It can be seen that each of them has achieved an accuracy of nearly 90%, which shows that the UNet network has a good recognition effect on this type of regular road environment.

[0112] S434. Based on the function values ​​of the first loss function and the second loss function, and based on the evaluation result of the mIOU index function, the parameters of the target neural network model are optimized to obtain the first neural network model.

[0113] Exemplarily, after calculating the cross entropy loss function value, the mean square error loss function value, and the mIOU index function value, these three values ​​can be compared with the expected function value of the target task to be performed. If the loss function value is too high, or the mIOU index function has too low recognition accuracy for a specific label category, the neural network model can be optimized by using gradient descent method, attention mechanism method, loss function adjustment method, etc. The present disclosure does not specifically limit the optimization method.

[0114] S5. Performing semantic segmentation on the image from the target area by using the first neural network model to generate a semantic segmentation image corresponding to the image of the target area;

[0115] In the embodiment of the present disclosure, the specific steps of S5 are:

[0116] S51, acquiring an image of the target area;

[0117] In the disclosed embodiments, the trained neural network model will be deployed on a mobile intelligent robot with limited computing power. The deployment method is based on the system of the intelligent robot. Taking the Android system as an example, the trained neural network model can be set in a folder and called through a script file when needed. During the movement of the intelligent robot, the front image can be captured through its own camera device.

[0118] S52. After the program receives the captured front image, the first neural network model is used to perform a semantic segmentation task on the image of the target area to generate a semantic segmentation image corresponding to the image of the target area, in which different colors correspond to different label categories.

[0119] In an optional implementation, the semantic segmentation image corresponding to the image of the target area includes: nine categories of semantic information, corresponding to nine categories of labels: road, path, grass, building, people, sky, margin, tree, and background; wherein the nine categories of semantic information are distinguished by nine categories of colors, respectively.

[0120] S6. Determine a route through the target area based on the semantic segmentation image.

[0121] In the embodiment of the present disclosure, the specific steps of S6 are:

[0122] S61, acquiring a semantic segmentation image of a target area from the deployed mobile intelligent robot in real time;

[0123] For example, due to the limited computing power of the device, video data with a resolution of 720P and 30 frames per second can be shot. After the video data is input into the neural network, a random algorithm is used to perform semantic segmentation on 10 random frames of images, generating 10 semantic segmentation images per second. These 10 semantic segmentation images will be used to perform classification tasks and path planning tasks.

[0124] S62, based on the semantic segmentation of the target area, taking the image having semantic information of at least one of road, path, and grass as the first area, and taking the image having semantic information of at least one of building, people, sky, margin, tree, and background as the second area;

[0125] For example, the mobility of each movable robot is different. Some robots use bionic limbs to move, so roads, paths, and grass can all be used as passable areas, while some robots use wheels to move, and may not be able to climb stairs or move on small paths, so it is necessary to determine passable areas and non-passable areas.

[0126] S63: Generate a travel route to the target area based on the first area and the second area.

[0127] In the embodiment of the present disclosure, regarding step S63, when generating the passage route of the target area based on the first area and the second area, it can also be divided into the following steps:

[0128] S631, determining a robot-traversable area based on the first area and the second area;

[0129] For example, following the above S62, if the intelligent robot only travels on grass, only the grass can be set as the passable area, and other areas are inaccessible or obstacles.

[0130] S632: Calculate the shortest path to the target area based on the robot's traversable area, and use the shortest path to the target area as the traversable route to the target area.

[0131] Exemplarily, after the traversable area of ​​the intelligent robot is determined, path finding and graph traversal algorithms can be used to calculate the shortest path. The embodiments of the present disclosure aim to propose a method for constructing a data set and a network model when a device with limited computing power performs traversable area detection and classification tasks in a target scene, so as to reduce device load and save costs. Therefore, the embodiments of the present disclosure do not limit specific path planning and mobile algorithms.

[0132] Based on the same inventive concept, the embodiment of the present disclosure also provides a system corresponding to a vision-based passable area detection and classification method. Since the principle of solving the problem by the system in the embodiment of the present disclosure is similar to the above-mentioned vision-based passable area detection and classification method in the embodiment of the present disclosure, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be repeated.

[0133] Reference Figure 3 , which is a schematic diagram of the structure of a vision-based traversable area detection and classification system provided in an embodiment of the present disclosure, the system includes: an acquisition module 10, a processing module 20, a construction module 30 and a generation module 40;

[0134] The acquisition module 10 is used to divide a plurality of target areas based on the campus map data, and acquire first image data of the target areas; wherein the first image data is composed of a plurality of sub-image data;

[0135] The processing module 20 is used to annotate the data set to obtain a training set corresponding to the data set; perform processing such as cropping, rotating, mirroring, intercepting, adding noise, sharpening, etc. on the data set to obtain a test data set;

[0136] A construction module 30 is used to train the fusion neural network of UNet and MobileNetV3 based on the training set data and the test set data, and optimize the parameters using the cross entropy loss function, the mean square error loss function, and the mIOU evaluation function to obtain a trained neural network model;

[0137] The generation module 40 is used to perform semantic segmentation on images from the campus through a trained neural network model to generate semantic segmentation images corresponding to the images on the campus; based on the semantic segmentation images corresponding to the images on the campus, a path planning algorithm is used to determine the travel route of the intelligent robot on the campus.

[0138] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0139] It should be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0140] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through a wired network or / and a wireless network. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0141] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0142] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the method and system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0143] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0144] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A vision-based traversable area detection and classification method, characterized in that: include: S1. Divide a plurality of target areas based on campus map data, and obtain first image data of the target areas; wherein the first image data is composed of a plurality of sub-image data; S2. Performing first image processing on the first image data to obtain first training data corresponding to the first image data; wherein the first image processing includes: image annotation; S3, performing second image processing on the first image data to obtain first test data; S4, training a target neural network model based on the first training data and the first test data, optimizing parameters by using optimization processing, and obtaining a first neural network model; S5. Performing semantic segmentation on the image from the target area by using the first neural network model to generate a semantic segmentation image corresponding to the image of the target area; S6. Determine a route through the target area based on the semantic segmentation image; The sub-image data includes: an RGB image of the target area obtained by shooting; performing first image processing on the first image data to obtain first training data corresponding to the first image data, including: S21, determining target area label information required for marking the first image data; S22, based on the label information of the target area, performing pixel-level image annotation on the first image data, so that each pixel in each sub-image data in the first image data has corresponding label information; S23, based on the label information corresponding to each pixel of each sub-image data in the first image data, determining the training set data and the annotation data corresponding to each image in the training set data, and obtaining the first training data corresponding to the first image data; The label information of the target area includes nine types of labels: road, path, grass, building, people, sky, margin, tree, and background; The performing second image processing on the first image data to obtain first test data includes: S31, cropping, rotating, mirroring, intercepting, adding noise, and sharpening the first image data to obtain first test data, wherein the first test data has the same size format as the first training data; The target neural network model is composed of a Unet neural network model and a MobileNetV3 neural network model; the target neural network model is trained based on the first training data and the first test data, and the parameters are optimized by optimization processing to obtain a first neural network model, including: S41. Based on the first training data, perform semantic segmentation capability training of the target area through a Unet neural network model; S42. During the training process of the Unet neural network model, the parameters of the MobileNetV3 neural network model are adjusted to generate a target neural network model. The function expression used to reduce operations in the MobileNetV3 neural network model is: ； in, is the activation function about x, where h stands for hard, which means converting the smooth, continuous, nonlinear form of the original swish function into a simple, piecewise, linear form; For each element value in the feature map of the first training data generated by the convolution layer of the target neural network model, ReLU6() is a maximum suppression function, and the maximum suppression function expression is: ； For each element value x in the feature map of the first training data, ReLU6 transforms x according to the above formula. When x<0, x becomes 0 after activation; when 0≤x≤6, x remains unchanged after activation; when x>6, x is limited to 6 after activation; S43. Optimize the parameters of the target neural network model using optimization processing to obtain a first neural network model.

2. The method for detecting and classifying traversable areas based on vision according to claim 1, characterized in that: The functions used in the optimization process include: a first loss function and a second loss function. The optimization process is used to optimize the parameters of the target neural network model to obtain the first neural network model, including: S431, estimating the consistency between the predicted value of the target neural network model and the true value based on the first loss function; wherein the first loss function is a cross entropy loss function, and the expression is: ； in, represents the cross entropy loss function value, N represents the number of training images in the first training data used, Indicates The labeled data value of the pixel point corresponding to the training image, at least one type of label in the label information of the target area will be regarded as the first target label, and the first target label for calculating the cross entropy loss function value is equal to 1, the remaining labels in the label information of the target area is equal to 0; Indicates that the target neural network model is The predicted value of the pixel corresponding to the image; when the cross entropy loss function value When it is greater than a first threshold, a gradient descent method is used to reduce the loss value of the target neural network model; S432. Calculate the sum of squares of the errors between the predicted value and the true value of the pixel point corresponding to the target neural network model based on the second loss function, wherein the second loss function is a mean square error loss function, and the expression is: ； in, represents the second loss function; when When it is greater than the second threshold, the gradient descent method is used to reduce the loss value of the target neural network model.

3. A visual-based traversable area detection and classification method according to claim 2, characterized in that The function used in the optimization process also includes: mIOU indicator function; the optimization process is used to optimize the parameters of the target neural network model to obtain the first neural network model, and further includes: S433, based on the mIOU index, the target neural network model is evaluated as a whole, and the semantic segmentation accuracy corresponding to the label information of the target regions is generated, and the parameters of the target neural network model are adjusted. The mIOU index function is: ； Wherein, mIOU represents the evaluation result of the mIOU indicator function, k is the number of labels of the target area, k+1 is the total number of labels of the target area plus the background label, and the label types corresponding to i and j are at least one of the label information of the target area respectively; Indicates the probability of predicting label type i as label type j; represents the probability that the target neural network model predicts label type i, and the true label type of the first test data is also i; represents the probability that the target neural network model predicts label type j, while the true label type of the first test data is i; represents the difference between the predicted value of the target neural network model and the true value of the first test data; S434. Based on the function values ​​of the first loss function and the second loss function, and based on the evaluation result of the mIOU index function, the parameters of the target neural network model are optimized to obtain the first neural network model.

4. The method for detecting and classifying traversable areas based on vision according to claim 3, characterized in that: The performing semantic segmentation on the image from the target area by using the first neural network model to generate a semantic segmentation image corresponding to the image of the target area includes: S51, acquiring an image of the target area; S52. Perform semantic segmentation on the image of the target area through the first neural network model to generate a semantic segmentation image corresponding to the image of the target area.

5. The method for detecting and classifying traversable areas based on vision according to claim 4, characterized in that: The semantic segmentation image corresponding to the image of the target area includes: nine types of semantic information, corresponding to nine types of labels: road, path, grass, building, people, sky, margin, tree, and background; wherein the nine types of semantic information are distinguished by nine types of colors.

6. The method for detecting and classifying traversable areas based on vision according to claim 5, characterized in that: Determining a route through the target area based on the semantic segmentation image includes: S61, acquiring a semantic segmentation image of the target area in real time; S62, based on the semantic segmentation of the target area, taking the image in which the semantic information is at least one of road, path, and grass as the first area, and taking the image in which the semantic information is at least one of building, people, sky, margin, tree, and background as the second area; S63: Generate a travel route to the target area based on the first area and the second area.

7. The method for detecting and classifying traversable areas based on vision according to claim 6, characterized in that: The generating a route to the target area based on the first area and the second area further includes: S631, determining a robot-traversable area based on the first area and the second area; S632: Calculate the shortest path to the target area based on the robot's traversable area, and use the shortest path to the target area as the traversable route to the target area.

8. A vision-based traversable area detection and classification system, characterized in that: include: Acquisition module, processing module, construction module and generation module; The acquisition module is used to divide a plurality of target areas based on the campus map data, and acquire first image data of the target areas; wherein the first image data is composed of a plurality of sub-image data; The processing module is used to perform a first image processing on the first image data to obtain first training data corresponding to the first image data; wherein the first image processing includes: image annotation; performing a second image processing on the first image data to obtain first test data; The construction module is used to train the target neural network model based on the first training data and the first test data, and optimize the parameters by using optimization processing to obtain the first neural network model; The generating module is used to perform semantic segmentation on the image from the target area through the first neural network model to generate a semantic segmentation image corresponding to the image of the target area; and determine a passage route to the target area based on the semantic segmentation image; The sub-image data includes: an RGB image of the target area obtained by shooting; performing first image processing on the first image data to obtain first training data corresponding to the first image data, including: Determining target area label information required for marking the first image data; Based on the label information of the target area, performing pixel-level image annotation on the first image data, so that each pixel in each sub-image data in the first image data has corresponding label information; Based on the label information corresponding to each pixel of each sub-image data in the first image data, determine the training set data and the annotation data corresponding to each image in the training set data to obtain the first training data corresponding to the first image data; The label information of the target area includes nine types of labels: road, path, grass, building, people, sky, margin, tree, and background; The performing second image processing on the first image data to obtain first test data includes: The first image data is cropped, rotated, mirrored, intercepted, noised, and sharpened to obtain first test data, wherein the first test data has the same size format as the first training data; The target neural network model is composed of a Unet neural network model and a MobileNetV3 neural network model; the target neural network model is trained based on the first training data and the first test data, and the parameters are optimized by optimization processing to obtain a first neural network model, including: Based on the first training data, the semantic segmentation capability of the target area is trained by using the Unet neural network model; During the training process of the Unet neural network model, the parameters of the MobileNetV3 neural network model are adjusted to generate the target neural network model. The function expression used to reduce operations in the MobileNetV3 neural network model is: ； in, is the activation function about x, where h stands for hard, which means converting the smooth, continuous, nonlinear form of the original swish function into a simple, piecewise, linear form; For each element value in the feature map of the first training data generated by the convolution layer of the target neural network model, ReLU6() is a maximum suppression function, and the maximum suppression function expression is: ； For each element value x in the feature map of the first training data, ReLU6 transforms x according to the above formula. When x<0, x becomes 0 after activation; when 0≤x≤6, x remains unchanged after activation; when x>6, x is limited to 6 after activation; The parameters of the target neural network model are optimized by using optimization processing to obtain a first neural network model.

Images