A method for visual image path segmentation

Abstract

The present invention provides a method for segmenting a visual image path, including: Step S1, preprocessing the original visual image to generate a preprocessed image; Step S2, calculating a segmentation threshold for the preprocessed image through a neural network, then updating the segmentation threshold as the used threshold and starting the count of N; Step S3, determining whether N is greater than K and determining whether the calculation of the preset algorithm fails: if any of the two determinations is yes, return to Step S2, update to the current used threshold and then enter Step S4; if both determinations are no, enter Step S4; Step S4, operating on the preprocessed image to generate a segmentation result; Step S5, evaluating the segmentation result and determining: if the evaluation is qualified, output the segmentation result; if the evaluation is not satisfied, invalidate the current used threshold, update the optimal threshold as the used threshold and then enter Step S4. The method for segmenting a visual image path of the present invention has high calculation accuracy and fast calculation speed.

Description

Technical Field

[0001] The present invention relates to the technical field of visual image technology, and in particular, to a method for segmenting a visual image path. Background Art

[0002] In recent years, the application of visual image technology has become more and more extensive. Among them, path segmentation of visual image technology is a kind of technology for extracting path regions in images. Robots can obtain corresponding path directions, path curvatures, etc. based on the extracted path information, and perform behavior planning according to the obtained information. Therefore, path segmentation is an important method for visual image technology.

[0003] The path segmentation methods of related technologies are generally implemented based on the flood filling algorithm. However, due to the huge amount of information contained in visual images, it is difficult for vision-based path segmentation algorithms to balance speed and accuracy, have poor environmental robustness, and are vulnerable to environmental interference. The flood filling algorithm requires manual calibration of the threshold of the flood filling algorithm. If the environment changes greatly, the effect of segmentation using the original threshold will be greatly reduced. Therefore, how to make the path segmentation method automatically adapt to different environments, further improve the robustness of the algorithm, and ensure high calculation accuracy and fast calculation speed is a technical problem to be solved. Summary of the Invention

[0004] In view of the above deficiencies of the prior art, the present invention proposes a method for segmenting a visual image path with high calculation accuracy and fast calculation speed.

[0005] To solve the above technical problems, an embodiment of the present invention provides a method for segmenting a visual image path, which is applied to a neural network. The method includes the following steps:

[0006] Step S1, obtain M frames of original visual images, preprocess the original visual images to generate preprocessed images, so as to eliminate the hard light influence caused by light in the original visual images; M is a positive integer;

[0007] Step S2, calculate a segmentation threshold and a road estimate value from the preprocessed images through the neural network, then update the calculated segmentation threshold as the used threshold, and store the used threshold before updating as the optimal threshold, and start counting N; the counting rule is: when updating the used threshold, N is set to 0; whenever a frame of the preprocessed image is obtained, N is incremented by 1; N is a natural number, and M > N;

[0008] Step S3, determine whether the current N corresponding to the preprocessed image obtained in step S1 is greater than a preset K, and determine whether the preset algorithm calculation in step S2 fails:

[0009] If any one of the two judgments is yes, return to step S2, update it to the current usage threshold, and then enter step S4; if both judgments are no, enter step S4;

[0010] Step S4: Perform operations on the currently obtained preprocessed image of step S1 according to a preset flood filling algorithm to generate a segmentation result;

[0011] Step S5: Evaluate the quality of the segmentation result of step S4 according to a preset evaluation rule and judge:

[0012] If the quality evaluation of the segmentation result is qualified, output the segmentation result; if the quality evaluation of the segmentation result does not meet the requirements of the evaluation rule, invalidate the current usage threshold, update the optimal threshold of step S2 to the usage threshold, and then enter step S4.

[0013] Preferably, in step S1, the preprocessing is to convert the original visual image into an HSV color space composed of a hue channel, a saturation channel, and a value channel, and then calculate the saturation channel and the value channel respectively through multiple Gaussian kernels to generate the preprocessed image.

[0014] Preferably, the size values of the multiple Gaussian kernels are different.

[0015] Preferably, in step S2, step S2 further includes:

[0016] Step S21: Perform pooling processing and convolutional processing on the currently obtained preprocessed image of step S1 in sequence to generate a first output result;

[0017] Step S22: Perform pooling processing, convolutional processing, and attention layer processing on the currently obtained preprocessed image of step S1 in sequence to generate a second output result;

[0018] Step S23: Combine the first output result generated by step S21 and the second output result generated by step S22 to generate a third output result;

[0019] Step S24: Perform pooling processing and fully connected layer processing on the third output result in sequence to generate a fourth output result; the fourth output result is 7-dimensional data; among them, the first 6 dimensions of the 7-dimensional data are the segmentation thresholds of the preset flood filling algorithm in step S4, and the 7th dimension is the road estimation value.

[0020] Preferably, in the step S2, the neural network realizes the calculation of the preprocessed image through a loss function, where the parameters of the loss function include the predicted value output by the neural network, the true threshold, the error weight of the data, and the numerical error; wherein, the predicted value output by the neural network is P, the true threshold is R, the error weight of the data is W, and the numerical error is L1; the following formula is satisfied:

[0021]

[0022] wherein, P = [p1, p2, p3, p4, p5, p6, p7] (2);

[0023] R = [r1, r2, r3, r4, r5, r6, r7] (3);

[0024] W = [w1, w2, w3, w4, w5, w6, w7] (4);

[0025] p1, p2, p3, p4, p5, p6, p7 are respectively the data of dimensions 1 to 7 in the predicted value output by the neural network, r1, r2, r3, r4, r5, r6, r7 are respectively the data of dimensions 1 to 7 in the true threshold, and w1, w2, w3, w4, w5, w6, w7 are respectively the data of dimensions 1 to 7 in the error weight of the data.

[0026] Preferably, in the step S4, the binary segmentation results respectively output by the flood filling algorithm through the predicted value and the true threshold output by the neural network are M p and M r ; in the flood filling algorithm, the exclusive OR operation is performed on two binary images to generate an exclusive OR result, and the number of white pixels in the exclusive OR result is counted to obtain the number of error pixels; the number of error pixels is S e , the pixel error is L2, and the following formula is satisfied:

[0027] S e = count(M r XOR M p )(5);

[0028]

[0029] wherein, count() is the function for calculating the number of error pixels; XOR is the exclusive OR operation symbol; S r is the number of white pixels in the binary segmentation result output by the flood filling algorithm through the true threshold, that is, the true area of the path.

[0030] Preferably, in the step S4, the total error of the segmentation result is L, and it satisfies the following formula:

[0031] L = L1 * (1 + L2) (7).

[0032] Preferably, the step S4 includes:

[0033] Step S41: Determine the region of interest in the preprocessed image based on the road estimated value in the step S2;

[0034] Step S42: Initialize the preset seed points and obtain the initial positions of the seed points;

[0035] Step S43: Perform operations on the preprocessed image according to the preset flood filling algorithm based on the seed points and the current usage threshold to generate the segmentation result;

[0036] Step S44: Score the segmentation result according to the preset scoring rule to generate a scoring result, and judge the scoring result:

[0037] If the scoring result is qualified, output the segmentation result;

[0038] If the scoring result is unqualified, perform iterative calculation on the seed points to generate new seed points, and then enter the step 43 after obtaining the initial positions of the new seed points.

[0039] Compared with the related technologies, the visual image path segmentation method of the present invention includes the following steps: Step S1, preprocess the original visual image to generate a preprocessed image; Step S2, calculate a segmentation threshold from the preprocessed image through a neural network, then update the segmentation threshold as the used threshold, and start counting N; Step S3, determine whether N is greater than K and whether the calculation by the preset algorithm fails. If either of the two determinations is yes, return to Step S2, update it to the current used threshold, and then enter Step S4. If both determinations are no, enter Step S4; Step S4, perform operations on the preprocessed image to generate a segmentation result; Step S5, evaluate the segmentation result and make a judgment. If the evaluation is qualified, output the segmentation result. If the evaluation is not satisfied, invalidate the current used threshold, update the optimal threshold as the used threshold, and then enter Step S4. By implementing Step S1, Gaussian kernel blur processing is performed on the input original visual image to soften the brightness changes in the image and increase its illumination robustness. By implementing Steps S2 to S5, for the contradiction between the computing power and accuracy of the segmentation method, the path segmentation performance is greatly improved while sacrificing a small amount of accuracy. The neural network and traditional machine vision algorithms are combined, and the recognition link is realized through the traditional machine vision algorithm, which reduces the requirement for high-computing-power devices brought by completely relying on neural network segmentation to a certain extent and greatly reduces the amount of calculation. As a result, the visual image path segmentation method of the present invention has high calculation accuracy and fast calculation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention will be described in detail below with reference to the accompanying drawings. Through the detailed description in combination with the following drawings, the above or other aspects of the present invention will become clearer and easier to understand. In the drawings:

[0041] Figure 1 is a flowchart of the visual image path segmentation method of the present invention;

[0042] Figure 2 is a flowchart of Step S2 in the visual image path segmentation method of the present invention;

[0043] Figure 3 is a flowchart of Step S4 in the visual image path segmentation method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0045] The specific embodiments / Examples described herein are specific embodiments of the present invention, which are used to illustrate the concept of the present invention. They are all explanatory and exemplary, and should not be construed as limiting the embodiments of the present invention and the scope of the present invention. Except for the embodiments described herein, those skilled in the art can also adopt other obvious technical solutions based on the content disclosed in the claims and the specification of this application. These technical solutions include technical solutions that make any obvious substitutions and modifications to the embodiments described herein, and all fall within the protection scope of the present invention.

[0046] The present invention provides a method for segmenting a visual image path. The method for segmenting a visual image path is applied to a neural network. The neural network refers to a computational model that mimics the structure and function of a biological neural network and can be trained through deep learning to enable the model to achieve a predetermined effect.

[0047] Please refer to Figure 1 as shown in Figure 1 which is a flowchart of the method for segmenting a visual image path of the present invention.

[0048] The method for segmenting a visual image path includes the following steps:

[0049] Step S1: Obtain M frames of original visual images, preprocess the original visual images to generate preprocessed images, so that the preprocessed images can eliminate the hard light influence caused by light in the original visual images.

[0050] M is a positive integer.

[0051] In step S1, the preprocessing is to convert the original visual images into an HSV color space composed of a hue channel (), a saturation channel, and a value channel, and then calculate the saturation channel and the value channel respectively through multiple Gaussian kernels to generate the preprocessed images.

[0052] In this embodiment, the size values of multiple Gaussian kernels are different. By implementing step S1, the preprocessing of the images is adopted, the color space is converted into the HSV color space, and different-sized Gaussian kernels are used to blur the saturation channel and the value channel respectively, softening the brightness changes in the images and increasing their light robustness, thereby reducing the influence of light to a certain extent.

[0053] Step S2: Calculate a segmentation threshold and a road estimation value for the preprocessed images through the neural network, then update the calculated segmentation threshold as the usage threshold, and store the usage threshold before updating as the optimal threshold, and start counting N.

[0054] The rule for counting is as follows: when updating the usage threshold, N is set to 0; whenever a frame of the preprocessed image is obtained, N is incremented by 1; N is a natural number, and M > N.

[0055] Please refer to Figure 2 as shown in Figure 2 which is a flowchart of step S2 in the visual image path segmentation method of the present invention. Specifically, step S2 further includes:

[0056] Step S21: The currently obtained preprocessed image of step S1 is sequentially subjected to pooling processing and convolutional processing to generate a first output result.

[0057] Step S22: The currently obtained preprocessed image of step S1 is sequentially subjected to pooling processing, convolutional processing, and attention layer processing to generate a second output result.

[0058] Step S23: The first output result generated in step S21 and the second output result generated in step S22 are combined to generate a third output result.

[0059] Step S24: The third output result is sequentially subjected to pooling processing and fully connected layer processing to generate a fourth output result.

[0060] The fourth output result is 7-dimensional data. Among them, the first 6 dimensions of the 7-dimensional data are the segmentation thresholds of the seeded region growing algorithm preset in step S4, and the 7th dimension is the road estimation value.

[0061] In step S2, the neural network calculates the preprocessed image through a loss function, where the parameters of the loss function include the predicted value output by the neural network, the true threshold, the error weight of the data, and the numerical error; among them, the predicted value output by the neural network is P, the true threshold is R, the error weight of the data is W, and the numerical error is L1; the following formula is satisfied:

[0062]

[0063] where, P = [p1, p2, p3, p4, p5, p6, p7] (2);

[0064] R = [r1, r2, r3, r4, r5, r6, r7] (3);

[0065] W = [w1, w2, w3, w4, w5, w6, w7] (4);

[0066] p1, p2, p3, p4, p5, p6, and p7 are the data of dimensions 1 to 7 in the predicted values output by the neural network, r1, r2, r3, r4, r5, r6, and r7 are the data of dimensions 1 to 7 in the true thresholds, and w1, w2, w3, w4, w5, w6, and w7 are the data of dimensions 1 to 7 in the error weights of the data.

[0067] Step S3: Determine whether the obtained N corresponding to the preprocessed image in Step S1 is greater than a preset K, and determine whether the calculation of the preset algorithm in Step S2 fails:

[0068] If either of the two determinations is yes, return to Step S2, update to the current usage threshold, and then enter Step S4;

[0069] If both determinations are no, enter Step S4.

[0070] Step S4: Perform an operation on the currently obtained preprocessed image in Step S1 according to a preset flood fill algorithm to generate a segmentation result.

[0071] The flood fill algorithm is a commonly used processing method in machine vision. This method starts from a "seed point" and searches for surrounding qualified pixel points with a specific threshold to achieve the function of finding similar pixels. In this embodiment, the preset flood fill algorithm is based on the original path segmentation method implemented by the flood fill algorithm in the visual image path segmentation method of the present invention, adding the neural network. The neural network model is trained through deep learning, and the network model is used to automatically generate thresholds to achieve automatic adaptation to different environments and further improve the robustness of the algorithm.

[0072] Please refer to Figure 3 as shown in Figure 3 is the flow chart of Step S4 in the visual image path segmentation method of the present invention. Specifically, Step S4 includes:

[0073] Step S41: Determine the region of interest (ROI) in the preprocessed image through the road estimate value in Step S2. The region of interest (ROI) is used for subsequent calculations, and all subsequent calculations are based on the region of interest.

[0074] Step S42: Initialize the preset seed point and obtain the initial position of the seed point.

[0075] Step S43: Perform an operation on the preprocessed image according to the preset flood fill algorithm based on the seed point and the current usage threshold to generate the segmentation result.

[0076] Step S44: Score the segmentation result according to a preset scoring rule to generate a scoring result, and judge the scoring result:

[0077] If the scoring result is qualified, output the segmentation result;

[0078] If the scoring result is unqualified, perform iterative calculation on the seed points to generate new seed points, and then enter step 43 after obtaining the initial positions of the new seed points.

[0079] In implementing step S4, the flood filling algorithm is adopted for path segmentation. By adding the method of seed point iteration, this algorithm improves the flood filling algorithm of the related technology into a new path segmentation method. The flood filling algorithm addresses the contradiction between the computing power and accuracy of the segmentation method, and greatly improves the path segmentation performance while sacrificing a small amount of accuracy.

[0080] Step S5: Evaluate and judge the quality of the segmentation result in step S4 according to a preset evaluation rule:

[0081] If the quality evaluation of the segmentation result is qualified, output the segmentation result;

[0082] If the quality evaluation of the segmentation result does not meet the requirements of the evaluation rule, invalidate the current usage threshold, update the optimal threshold in step S2 to the usage threshold, and then enter step S4. Using the optimal threshold used last time is conducive to ensuring the quality of the segmentation result.

[0083] In step S4, the binarized segmentation results respectively output by the flood filling algorithm through the predicted value output by the neural network and the true threshold are M p and M r ; in the flood filling algorithm, perform exclusive OR operation on the two binarized images to generate an exclusive OR result, count the number of white pixels in the exclusive OR result to obtain the number of error pixels; the number of error pixels is S e , the pixel error is L2, and it satisfies the following formula:

[0084] S e = count(M r XOR M p )(5);

[0085]

[0086] where count() is the function for calculating the number of error pixels; XOR is the exclusive OR operation symbol; S ris the number of white pixels in the binary segmentation result output by the flood filling algorithm through the true threshold, that is, the true area of the path.

[0087] The total error of the segmentation result is L and satisfies the following formula:

[0088] L = L1 * (1 - L2) (7).

[0089] In this embodiment, through the implementation of the steps S1 to S5, the neural network adopted is a relatively lightweight attention-convolution neural network, and a lightweight neural network model composed of a self-attention layer, a convolutional layer, a pooling layer, and a fully connected layer is constructed. The input of the model is an HSV image with a resolution of 640×480, and the output of the model is the threshold required by the flood filling algorithm and the estimated road width value. In terms of constructing the loss function in the flood filling algorithm, a method combining numerical difference and pixel difference is used. The present invention outputs the required threshold through this neural network to achieve environmental adaptability. More preferably, the neural network and the traditional machine vision algorithm are combined, and the recognition link is realized through the traditional machine vision algorithm, which reduces to a certain extent the requirement for high-computing power devices brought by completely relying on neural network segmentation and greatly reduces the amount of calculation.

[0090] In summary, the visual image path segmentation method of the present invention aims at the problem of threshold determination caused by environmental changes and the contradiction between speed and accuracy existing in the path segmentation method. It can effectively achieve a good balance between speed and accuracy. By combining the neural network and the traditional machine vision algorithm, it reduces the requirement for computing power of the path segmentation algorithm and provides a reliable solution for the lightweight and low-cost deployment of robots.

[0091] Compared with the related technologies, the visual image path segmentation method of the present invention includes the following steps: Step S1, preprocess the original visual image to generate a preprocessed image; Step S2, calculate the segmentation threshold from the preprocessed image through a neural network, then update the segmentation threshold as the used threshold, and start counting N; Step S3, determine whether N is greater than K and determine whether the calculation of the preset algorithm fails: if any of the two determinations is yes, return to Step S2, update it as the current used threshold, and then enter Step S4; if both determinations are no, enter Step S4; Step S4, operate on the preprocessed image to generate a segmentation result; Step S5, evaluate the segmentation result and make a judgment: if the evaluation is qualified, output the segmentation result; if the evaluation is not satisfied, invalidate the current used threshold, update the optimal threshold as the used threshold, and then enter Step S4. By implementing Step S1, Gaussian kernel blurring is performed on the input original visual image to soften the brightness change in the image and increase its illumination robustness; by implementing Steps S2 to S5, for the contradiction between the computing power and accuracy of the segmentation method, the path segmentation performance is greatly improved while sacrificing less accuracy; and the neural network and traditional machine vision algorithms are combined, and the recognition link is realized through the traditional machine vision algorithm, which reduces the requirement for high-computing-power devices brought by relying entirely on neural network segmentation to a certain extent and greatly reduces the amount of calculation. Therefore, the visual image path segmentation method of the present invention has high calculation accuracy and fast calculation speed.

Claims

1. A method for visual image path segmentation, which is applied to a neural network, and is characterized in that, The method includes the following steps: Step S1: Obtain M frames of original visual images, preprocess the original visual images to generate preprocessed images, so as to eliminate the hard light influence brought by light in the original visual images in the preprocessed images; M is a positive integer; Step S2: Calculate a segmentation threshold and a road estimation value from the preprocessed images through the neural network, then update the calculated segmentation threshold as the usage threshold, store the usage threshold before the update as the optimal threshold, and start counting N; The counting rule is: when updating the usage threshold, N is set to 0; whenever a frame of the preprocessed image is obtained, N is incremented by 1; N is a natural number, and M > N; Step S3: Determine whether the N corresponding to the preprocessed image obtained in step S1 is greater than a preset K, and determine whether the calculation of the preset algorithm in step S2 fails: If either of the two judgments is yes, return to step S2, update to the current usage threshold, and then enter step S4; if both judgments are no, enter step S4; Step S4: Perform an operation on the preprocessed image obtained in step S1 according to a preset flood filling algorithm to generate a segmentation result; Step S5: Evaluate and judge the quality of the segmentation result in step S4 according to a preset evaluation rule: If the quality evaluation of the segmentation result is qualified, output the segmentation result; if the quality evaluation of the segmentation result does not meet the requirements of the evaluation rule, invalidate the current usage threshold, update the optimal threshold in step S2 to the usage threshold, and then enter step S4; In step S2, the neural network calculates the preprocessed image through a loss function, where the parameters of the loss function include the predicted value output by the neural network, the true threshold, the error weight of the data, and the numerical error; among them, the predicted value output by the neural network is P, the true threshold is R, the error weight of the data is W, and the numerical error is L1; the following formula is satisfied: where, P = [p1, p2, p3, p4, p5, p6, p7] (2); R＝[r1,r2,r3,r4,r5,r6,r7] (3)； W = [w1, w2, w3, w4, w5, w6, w7] (4); p1, p2, p3, p4, p5, p6, p7 are the data of dimensions 1 to 7 in the predicted value output by the neural network respectively, r1, r2, r3, r4, r5, r6, r7 are the data of dimensions 1 to 7 in the true threshold respectively, w1, w2, w3, w4, w5, w6, w7 are the data of dimensions 1 to 7 in the error weight of the data respectively; In the step S4, the binarized segmentation results respectively output by the flood filling algorithm through the predicted value output by the neural network and the true threshold are M p and M r ; in the flood filling algorithm, an exclusive OR operation is performed on the two binarized images to generate an exclusive OR result, and the number of white pixels in the exclusive OR result is counted to obtain the number of error pixels; the number of error pixels is S e , the pixel error is L2, and the following formula is satisfied: S e = count(M r XOR M p ) (5); Among them, count() is the error pixel number operation function; XOR is the exclusive OR operation symbol; S r is the number of white pixels in the binary segmentation result output by the flood filling algorithm through the true threshold, that is, the true area of the path.

2. The visual image path segmentation method according to claim 1, wherein In step S1, the preprocessing is to convert the original visual image into an HSV color space composed of a hue channel, a saturation channel, and a lightness channel, and then calculate the saturation channel and the lightness channel respectively through multiple Gaussian kernels to generate the preprocessed image.

3. The visual image path segmentation method according to claim 2, characterized in that, The size values of the multiple Gaussian kernels are different.

4. The visual image path segmentation method according to claim 1, wherein In step S2, step S2 further includes: Step S21: The preprocessed image obtained in the current step S1 is sequentially subjected to pooling processing and convolutional processing to generate a first output result; Step S22: The preprocessed image obtained in the current step S1 is sequentially subjected to pooling processing, convolutional processing, and attention layer processing to generate a second output result; Step S23: The first output result generated in the step S21 and the second output result generated in the step S22 are combined to generate a third output result; Step S24: The third output result is sequentially subjected to pooling processing and fully connected layer processing to generate a fourth output result; the fourth output result is 7-dimensional data; among them, the first 6 dimensions of the 7-dimensional data are the segmentation thresholds of the preset flood filling algorithm in the step S4, and the 7th dimension is the road estimation value.

5. The visual image path segmentation method according to claim 1, wherein In the step S4, the total error of the segmentation result is L, and it satisfies the following formula: L = L1 * (1 - L2) (7).

6. The visual image path segmentation method according to claim 1, characterized in that The step S4 includes: Step S41: Determine the region of interest in the preprocessed image through the road estimation value in the step S2; Step S42: Initialize the preset seed points and obtain the initial positions of the seed points; Step S43: Perform operations on the preprocessed image according to the preset flood filling algorithm based on the seed points and the current usage threshold to generate the segmentation result; Step S44: Score the segmentation result according to the preset scoring rule to generate a scoring result, and judge the scoring result: If the scoring result is qualified, output the segmentation result; If the scoring result is unqualified, perform iterative calculation on the seed points to generate new seed points, and then enter the step 43 after obtaining the initial positions of the new seed points.

Images