Real-time monitoring method for high-speed mobile robots based on optical flow and convolutional neural network

Abstract

The present invention discloses a real-time monitoring method for a high-speed mobile robot based on optical flow and convolutional neural network, and relates to the field of robot visual monitoring technology. The present invention obtains rough information of a dynamic mobile robot through an optical flow algorithm from a collected image, and then obtains a binarized image of the mobile robot by using mean filter fuzzification processing, morphological operation, and adaptive threshold binarization processing, and finally uses a convolutional neural network to detect the position information of the high-speed mobile robot. Compared with the existing detection algorithm, the present algorithm is real-time, and can quickly detect the position of the robot after completing the training of the neural network model; it does not require the establishment of an environmental model, and most existing algorithms require a detailed environmental model, which is difficult to build in practice; it is low-cost, and only requires a computer and an image acquisition device, and does not require the operation of professionals.

Description

Technical Field

[0001] The present invention relates to the technical field of robot vision monitoring, and in particular to a real-time monitoring method for a high-speed mobile robot based on optical flow and convolutional neural network. Background Art

[0002] In recent years, with the rapid development of robot technology, mobile robots have gradually entered various fields of production and life. Whether they are used for material handling and palletizing in industrial production environments, or for garbage cleaning and tennis ball picking in daily life, mobile robots have performed very well. However, in the actual use of mobile robots, there is a problem: high-speed moving robots often increase the probability of accidents, and accidents caused by excessively high moving speeds can lead to huge economic losses and casualties.

[0003] Although scientific researchers have done a lot of work and designed many obstacle avoidance algorithms for mobile robots, most of these algorithms are based on the fact that the environmental model is known. In actual production and life, it is very difficult to establish an environmental model. Moreover, when the environment is dynamic, that is, when the environmental information changes over time, the environment must be remodeled, which is not conducive to the actual use of the robot. In addition, obstacle monitoring methods based on lidar information and ultrasound can play a certain role, but this is limited to medium and low-speed robots. For high-speed robots, even if obstacles are successfully detected, it is difficult to avoid them in a short time. Summary of the invention

[0004] In view of this, it is necessary to provide a real-time monitoring method for high-speed mobile robots based on optical flow and convolutional neural networks to address the above problems. The collected images are processed by the optical flow algorithm to obtain rough information of the dynamic mobile robot, and then the algorithm is used to obtain the binary image of the mobile robot. Finally, the convolutional neural network is used to detect the position information of the high-speed mobile robot, thereby realizing real-time monitoring and obstacle avoidance of the high-speed mobile robot without the need to establish an environmental model.

[0005] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0006] A real-time monitoring method for a high-speed mobile robot based on optical flow and convolutional neural network comprises the following steps:

[0007] Step S1: installing an image acquisition device at the edge of the working environment of the high-speed mobile robot so that the environmental information captured by the device includes the entire working environment;

[0008] Step S2: Turn on the image acquisition device to obtain environmental information in a static environment;

[0009] Step S3: The high-speed mobile robot starts working, and the image acquisition device acquires an environmental RGB image at regular intervals and transmits the acquired image to the computer system;

[0010] Step S4: converting the collected RGB image into a grayscale image, and using a mean filter algorithm to perform blur processing on the motion information grayscale image;

[0011] Step S5: using an optical flow algorithm to calculate two adjacent frames of images to generate a ROI (Region Of Interest, which represents a small area of ​​interest in the image) to express the approximate position information of the high-speed mobile robot, and perform morphological operations on it;

[0012] Step S6: using an adaptive threshold algorithm to perform binarization processing on the image to obtain a binary segmentation image of the mobile robot;

[0013] Step S7: repeat the above steps S3-S6 to construct a sample set of the motion position of the high-speed mobile robot, annotate the sample set with a rectangular box, and use a convolutional neural network (CNN) to perform model training on the sample set;

[0014] Step S8: In actual robot operation, the trained model is used to test the collected images to obtain the position information of the high-speed mobile robot and the position information of the obstacles;

[0015] Step S9: During the actual operation of the robot, the motion state of the robot is controlled according to the position information of the high-speed mobile robot and the position information of the obstacles.

[0016] Furthermore, in step S2, the environmental information in the static environment includes obstacles and reachable space of the robot.

[0017] Furthermore, the above step S4 specifically includes the following steps:

[0018] Use an R*R filter to scan the entire image with a certain step size. In each window, take the average of the weighted average sum of R*R pixels and fill it into each pixel, expressed as:

[0019]

[0020] Among them, R is the size of the filter, w(u, v) represents the weight of the filter, f(x, y) represents the pixel value of the original image, and g(x, y) represents the pixel value of the filtered image.

[0021] Furthermore, in the above step S5, the specific implementation process of the optical flow algorithm is:

[0022] A pixel I moves a certain distance in a certain time in the first frame to the next frame. The light intensity of the same pixel I before and after the movement remains unchanged, that is:

[0023] I(x,y,z,t)=I(x+Δx,y+Δy,z+Δz,t+Δt),Δt→0

[0024] Where I(x, y, z, t) represents the x-axis, y-axis, z-axis position and time of the pixel I in the first frame, Δx, Δy, Δz, Δt represent the distance and time of movement of the pixel I on the x-axis, y-axis, and z-axis;

[0025] That is:

[0026]

[0027] According to Taylor expansion, we can get:

[0028]

[0029] From the above equation we get:

[0030]

[0031] Where Vx and Vy represent and Let u and v represent the velocity vectors along the x-axis and y-axis respectively, then Vx=u, Vy=v.

[0032] Furthermore, in step S5, the morphological operation is specifically a closing operation:

[0033] The closing operation uses the method of expansion first and then erosion. The algorithm is described as:

[0034] Expansion operation:

[0035] Erosion Operation:

[0036] Closing operation:

[0037] Among them, M and N represent R 2 The set of spaces, r represents the set of all pixels that meet the conditions.

[0038] Furthermore, in step S6, the adaptive threshold algorithm is specifically:

[0039] The local 5*5 neighborhood block is filtered by using mean filtering, and a constant is subtracted from the pixel mean to obtain the threshold of the pixel point. For the pixel points whose original pixel value is greater than the threshold, it is set to 255, otherwise it is set to 0, which is expressed as:

[0040]

[0041] Where D represents the binary image after the adaptive threshold algorithm, I represents the pixel value of the original image, and I R Represents the pixel value after R*R mean filtering, and t represents a constant threshold greater than 0.

[0042] Compared with the prior art, the advantages and positive effects of the present invention include at least:

[0043] The present invention obtains rough information of the dynamic mobile robot through the optical flow algorithm of the collected image, and then obtains the binarized image of the mobile robot by using mean filter fuzzification processing, morphological operation and adaptive threshold binarization processing, and finally uses convolutional neural network to detect the position information of the high-speed mobile robot. Compared with the existing detection algorithm, the algorithm of the present invention is real-time, and can quickly detect the position of the robot after completing the training of the neural network model; and the present invention does not require the establishment of an environmental model, which solves the problem that most of the existing technologies require a detailed environmental model but the model is difficult to build in practice; the present invention is low-cost, only requires a computer and an image acquisition device, does not require the operation of professionals, and saves a lot of manpower, material resources and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a schematic diagram of the process of the real-time monitoring method of the present invention;

[0046] Figure 2 A schematic diagram of hardware device arrangement in an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the image acquisition and processing process of a high-speed mobile robot in one embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of the convolutional neural network structure used in the present invention.

[0049] In the figure, Figure 3 (a) is the collected RGB image; Figure 3 (b) is a grayscale image; Figure 3 (c) is the image after filtering; Figure 3(d) is the result of optical flow algorithm processing; Figure 3 (e) is the adaptive threshold segmentation image; Figure 3 (f) is the result of the closing operation. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be pointed out that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0052] It should be noted that the specific parameters or quantities used in the embodiments of the present invention are only several possible or preferred combinations used in the embodiments of the present invention, but they cannot be understood as limiting the scope of the patent of the present invention; it should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the attached claims.

[0053] Example 1

[0054] Figure 1 A flow chart of the real-time monitoring method of the present invention is given, as shown in FIG. Figure 1 As shown, the present invention provides a real-time monitoring method for a high-speed mobile robot based on optical flow and convolutional neural network, comprising the following steps:

[0055] Step S1: Install an image acquisition device at the edge of the working environment of the high-speed mobile robot so that the environmental information captured by the device includes the entire working environment.

[0056] Figure 2 A schematic diagram of hardware device layout in an embodiment of the present invention is given. Figure 2 As shown, the environmental information described in the present invention includes image acquisition equipment, computers, mobile robots, staff, obstacles, etc.

[0057] Specifically, the image acquisition device is a passive measurement camera device, which can be a monocular camera or a binocular camera, but the breadth of the image acquisition should be as large as possible to ensure that the complete robot working environment is acquired. The specific situation can be determined according to the actual situation.

[0058] The computer should at least include a CPU. To achieve a higher processing speed, the computer should preferably include a GPU. The computer software should be installed with python, opencv, and pytorch for processing the collected images.

[0059] The image acquisition device is installed at the edge of the mobile robot's working environment to ensure that the environmental information captured by the device is as comprehensive as possible.

[0060] Step S2: Turn on the image acquisition device to obtain environmental information in a static environment. Before the mobile robot starts working, the image acquisition device is turned on and the computer system starts running to ensure that the collected environmental images can be received. The environmental information in the static environment should include obstacles and robot accessible space.

[0061] Step S3: The high-speed mobile robot starts working, and the image acquisition device acquires an environmental RGB image at regular intervals and transmits the acquired image to the computer system. The acquired image should contain the environmental information, including the location of obstacles, the location of robots that are not working, and the location of workers. The image acquired by the image acquisition device should be a three-channel RGB image. In order to ensure that the information difference between two adjacent images is not too large or too small, the acquisition time interval between two adjacent frames of images is 100ms in this embodiment, that is, the frame rate of image acquisition is Fps.

[0062] Step S4: convert the collected RGB image into a grayscale image to reduce the number of image channels for ease of calculation, and use a mean filter algorithm to perform blur processing on the motion information grayscale image.

[0063] Mean filtering can eliminate some small noise points in the image, which is beneficial to reduce the impact of noise on the final detection accuracy. However, a filter that is too large will affect the accuracy of the original image. Therefore, in this embodiment, a 5*5 filter is selected to filter the acquired full image.

[0064] The specific steps include:

[0065] Use an R*R filter to scan the entire image with a certain step size. In each window, take the average of the weighted average sum of R*R pixels and fill it into each pixel, expressed as:

[0066]

[0067] Among them, R is the size of the filter, w(u,v) represents the weight of the filter, f(x,y) represents the pixel value of the original image, and g(x,y) represents the pixel value of the filtered image.

[0068] Step S5: Optical flow algorithm is used to calculate ROI (Region Of Interest) for two adjacent frames of images to express the approximate position information of the high-speed mobile robot, and morphological operations are performed on it.

[0069] The optical flow algorithm is the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane. It is a method that uses the change of pixels in the time domain in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. In this embodiment, the LK (Lucas-Kanada) optical flow algorithm is used to extract the rough position information of the mobile robot.

[0070] The optical flow algorithm obtains the approximate ROI of the mobile robot, but some important details will be lost. Therefore, the morphological algorithm of the image is used, specifically the closing operation. Since some parts of the robot are in motion and are not detected by the optical flow algorithm, the incoherence of some images is caused. The morphological operation of the image can repair some details in the image. In this embodiment, the morphological closing operation is used to improve the image of the robot. The closing operation adopts the method of first dilation and then erosion. Dilation can increase the size of the foreground in the image and amplify the details of the ROI, but it will also remove the noise. Then, the erosion method is used to remove the noise while ensuring the integrity of the robot's outline.

[0071] The specific implementation process of the optical flow algorithm is as follows:

[0072] A pixel I moves a certain distance in a certain time in the first frame to the next frame. The light intensity of the same pixel I before and after the movement remains unchanged, that is:

[0073] I(x,y,z,t)=I(x+Δx,y+Δy,z+Δz,t+Δt),Δt→0

[0074] Where I(x, y, z, t) represents the x-axis, y-axis, z-axis position and time of the pixel I in the first frame, Δx, Δy, Δz, Δt represent the distance and time of movement of the pixel I on the x-axis, y-axis, and z-axis;

[0075] That is:

[0076]

[0077] According to Taylor expansion, we can get:

[0078]

[0079] From the above equation we get:

[0080]

[0081] Where Vx and Vy represent and Let u and v represent the velocity vectors along the x-axis and y-axis respectively, then Vx=u, Vy=v.

[0082] The morphological operation is specifically a closing operation:

[0083] The closing operation uses the method of expansion first and then erosion. The algorithm is described as:

[0084] Expansion operation:

[0085] Erosion Operation:

[0086] Closing operation:

[0087] Among them, M and N represent R 2 The set of spaces, r represents the set of all pixels that meet the conditions.

[0088] Step S6: Binarize the image using an adaptive threshold algorithm to obtain a binary segmentation image of the mobile robot.

[0089] The working environment of the robot often has uncertain lighting conditions, so the collected images will show some bright areas and some dark areas, which is not conducive to direct threshold segmentation. Based on this, an adaptive threshold algorithm is proposed to process it.

[0090] The adaptive threshold algorithm is different from the global threshold. It pays more attention to the connection of context and divides the original image into smaller areas for judgment, which greatly reduces the impact of shadows on the image itself. The idea of ​​adaptive threshold is not to calculate the threshold of the global image, but to calculate the local threshold according to the brightness distribution of different areas of the image, so different thresholds can be adaptively calculated for different areas of the image. In this embodiment, the neighborhood block size of the adaptive threshold algorithm is 5*5, and the Gaussian mean of the neighborhood minus 10 is used as the segmentation threshold of the image.

[0091] The adaptive threshold algorithm is specifically:

[0092] The local 5*5 neighborhood block is filtered by using mean filtering, and a constant is subtracted from the pixel mean to obtain the threshold of the pixel point. For the pixel points whose original pixel value is greater than the threshold, it is set to 255, otherwise it is set to 0, which is expressed as:

[0093]

[0094] Where D represents the binary image after the adaptive threshold algorithm, I represents the pixel value of the original image, and I R Represents the pixel value after R*R mean filtering, and t represents a constant threshold greater than 0.

[0095] Figure 3 The effects of the above steps S3-S6 of the mobile robot in an actual working environment are demonstrated, where (a) is the captured RGB image; (b) is the grayscale image; (c) is the image after filtering; (d) is the result of the optical flow algorithm; (e) is the image segmented by adaptive threshold; and (f) is the result of the closing operation.

[0096] After processing the image with the above method, a binary image of the robot in a moving state is obtained. However, the binary image obtained at this time still contains all the robots in the moving state, and the image of the robot moving at a low speed is not what we want. Convolutional neural network (CNN) is a common method for target detection, which can cleverly solve this problem.

[0097] Step S7: Repeat the above steps S3-S6 to construct a sample set of the motion position of the high-speed mobile robot, annotate the sample set with a rectangular box, and use a convolutional neural network (CNN) to perform model training on the sample set.

[0098] The image acquisition device can collect 10 real-time images of the robot's motion environment per second, so a neural network data set can be constructed in a short time. The input is the image collected and processed by the above S3-S6, and the corresponding output is the position of the rectangular frame corresponding to the image. In this embodiment, the coordinates of the upper right corner and the lower left corner of the rectangular frame are used as output.

[0099] The convolutional neural network (CNN) trains image samples through a series of convolution and pooling operations, continuously adjusting the network parameters, learning the slight differences of high-speed mobile robots in the image, and realizing the mapping relationship between input and position. The network originally used for positioning was RCNN, but it was slow and consumed a lot of memory, making it inconvenient for practical use.

[0100] The neural network used in this invention is Faster RCNN, which is an improved network structure of RCNN and Fast RCNN. Compared with the former two, it proposes a region proposal subnetwork, which not only improves the accuracy of the network, but also effectively improves the running speed of the network. By utilizing its advantages, the real-time detection function of high-speed robots can be realized. The architecture of this network is as follows Figure 4 As shown, the main structure includes an anchor point generation layer, a region proposal layer, a ROI pooling layer, and a classification layer. In this embodiment, 1500 and 500 images are taken as the training set and validation set of the network respectively.

[0101] Before network training, the data set must be prepared. The image processed in steps S3-S6 is used as the input of the data set, and the high-speed moving robot target in the image is marked with a rectangular frame. The position of the rectangular frame corresponding to the image is used as the output of the data set. Fast RCNN is used to train the training set to obtain the model required for actual detection.

[0102] Step S8: In actual robot operation, the trained model is used to test the collected images to obtain the position information of the high-speed mobile robot and the position information of the obstacles; and they are marked with rectangular frames.

[0103] Step S9: During the actual operation of the robot, the motion state of the robot is controlled according to the position information of the high-speed mobile robot and the position information of the obstacles.

[0104] Compared with the prior art, the advantages and positive effects of the present invention include at least:

[0105] The present invention obtains rough information of the dynamic mobile robot through the optical flow algorithm of the collected image, and then obtains the binarized image of the mobile robot by using mean filter fuzzification processing, morphological operation and adaptive threshold binarization processing, and finally uses convolutional neural network to detect the position information of the high-speed mobile robot. Compared with the existing detection algorithm, the algorithm of the present invention is real-time, and can quickly detect the position of the robot after completing the training of the neural network model; and the present invention does not require the establishment of an environmental model, which solves the problem that most of the existing technologies require a detailed environmental model but the model is difficult to build in practice; the present invention is low-cost, only requires a computer and an image acquisition device, does not require the operation of professionals, and saves a lot of manpower, material resources and maintenance costs.

[0106] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A real-time monitoring method for a high-speed mobile robot based on optical flow and convolutional neural network, characterized in that: The following steps are involved: Step S1: installing an image acquisition device at the edge of the working environment of the high-speed mobile robot so that the environmental information captured by the device includes the entire working environment; Step S2: Turn on the image acquisition device to obtain environmental information in a static environment; Step S3: The high-speed mobile robot starts working, and the image acquisition device acquires an environmental RGB image at regular intervals and transmits the acquired image to the computer system; Step S4: converting the collected RGB image into a grayscale image, and using a mean filter algorithm to perform blur processing on the motion information grayscale image; Step S5: using the optical flow algorithm to calculate the ROI of two adjacent frames of images, expressing the approximate position information of the high-speed mobile robot, and performing morphological operations on it; Step S6: using an adaptive threshold algorithm to perform binarization processing on the image to obtain a binary segmentation image of the mobile robot; Step S7: repeat the above steps S3-S6 to construct a sample set of the moving position of the high-speed mobile robot, annotate the sample set with a rectangular frame, and use a convolutional neural network to perform model training on the sample set; Step S8: In actual robot operation, the trained model is used to test the collected images to obtain the position information of the high-speed mobile robot and the position information of the obstacles; Step S9: During the actual operation of the robot, the motion state of the robot is controlled according to the position information of the high-speed mobile robot and the position information of the obstacles.

2. The real-time monitoring method for a high-speed mobile robot based on optical flow and convolutional neural network according to claim 1 is characterized in that: In step S2, the environmental information in the static environment includes obstacles and reachable space of the robot.

3. The real-time monitoring method for a high-speed mobile robot based on optical flow and convolutional neural network according to claim 2 is characterized in that: The above step S4 specifically includes the following steps: Use an R*R filter to scan the entire image with a certain step size. In each window, take the average of the weighted average sum of R*R pixels and fill it into each pixel, expressed as: Among them, R is the size of the filter, w(u,v) represents the weight of the filter, f(x,y) represents the pixel value of the original image, and g(x,y) represents the pixel value of the filtered image.

4. The real-time monitoring method for a high-speed mobile robot based on optical flow and convolutional neural network according to claim 2 is characterized in that: In the above step S5, the specific implementation process of the optical flow algorithm is: A pixel I moves a certain distance in a certain time in the first frame to the next frame. The light intensity of the same pixel I before and after the movement remains unchanged, that is: Where Vx and Vy represent and Let u and v represent the velocity vectors along the x-axis and y-axis respectively, then Vx=u, Vy=v.

5. The real-time monitoring method for a high-speed mobile robot based on optical flow and convolutional neural network according to claim 2 is characterized in that: In step S5, the morphological operation is specifically a closing operation: The closing operation uses the method of expansion first and then erosion. The algorithm is described as: Expansion operation: Erosion Operation: Closing operation: Among them, M and N represent R 2 The set of spaces, r represents the set of all pixels that meet the conditions.

6. The real-time monitoring method for a high-speed mobile robot based on optical flow and convolutional neural network according to claim 2 is characterized in that: In step S6, the adaptive threshold algorithm is specifically: The local 5*5 neighborhood block is filtered by using mean filtering, and a constant is subtracted from the pixel mean to obtain the threshold of the pixel point. For the pixel points whose original pixel value is greater than the threshold, it is set to 255, otherwise it is set to 0, which is expressed as: Where D represents the binary image after the adaptive threshold algorithm, I represents the pixel value of the original image, and I R Represents the pixel value after R*R mean filtering, and t represents a constant threshold greater than 0.

Images