Obstacle avoidance method and device based on machine vision

Abstract

This application provides a machine vision-based obstacle avoidance method and device. The method includes the following steps: responding to a destination command, obtaining destination information, and planning a travel path from the current location to the destination; traveling along the planned travel path and capturing real-time images of the environment in front of the traveling body during travel; using a pre-built obstacle recognition model to identify the environment image and obtain obstacle information; and replanning the travel path from the current location to the destination based on the obstacle information. This application improves the accuracy of obstacle recognition on the traveling body's path and plans the obstacle avoidance path in advance, thereby improving travel efficiency.

Description

Technical Field

[0001] The present application relates to the field of mobile communication technology, and in particular to a machine vision-based obstacle avoidance method and device. Background Art

[0002] Autonomous vehicles, robots, and blind people rely on obstacle recognition to make driving decisions, ensuring they avoid obstacles and avoid collisions. Robots include sweeping robots, inspection robots, and guide robots. Guide robots are auxiliary tools that provide environmental guidance for the visually impaired. They fall under the category of service robots and use a variety of sensors to detect their surroundings and provide feedback to the visually impaired, helping them compensate for their lack of visual information.

[0003] The presence of unknown obstacles in the environment poses a significant threat to moving entities (such as self-driving cars, robots, or blind people). Currently, object recognition and positioning devices are used to identify and locate obstacles, alerting moving entities to their presence. However, obstacle recognition accuracy is currently low, and without pre-planned obstacle avoidance paths, the system may encounter other obstacles during the avoidance process, resulting in low travel efficiency.

[0004] Therefore, the current technical problem to be solved is: how to improve the accuracy of obstacle recognition on the moving body's path, and plan the obstacle avoidance path in advance to improve travel efficiency. Summary of the Invention

[0005] The purpose of this application is to provide an obstacle avoidance method and device based on machine vision, to improve the accuracy of obstacle recognition on the moving path of the moving body, and to plan the obstacle avoidance path in advance to improve the travel efficiency.

[0006] To achieve the above-mentioned objectives, the present application provides an obstacle avoidance method based on machine vision, which includes the following steps: responding to a destination instruction, obtaining destination information, and planning a travel path from the current location to the destination; traveling according to the planned travel path, and collecting environmental images in front of the traveling body in real time during the travel process; identifying the environmental images through a pre-built obstacle recognition model to obtain obstacle information; and replanning the travel path from the current location to the destination based on the obstacle information.

[0007] The machine vision-based obstacle avoidance method described above, wherein the method of pre-building an obstacle recognition model includes the following steps: obtaining a training data set; inputting the training data set into a convolutional neural network model for training to obtain an obstacle recognition model; and optimizing and verifying the obstacle recognition model.

[0008] In the machine vision-based obstacle avoidance method described above, the method for optimizing and verifying the obstacle recognition model includes: obtaining multiple images of known obstacles as a verification set; inputting the images in the verification set into the obstacle recognition model for verification, and obtaining recognition results of the obstacle recognition model; calculating the recognition accuracy of the obstacle recognition model based on the recognition results of the obstacle recognition model; and comparing the calculated recognition accuracy with a preset threshold. If the calculated recognition accuracy is less than the preset threshold, repeatedly optimizing and training the obstacle recognition model until the recognition accuracy of the obstacle recognition model exceeds the preset threshold.

[0009] As described above, the obstacle avoidance method based on machine vision, wherein the method of replanning the travel path from the current location to the destination according to obstacle information includes: obtaining the position information of the next path turning node closest to the obstacle in the original travel path according to the obstacle information; inserting an obstacle avoidance node between the current point and the next path turning node closest to the obstacle.

[0010] As described above, the machine vision-based obstacle avoidance method, wherein the method of inserting an obstacle avoidance node between the current point and the next path turning node closest to the obstacle includes: establishing an obstacle avoidance node optimization model; optimizing the position of the obstacle avoidance node according to the obstacle avoidance node optimization model; and inserting the obstacle avoidance node after the optimized position on the basis of the original travel path to re-plan the travel path from the current point to the destination.

[0011] The obstacle avoidance method based on machine vision as described above, wherein the method of obtaining destination information in response to a destination instruction and planning a travel path from the current location to the destination includes: obtaining destination information in response to a destination instruction; returning the obtained destination information to the traveling entity for confirmation to determine whether the destination information is correct; if correct, planning a travel path from the current location to the destination based on the confirmed correct destination information, otherwise, re-acquiring the destination information.

[0012] The present application also provides an obstacle avoidance device based on machine vision, which is installed on a moving body. The obstacle avoidance device includes: a path planning module, which is used to respond to destination instructions, obtain destination information, and plan a travel path from the current location to the destination; the moving body moves according to the planned travel path; an image acquisition device, which is used to collect environmental images in front of the moving body in real time during the travel process; an obstacle information acquisition module, which is used to identify the environmental image through a pre-built obstacle recognition model and obtain obstacle information; and a path correction module, which is used to re-plan the travel path from the current location to the destination based on the obstacle information.

[0013] The machine vision-based obstacle avoidance device as described above further includes an operation monitoring module, which is used to monitor the operation status of the obstacle avoidance device.

[0014] As described above, the obstacle avoidance device based on machine vision, wherein the path correction module includes: an acquisition module, used to obtain the position information of the next path turning node closest to the obstacle in the original travel path based on the obstacle information; an obstacle avoidance node insertion module, used to insert an obstacle avoidance node between the current point and the next path turning node closest to the obstacle.

[0015] The obstacle avoidance device based on machine vision as described above further includes an alarm for issuing an alarm when obstacle information is obtained.

[0016] The beneficial effects achieved by this application are as follows:

[0017] (1) This application pre-trains an obstacle recognition model, calculates the recognition accuracy of the obstacle recognition model, and obtains an obstacle recognition model whose recognition accuracy meets a preset threshold value, so as to recognize obstacles in the direction of the current travel path and improve the recognition accuracy of obstacles in front of the moving body.

[0018] (2) This application inserts an obstacle avoidance node between the current point and the next path turning point, plans the obstacle avoidance path in advance, and improves the travel efficiency of the moving body. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can also be obtained based on these drawings.

[0020] Figure 1 This is a flowchart of a machine vision-based obstacle avoidance method according to an embodiment of the present application.

[0021] Figure 2 This is a flow chart of a method for planning a travel path from a current location to a destination according to an embodiment of the present application.

[0022] Figure 3 This is a flow chart of the method for collecting environmental images according to an embodiment of the present application.

[0023] Figure 4 This is a flow chart of a method for constructing an obstacle recognition model according to an embodiment of the present application.

[0024] Figure 5 This is a flow chart of a method for replanning a travel path from a current location to a destination based on obstacle information according to an embodiment of the present application.

[0025] Figure 6 This is a flow chart of a method for inserting an obstacle avoidance node according to an embodiment of the present application.

[0026] Figure 7 This is a schematic structural diagram of a machine vision-based obstacle avoidance device according to an embodiment of the present application.

[0027] Figure 8 This is a schematic structural diagram of an electronic device according to an embodiment of the present application.

[0028] Figure numerals: 10-path planning module; 20-image acquisition device; 30-obstacle information acquisition module; 40-path correction module; 41-acquisition module; 42-obstacle avoidance node insertion module; 43-high-precision map module; 50-device operation monitoring module; 60-data processor; 70-data comparator; 80-alarm; 100-machine vision-based obstacle avoidance device; 200-electronic device; 201-memory; 202-processor. DETAILED DESCRIPTION

[0029] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0030] Example 1

[0031] like Figure 1 As shown, the present application provides an obstacle avoidance method based on machine vision, which includes the following steps:

[0032] Step S1, in response to a destination instruction, obtain destination information and plan a travel path from the current location to the destination.

[0033] like Figure 2 As shown, step S1 includes the following sub-steps:

[0034] Step S110 , in response to the destination instruction, obtaining destination information.

[0035] In one specific embodiment of the present invention, a traveling object issues a voice destination command to an obstacle avoidance device. The obstacle avoidance device uses voice recognition to receive the destination command and, in response, identifies the destination information. This destination information includes the destination type and name. The traveling object can be an inspection robot, a robot vacuum, a blind person, or an unmanned vehicle.

[0036] In a specific embodiment of the present invention, a voice destination command or a destination instruction segment is sent to the obstacle avoidance device via a remote control terminal. The obstacle avoidance device then recognizes the destination information through voice recognition or semantic recognition. Specifically, voice recognition is used to recognize the voice destination command and obtain the destination information, while semantic recognition uses semantic recognition technology to recognize the destination instruction segment and obtain the destination information. The destination information may include, for example, a restaurant, a bedroom, or another location (such as a bank, supermarket, or bus station).

[0037] Step S120: Return the acquired destination information to the traveling entity for confirmation to determine whether the destination information is correct.

[0038] Specifically, the destination information recognized by the obstacle avoidance device is fed back to the traveling entity in the form of voice or text (sentences). It can be understood that the obstacle avoidance device sends the recognized destination information to the traveling entity in the form of voice or text (sentences) so that the traveling entity can confirm the destination information recognized by the obstacle avoidance device. The traveling entity can have a voice recognition function for recognizing the destination information in the form of voice sent by the obstacle avoidance device, or the traveling entity can have a semantic recognition function for recognizing the destination information in the form of text (sentences) sent by the obstacle avoidance device through semantic recognition technology.

[0039] As a specific embodiment of the present invention, the obstacle avoidance device broadcasts the destination information in the form of voice. After the moving body recognizes the broadcast voice, it determines whether the broadcast destination information is correct. If correct, it returns confirmation information to the obstacle avoidance device. Otherwise, it returns confirmation information of error, and the obstacle avoidance device re-identifies and feeds back the destination information.

[0040] As another specific embodiment of the present invention, the obstacle avoidance device feeds back the destination information in text form to the traveling body. The traveling body compares the destination information fed back by the obstacle avoidance device with the text in the voice destination instruction it issues to determine whether they are consistent. If so, it returns confirmation information to the obstacle avoidance device. Otherwise, it returns confirmation information of error, and the obstacle avoidance device re-identifies and feeds back the destination information.

[0041] Step S130: If the destination information is correct, a travel route from the current location to the destination is planned based on the confirmed correct destination information; otherwise, the destination information is acquired again.

[0042] Specifically, the current location is located by a satellite navigation system to obtain the current location information of the moving body, and the location information of the destination is obtained based on the destination information to plan the travel path from the current location to the destination. The satellite navigation system is installed on the obstacle avoidance device, and the obstacle avoidance device is installed on the moving body.

[0043] Step S2: The vehicle moves along the planned path and collects images of the environment in front of the vehicle in real time during the movement.

[0044] Specifically, the moving body moves along the planned moving path, and an image acquisition device collects images of the environment in front of the moving body in real time during the moving body's movement. The image acquisition device is, for example, a video camera.

[0045] like Figure 3 As shown, step S2 includes the following steps:

[0046] Step S210: Travel according to the planned travel path.

[0047] Specifically, in response to the completion of the travel path planning, the planned travel path is broadcast in real time by voice. After the travel body recognizes the content of the voice broadcast, it moves according to the content of the voice broadcast. The content of the voice broadcast includes: move forward 10 meters, move left 50 meters, and move right 10 meters.

[0048] Step S220: collecting images of the environment in front of the moving body in real time during the moving process.

[0049] Specifically, the video camera collects environmental images in the direction of the moving body, which include images of the ground, objects on the ground, and images above the height of the moving body (such as a desktop or objects on the desktop).

[0050] Step S3: Using a pre-built obstacle recognition model, the environment image is recognized to obtain obstacle information.

[0051] As a specific embodiment of the present invention, information about obstacles during movement is obtained through a laser radar, and obstacle avoidance reminders are issued based on the obstacle information to modify the moving direction.

[0052] like Figure 4 As shown, the method for pre-building an obstacle recognition model includes the following steps:

[0053] Step Y1: Obtain a training data set.

[0054] The method to obtain the training data set is:

[0055] Collect multiple images of common obstacles.

[0056] As a specific embodiment of the present invention, common obstacles collected include, for example, tables, chairs, people, stones on the ground, telephone poles, and wall protrusions.

[0057] Perform image enhancement processing on the collected images.

[0058] In step Y2, the training data set is input into the convolutional neural network model for training to obtain an obstacle recognition model.

[0059] Specifically, images of common obstacle objects are input into a convolutional neural network model for training to obtain an obstacle recognition model. The obstacle recognition model is used to identify different obstacles in the image and obtain obstacle data.

[0060] Step Y3: Optimize and verify the obstacle recognition model.

[0061] Step Y3 includes the following sub-steps:

[0062] Step Y310, obtain multiple images of known obstacles as a verification set.

[0063] Step Y320: Input the images in the verification set into the obstacle recognition model for verification, and obtain the recognition results of the obstacle recognition model.

[0064] The recognition result is to identify the type of obstacle in the image.

[0065] Step Y330: Calculate the recognition accuracy of the obstacle recognition model based on the recognition result of the obstacle recognition model.

[0066] The recognition accuracy of the obstacle recognition model is calculated as: Q = Ks / Kz, where Q represents the recognition accuracy of the obstacle recognition model; Ks represents the number of obstacles recognized by the obstacle recognition model that are consistent with the known obstacle types; and Kz represents the total number of images input into the obstacle recognition model.

[0067] In step Y340, the calculated recognition accuracy is compared with a preset threshold. If the calculated recognition accuracy is less than the preset threshold, the obstacle recognition model is repeatedly optimized and trained to improve its recognition accuracy until the recognition accuracy is greater than the preset threshold. Otherwise, there is no need to repeatedly optimize and train the obstacle recognition model.

[0068] Among them, the existing loss function optimization method is used to optimize the training of the obstacle recognition model.

[0069] Step S3 also includes, when obstacle information is identified, issuing a voice alarm of the obstacle information through the alarm, for example, the obstacle information broadcast by the voice includes the obstacle ahead, the obstacle type or the distance between the obstacle and the moving body, etc.

[0070] As a specific embodiment of the present invention, a pre-built obstacle recognition model is placed in the obstacle avoidance device to identify obstacles in the image captured by the image capture device.

[0071] Step S4: replan the travel path from the current location to the destination based on the obstacle information, and return to step S2 until the destination is reached.

[0072] like Figure 5 As shown, the method of avoiding obstacles and replanning the travel path from the current location to the destination based on the obtained obstacle information includes:

[0073] Step S410: Obtain the position information of the next path turning node closest to the obstacle in the original travel path according to the obstacle information.

[0074] Step S420: insert an obstacle avoidance node between the current point and the next path turning node closest to the obstacle.

[0075] Specifically, the original path between the current point and the next turning point is replaced with the path from the current point to the obstacle avoidance node, and then from the obstacle avoidance node to the next turning point. It can be understood that inserting the obstacle avoidance node allows the current point to travel to the obstacle avoidance node and then from the obstacle avoidance node along the original planned path, achieving the shortest possible path while safely avoiding obstacles.

[0076] Specifically, based on the image of the obstacle in the obtained obstacle information, the maximum width of the obstacle outline from the center point is obtained; combined with the position information of the obstacle and the preset obstacle avoidance safety distance, an obstacle avoidance node is inserted near the obstacle.

[0077] As a specific embodiment of the present invention, the category of the obstacle is obtained based on the obtained obstacle information. When the category of the obstacle is an object that may be active, for example, a person, the moving body remains stationary and waits for a preset time (for example, 3S, 5S or 8S, etc.), and the image acquisition device installed on the moving body captures the environment image in front of the moving body again, and the obstacle in the environment image is identified and compared with the obstacle image identified 5S ago. If the obstacle is identified to be still in the original position, the judgment result is: the obstacle is inactive (not moving); if the identified obstacle no longer exists or has changed its position, the judgment result is: the obstacle is active.

[0078] As a specific embodiment of the present invention, when the obstacle is detected to be a stationary object or a person, an obstacle avoidance node is inserted between the current point and the next path turning node closest to the obstacle. Otherwise, there is no need to insert an obstacle avoidance node and the vehicle continues to move along the original path.

[0079] like Figure 6 As shown, the method for inserting an obstacle avoidance node includes the following steps:

[0080] Step S421: Establish an obstacle avoidance node optimization model.

[0081] Among them, the obstacle avoidance node optimization model is established as follows:

[0082] D = max(DL, DR);

[0083]

[0084]

[0085] Where D represents the obstacle avoidance node with the largest fitness value; max(DL, DR) represents the maximum value of DL and DR; DL represents the fitness value of the obstacle avoidance node when traveling from the left side of the obstacle; DR represents the fitness value of the obstacle avoidance node when traveling from the right side of the obstacle; SL represents the obstruction value of the obstacle avoidance node on the left side; SR represents the obstruction value of the obstacle avoidance node on the right side; 12 Indicates the distance between the current point and the obstacle along the direction from the current point to the next turning point of the path; dz i Indicates the width distance between the i-th pixel point in the left contour of the obstacle and the original path (the connection direction from the current point to the next path turning point); dz j represents the width distance between the jth pixel point in the right contour of the obstacle and the original path; L represents the total number of pixels in the left contour of the obstacle; R represents the total number of pixels in the right contour of the obstacle; δ i represents the obstacle factor of the i-th pixel in the left contour of the obstacle; δ j represents the obstacle factor of the jth pixel in the right contour of the obstacle; A represents the obstacle avoidance safety distance; Bx represents the width of the moving body; max(δ i ×dz i ) indicates that when i ranges from 1 to L, calculate δ i ×dz i , obtain L values, and take the maximum value among the L values. The left contour of the obstacle refers to the contour located on the left side of the connection direction from the current point to the next path turning point, and the right contour of the obstacle refers to the contour located on the right side of the connection direction from the current point to the next path turning point.

[0086] As a specific embodiment of the present invention, an environmental image is captured by a camera, and the camera's shooting direction extends along the path between the current point and the next path turning point. When an obstacle is identified on the path between the current point and the next path turning point by an obstacle recognition model, an obstacle image is output. The obstacle in the obstacle image is segmented using a vertical line at the intersection of the path between the current point and the next path turning point and the obstacle. The pixel points of the obstacle outline located to the left of the segmentation line (i.e., the vertical line) are extracted as left contour pixel points, and the pixel points of the obstacle outline located to the right of the segmentation line are extracted as right contour pixel points.

[0087] Among them, the obstacle factor δ of the i-th pixel in the left contour of the obstacle i The calculation formula is as follows:

[0088]

[0089] Where Hzi represents the height of the i-th pixel point in the left contour of the obstacle from the ground; Hx represents the height of the moving body. The obstacle factor δ of the j-th pixel point in the right contour of the obstacle j The obstacle factor δ of the i-th pixel in the left contour of the obstacle i The calculation method is the same.

[0090] As a specific embodiment of the present invention, the ground in the obstacle image is identified by a pre-trained ground recognition model, thereby obtaining the vertical height of the obstacle's pixel point from the ground. The method of pre-training the ground recognition model is the same as the direction of pre-training the obstacle recognition model.

[0091] As a specific embodiment of the present invention, an image acquisition device is aligned with the position of the left obstacle avoidance node to capture an environmental image of the left obstacle avoidance node; a new obstacle in the environmental image is identified based on an obstacle recognition model to obtain a new obstacle contour image; and the obstacle value of the left obstacle avoidance node is calculated based on the distance between the new obstacle and the left obstacle avoidance node in the new obstacle contour image.

[0092] The calculation method of the obstacle value SL of the left obstacle avoidance node is:

[0093]

[0094] Where K represents the total number of right contour pixels of the new obstacle located at the left obstacle avoidance node; δ z represents the obstacle factor of the z-th pixel in the right contour of the new obstacle located at the left obstacle avoidance node; δ z Calculation method and δ i The calculation method is the same as db z Indicates the width distance between the right contour pixel point of the new obstacle located at the left obstacle avoidance node and the left obstacle avoidance node, that is, the distance perpendicular to the current point and the left obstacle avoidance node.

[0095] As a specific real-time example of the present invention, an image acquisition device is aligned with the position of the right obstacle avoidance node to capture an environmental image of the right obstacle avoidance node; a new obstacle in the environmental image is identified based on an obstacle recognition model to obtain a new obstacle contour image; and the obstacle value of the right obstacle avoidance node is calculated based on the distance between the new obstacle and the right obstacle avoidance node in the new obstacle contour image.

[0096] The calculation method of the obstacle value SR of the right obstacle avoidance node is:

[0097]

[0098] Where P represents the total number of left contour pixels of the new obstacle at the right obstacle avoidance node; δ h represents the obstacle factor of the h-th pixel in the left contour of the new obstacle located at the right obstacle avoidance node; δ h Calculation method and δ i The calculation method is the same as dc h Indicates the width distance between the left contour pixel of the new obstacle located at the right obstacle avoidance node and the right obstacle avoidance node, that is, the distance perpendicular to the current point and the right obstacle avoidance node.

[0099] Step S422: Optimize the position of the obstacle avoidance node according to the obstacle avoidance node optimization model.

[0100] According to the path optimization model, the fitness value DL of the obstacle avoidance node when traveling from the left side of the obstacle and the fitness value DR of the obstacle avoidance node when traveling from the right side of the obstacle are calculated; the position of the obstacle avoidance node corresponding to the larger fitness value of DL and DR is selected as the optimized position of the obstacle avoidance node.

[0101] Step S423: On the basis of the original travel path, the obstacle avoidance node with the optimized position is inserted to replan the travel path from the current point to the destination.

[0102] Specifically, the moving body moves from the current position to the obstacle avoidance node, and then moves from the obstacle avoidance node to the next node in the originally planned moving path, and continues to move according to the originally planned moving path, and avoids obstacles in real time. The obstacle avoidance method is based on steps S2-S4.

[0103] As a specific embodiment of the present invention, the position of inserting the obstacle avoidance node is: when the obstacle avoidance node is located on the left side of the obstacle, the distance between the obstacle avoidance node and the original path (the connection path between the current point and the next node of the original planned path) is That is, the position of the obstacle avoidance node on the left, or when the obstacle avoidance node is on the right side of the obstacle, the distance between the obstacle avoidance node and the original path (the connection path between the current point and the next node of the original planned path) is That is the position of the obstacle avoidance node on the right.

[0104] Step S5: collecting the operating status data of the obstacle avoidance device in real time.

[0105] Preferably, the operating status data of the obstacle avoidance device is collected in real time, and the operating status data of the obstacle avoidance device includes: remaining power and fault data. The fault data includes low voltage or high voltage, high temperature, etc.

[0106] Step S6: Calculate the operational reliability value of the obstacle avoidance device based on the operational status data of the obstacle avoidance device.

[0107] The calculation formula for the operational reliability value of the obstacle avoidance device is:

[0108]

[0109] Where W represents the operational reliability value of the obstacle avoidance device; α1 represents the influence weight of the remaining power usage time; E represents the remaining power; Ec represents the power consumption per hour; α2 represents the influence weight of the successful obstacle avoidance; V1 represents the number of successful obstacle avoidances; V represents the total number of obstacle avoidances; α3 represents the influence weight of the fault data; G represents the total number of fault data types; q a Indicates the impact factor of the a-th fault data; U a represents the measured value of the ath fault data; U e Represents the standard value of the ath fault data; α1+α2+α3=1.

[0110] Step S7, comparing the operational reliability value of the obstacle avoidance device with a preset safety threshold. If the operational reliability value of the obstacle avoidance device is less than the preset safety threshold, an alarm is issued; otherwise, the operational status of the obstacle avoidance device continues to be monitored.

[0111] Among them, if the operating reliability value of the obstacle avoidance device is less than the preset safety threshold, an alarm reminder will be issued through the alarm device, and the alarm reminder can be a voice alarm broadcast, vibration response, etc.

[0112] Example 2

[0113] like Figure 7 As shown, the present application provides a machine vision-based obstacle avoidance device 100, which is installed on a moving body. Preferably, the machine vision-based obstacle avoidance device 100 is installed directly in front of the moving body. The machine vision-based obstacle avoidance device 100 includes:

[0114] The path planning module 10 is used to respond to the destination instruction, obtain the destination information, and plan the travel path from the current location to the destination; the travel body travels according to the planned travel path.

[0115] The image acquisition device 20 is used to acquire the environment image in front of the moving body in real time during the moving process. The image acquisition device is a video camera.

[0116] The obstacle information acquisition module 30 is used to identify the environment image through a pre-built obstacle recognition model to obtain obstacle information.

[0117] The path correction module 40 is used to re-plan the travel path from the current location to the destination according to the obstacle information.

[0118] like Figure 7 As shown, the path correction module 40 includes:

[0119] The acquisition module 41 is used to acquire the position information of the next path turning node closest to the obstacle in the original travel path according to the obstacle information.

[0120] The obstacle avoidance node insertion module 42 is used to insert an obstacle avoidance node between the current point and the next path turning node closest to the obstacle.

[0121] High-precision mapping module 43 is used to collect high-precision maps of fixed locations using RTK equipment to generate a site map. It also projects obstacle location information or high-precision GPS information to obtain the obstacle's coordinates on the site map. Obstacle high-precision GPS information is obtained by capturing an image of the obstacle within the fixed location using a camera, identifying the obstacle's center point using an image recognition algorithm, and obtaining the coordinates of the center point in the image. Using these coordinates, the module then queries an image pre-calibrated using RTK equipment to obtain high-precision GPS information about the obstacle.

[0122] like Figure 7 As shown, the machine vision-based obstacle avoidance device 100 further includes:

[0123] The device operation monitoring module 50 is used to collect the operation status data of the obstacle avoidance device in real time.

[0124] The data processor 60 is used to calculate the operation reliability value of the obstacle avoidance device according to the operation status data of the obstacle avoidance device.

[0125] The data comparator 70 is used to compare the operation reliability value of the obstacle avoidance device with the preset safety threshold. If the operation reliability value of the obstacle avoidance device is less than the preset safety threshold, an alarm is issued. Otherwise, the device operation monitoring module continues to monitor the operation status of the obstacle avoidance device.

[0126] The alarm device 80 is used to issue an alarm when obstacle information is obtained.

[0127] The battery module is used to provide power to the power-consuming modules in the obstacle avoidance device. The power-consuming modules are modules that use electric energy to work. For example, the path planning module 10, the image acquisition device 20, the obstacle information acquisition module 30, the path correction module 40, the device operation monitoring module 50, the data processor 60, the data comparator 70 and the alarm 80 are all power-consuming modules.

[0128] As a specific embodiment of the present invention, the machine vision-based obstacle avoidance device 100 further includes:

[0129] The communication module is used to communicate with external devices and send alarm information or related location information to the external devices. The external devices can be mobile phones or computers.

[0130] As another specific embodiment of the present invention, the machine vision-based obstacle avoidance device 100 further includes:

[0131] The electronic compass is primarily driven by the executive controller, which performs initialization, data reading, and angular coordinate integration. Once the electronic compass's direction sensing function is enabled, the main controller transmits the startup information to the executive controller, which then drives the electronic compass module to begin direction recognition, obtain the azimuth's X, Y, and Z coordinates, and calculate the azimuth declination according to existing formulas. The existing electronic compass senses the azimuth declination of the moving object during travel, effectively compensating for the GPS signal and improving the accuracy of GPS navigation and orientation information.

[0132] Example 3

[0133] like Figure 8 As shown, the present application provides an electronic device 200, including a memory 201, a processor 202, and a computer program stored in the memory 201 and executable on the processor 202. When the processor 202 executes the computer program, the steps of the machine vision-based obstacle avoidance method in Example 1 are implemented. Alternatively, when the processor 202 executes the computer program, the functions of each module / unit in the machine vision-based obstacle avoidance device 100 in Example 2 are implemented.

[0134] The beneficial effects achieved by this application are as follows:

[0135] (1) This application pre-trains an obstacle recognition model, calculates the recognition accuracy of the obstacle recognition model, and obtains an obstacle recognition model whose recognition accuracy meets a preset threshold value, so as to recognize obstacles in the direction of the current travel path and improve the recognition accuracy of obstacles in front of the moving body.

[0136] (2) This application inserts an obstacle avoidance node between the current point and the next path turning point, plans the obstacle avoidance path in advance, and improves the travel efficiency of the moving body.

[0137] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage) containing computer-usable program code.

[0138] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, devices, and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0140] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. An obstacle avoidance method based on machine vision, characterized in that: The method comprises the following steps: In response to a destination instruction, obtaining destination information and planning a travel path from the current location to the destination; The vehicle moves along the planned path and collects images of the surroundings in front of it in real time during the movement. Through the pre-built obstacle recognition model, the environmental image is identified to obtain obstacle information; Replan the path from the current location to the destination based on obstacle information; Real-time collection of operating status data of obstacle avoidance devices; Calculating the operational reliability value of the obstacle avoidance device based on the operational status data of the obstacle avoidance device; Compare the operational reliability value of the obstacle avoidance device with the preset safety threshold. If the operational reliability value of the obstacle avoidance device is less than the preset safety threshold, issue an alarm. Otherwise, continue to monitor the operational status of the obstacle avoidance device. Among them, the method of replanning the travel path from the current location to the destination based on the obstacle information includes: According to the obstacle information, the location information of the next path turning point closest to the obstacle in the original path is obtained; Insert an obstacle avoidance node between the current point and the next path turning node closest to the obstacle; The method of inserting an obstacle avoidance node between the current point and the next path turning node closest to the obstacle includes: Establish an obstacle avoidance node optimization model; Optimize the position of obstacle avoidance nodes according to the obstacle avoidance node optimization model; On the basis of the original travel path, the obstacle avoidance nodes with optimized positions are inserted to replan the travel path from the current location to the destination; The obstacle avoidance node optimization model is established as follows: D = max(DL, DR); Where D represents the obstacle avoidance node with the largest fitness value; max(DL, DR) represents the maximum value of DL and DR; DL represents the fitness value of the obstacle avoidance node when traveling from the left side of the obstacle; DR represents the fitness value of the obstacle avoidance node when traveling from the right side of the obstacle; SL represents the obstruction value of the obstacle avoidance node on the left side; SR represents the obstruction value of the obstacle avoidance node on the right side; 12 Indicates the distance between the current point and the obstacle along the direction from the current point to the next turning point of the path; dz i Indicates the width distance between the i-th pixel point in the left contour of the obstacle and the original path; dz j represents the width distance between the jth pixel point in the right contour of the obstacle and the original path; L represents the total number of pixels in the left contour of the obstacle; R represents the total number of pixels in the right contour of the obstacle; δ i represents the obstacle factor of the i-th pixel in the left contour of the obstacle; δ j represents the obstacle factor of the jth pixel in the right contour of the obstacle; A represents the obstacle avoidance safety distance; Bx represents the width of the moving body; max(δ i ×dz i ) indicates that when i ranges from 1 to L, calculate δ i ×dz i , get L values, and take the maximum value of the L values; among them, the left contour of the obstacle refers to the contour located on the left side of the connection direction from the current point to the next path turning point, and the right contour of the obstacle refers to the contour located on the right side of the connection direction from the current point to the next path turning point.

2. The obstacle avoidance method based on machine vision according to claim 1, characterized in that: The method of pre-building an obstacle recognition model includes the following steps: Get the training dataset; Input the training data set into the convolutional neural network model for training to obtain the obstacle recognition model; Optimize and verify the obstacle recognition model.

3. The obstacle avoidance method based on machine vision according to claim 2, characterized in that: Methods for optimizing and verifying the obstacle recognition model include: Obtain multiple images of known obstacles as a validation set; Input the images in the verification set into the obstacle recognition model for verification and obtain the recognition results of the obstacle recognition model; Calculate the recognition accuracy of the obstacle recognition model based on the recognition result of the obstacle recognition model; The calculated recognition accuracy is compared with a preset threshold. If the calculated recognition accuracy is less than the preset threshold, the obstacle recognition model is repeatedly optimized and trained until the recognition accuracy of the obstacle recognition model is greater than the preset threshold.

4. The obstacle avoidance method based on machine vision according to claim 1, characterized in that: The method of obtaining destination information in response to a destination instruction and planning a travel path from a current location to a destination includes: Responding to the destination instruction, obtaining destination information; Return the acquired destination information to the traveling entity for confirmation to determine whether the destination information is correct; If it is correct, the travel path from the current location to the destination is planned based on the confirmed correct destination information; otherwise, the destination information is obtained again.

5. An obstacle avoidance device based on machine vision, used to execute the method according to any one of claims 1 to 4, characterized in that: Installed on the traveling body, the obstacle avoidance device includes: The path planning module is used to respond to the destination instruction, obtain the destination information, and plan the travel path from the current location to the destination; the travel body moves according to the planned travel path; An image acquisition device is used to acquire an image of the environment in front of the moving body in real time during the moving process; The obstacle information acquisition module is used to identify the environment image and obtain obstacle information through the pre-built obstacle recognition model; The path correction module is used to replan the travel path from the current location to the destination based on obstacle information.

6. The obstacle avoidance device based on machine vision according to claim 5, characterized in that: The obstacle avoidance device further comprises an operation monitoring module, which is used to monitor the operation status of the obstacle avoidance device.

7. The obstacle avoidance device based on machine vision according to claim 5, characterized in that: The path correction module includes: An acquisition module is used to obtain the position information of the next path turning node closest to the obstacle in the original travel path based on the obstacle information; The obstacle avoidance node insertion module is used to insert an obstacle avoidance node between the current point and the next path turning node closest to the obstacle.

8. The obstacle avoidance device based on machine vision according to claim 5, characterized in that: The obstacle avoidance device also includes an alarm for giving an alarm when obstacle information is obtained.

Images