Intelligent transfer robot obstacle avoidance method and system based on visual neural network

Abstract

The invention provides an intelligent transfer robot obstacle avoidance method and system based on an optic neural network, and belongs to the technical field of image sensing, and if a comparison result shows that an image change is large, it is proved that an obstacle may exist in an image picture at the moment, and at the moment, a controller can perform image recognition on an image and perform specific detection on the obstacle. According to the intelligent transfer robot obstacle avoidance method based on the visual neural network, in the periodic transfer process of the intelligent transfer robot, whether the obstacle exists or not is judged according to the pictures of the images in different periods, and the obstacle on the path does not need to be recognized by adopting an image recognition technology in real time; and the image is identified only when needed, so that the requirement on the computing power is low, and the production and use cost is saved.

Description

Technical Field

[0001] The present application relates to the field of image sensing, and in particular to an obstacle avoidance method and system for an intelligent handling robot based on a visual neural network. Background Art

[0002] In the field of modern industry and logistics, intelligent handling robots play an increasingly important role. In order to ensure that these robots operate safely and efficiently in complex environments, obstacle avoidance technology is particularly important. The intelligent handling robot obstacle avoidance system based on visual neural network can identify and process various obstacles in the surrounding environment in real time by simulating the human visual perception mechanism, thereby achieving accurate obstacle avoidance.

[0003] The core of the system is the visual neural network, which captures image information of the surrounding environment through the camera installed on the robot. These image data are transmitted to the neural network for processing. The neural network analyzes and identifies the images through a deep learning algorithm and extracts the characteristics of obstacles. Subsequently, the system calculates the position, size and movement trajectory of the obstacle based on these characteristics, and generates the optimal obstacle avoidance strategy based on the current state of the robot and the planned path.

[0004] The advantage of visual neural networks lies in their powerful image processing and adaptive learning capabilities. Compared with traditional obstacle avoidance systems, systems based on visual neural networks can better handle complex and changing environments, identify obstacles of various shapes and colors, and adapt to environmental changes to continuously optimize obstacle avoidance effects. In addition, the system also has the characteristics of high real-time performance and high accuracy, which can effectively reduce accidents and losses caused by improper obstacle avoidance.

[0005] However, for intelligent transport robots that operate periodically, they use visual neural networks to constantly identify surrounding images, which requires a lot of computing power. The processing equipment required by intelligent transport robots has higher specifications, which makes the cost higher. Summary of the invention

[0006] The embodiments of the present application provide an intelligent handling robot obstacle avoidance method and system based on a visual neural network to improve the above-mentioned problems.

[0007] In order to achieve the above objectives, this application adopts the following technical solutions:

[0008] In a first aspect, an embodiment of the present application proposes an intelligent handling robot obstacle avoidance method based on a visual neural network, which is applicable to an intelligent handling robot obstacle avoidance system based on a visual neural network. The intelligent handling robot obstacle avoidance system based on a visual neural network includes a controller and an intelligent handling robot. The intelligent handling robot includes an image acquisition component. The method is applicable to the controller, including:

[0009] The controller acquires m frames of detection images of the intelligent transport robot in the first transport cycle T1, which are sorted by acquisition time based on the image acquisition component, and are the first frame, the second frame...the mth frame, respectively. The detection images are images of the intelligent transport robot when it moves on the transport path;

[0010] The controller continuously acquires a plurality of frames of sequential target images based on the image acquisition component in the second handling period T2, and determines the corresponding relationship between each frame of the target image and the detection image;

[0011] When the controller acquires the nth frame of the target image, the controller compares the nth frame of the detection image with the nth frame of the target image, and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n;

[0012] If the comparison parameters meet the preset conditions, the controller performs image recognition on the target image of the nth frame, and confirms whether there are obstacles on the transport path according to the result of image recognition;

[0013] If the controller determines that there is an obstacle, it controls the intelligent transport robot to execute an obstacle avoidance program.

[0014] In combination with the first aspect, in some embodiments, the controller continuously acquires multiple frames of sequential target images in the second handling period T2 based on the image acquisition component, and determines the corresponding relationship between each frame of the target image and the detection image, including:

[0015] The controller acquires a total of m frames of target images in the second transport cycle T2, and each frame of the detection image corresponds to each frame of the target image in a one-to-one correspondence according to the time sequence of acquisition.

[0016] In combination with the first aspect, in some embodiments, the intelligent handling robot obstacle avoidance system based on the visual neural network further includes a positioning component, and the controller continuously acquires multiple frames of sequential target images in the second handling cycle based on the image acquisition component, and determines the corresponding relationship between each frame of the target image and the detection image, including:

[0017] The controller acquires m frames of detection images within the first handling cycle T1, and acquires corresponding position information of each frame of detection image based on the positioning component;

[0018] The controller determines the acquisition position of each frame of the target image based on the corresponding position information of each frame of the detection image.

[0019] In combination with the first aspect, in some embodiments, when the controller acquires the nth frame of the target image, the controller compares the nth frame of the detection image with the nth frame of the target image and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n, including:

[0020] The controller obtains a first average contrast, a first average brightness, and a first average color temperature corresponding to the entire frame of the nth frame detection image;

[0021] The controller obtains the second average contrast, the second average brightness and the second average color temperature corresponding to the entire frame of the nth frame target image;

[0022] The controller determines a contrast parameter based on a difference between the first average contrast and the second average contrast, a difference between the first average brightness and the second average brightness, and a difference between the first average color temperature and the second average color temperature.

[0023] In combination with the first aspect, in some embodiments, the controller determines the contrast parameter based on a difference between the first average contrast and the second average contrast, a difference between the first average brightness and the second average brightness, and a difference between the first average color temperature and the second average color temperature, including:

[0024] The controller obtains comparison parameters, which meet the following requirements:

[0025] D=a|x1-x2|+b|y1-y2|+c|z1-z2|;

[0026] Among them, x1 is the first average contrast, x2 is the second average contrast, y1 is the first average brightness, y2 is the second average brightness, z1 is the first average color temperature, z2 is the second average color temperature, a, b, c are different weight parameters respectively, and D is the contrast parameter.

[0027] In combination with the first aspect, in some embodiments, if the comparison parameter meets the preset condition, the controller performs image recognition on the nth frame target image, and confirms whether there is an obstacle on the transport path according to the result of the image recognition, including:

[0028] The controller obtains a preset threshold value and compares the preset threshold value with a comparison parameter;

[0029] If the comparison parameter is greater than or equal to the preset threshold, the controller performs image recognition on the nth frame target image.

[0030] In combination with the first aspect, in some implementations, if the comparison parameter is greater than or equal to a preset threshold, the controller performs image recognition on the nth frame target image, including:

[0031] The controller determines the target features and performs feature extraction on the target image of the nth frame;

[0032] The controller inputs the target features extracted from the target image of the nth frame into the classifier and obtains the output result of the classifier;

[0033] Based on the output results, the controller determines the category and location information of the obstacle in the nth frame target image.

[0034] In combination with the first aspect, in some embodiments, if the controller determines that there is an obstacle, the intelligent transport robot is controlled to execute an obstacle avoidance program, including:

[0035] If the controller determines that there is an obstacle, the controller identifies an alternative path from multiple alternative paths and uses it as a new transport path.

[0036] In the second aspect, the embodiment of the present application further proposes an intelligent handling robot obstacle avoidance system based on a visual neural network. The intelligent handling robot obstacle avoidance system based on a visual neural network includes a controller and an intelligent handling robot. The intelligent handling robot includes an image acquisition component. The system is configured as follows:

[0037] The controller acquires m frames of detection images of the intelligent transport robot in the first transport cycle T1, which are sorted by acquisition time based on the image acquisition component, and are the first frame, the second frame...the mth frame, respectively. The detection images are images of the intelligent transport robot when it moves on the transport path;

[0038] The controller continuously acquires a plurality of frames of sequential target images based on the image acquisition component in the second handling period T2, and determines the corresponding relationship between each frame of the target image and the detection image;

[0039] When the controller acquires the nth frame of the target image, the controller compares the nth frame of the detection image with the nth frame of the target image, and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n;

[0040] If the comparison parameters meet the preset conditions, the controller performs image recognition on the target image of the nth frame, and confirms whether there are obstacles on the transport path according to the result of image recognition;

[0041] If the controller determines that there is an obstacle, it controls the intelligent transport robot to execute an obstacle avoidance program.

[0042] In some embodiments, the system is configured to:

[0043] The controller continuously acquires multiple frames of sequential target images based on the image acquisition component in the second handling period T2, and determines the corresponding relationship between each frame of the target image and the detection image, including:

[0044] The controller acquires a total of m frames of target images in the second transport cycle T2, and each frame of the detection image corresponds to each frame of the target image in a one-to-one correspondence according to the time sequence of acquisition.

[0045] In some embodiments, the system is configured to:

[0046] The intelligent handling robot obstacle avoidance system based on visual neural network also includes a positioning component. The controller continuously acquires multiple frames of sequential target images in the second handling cycle based on the image acquisition component, and determines the corresponding relationship between each frame of the target image and the detection image, including:

[0047] The controller acquires m frames of detection images within the first handling cycle T1, and acquires corresponding position information of each frame of detection image based on the positioning component;

[0048] The controller determines the acquisition position of each frame of the target image based on the corresponding position information of each frame of the detection image.

[0049] In some embodiments, the system is configured to:

[0050] When the controller obtains the nth frame target image, the controller compares the nth frame detection image with the nth frame target image and obtains the comparison parameter, wherein, m and n are both natural numbers greater than zero and m is greater than n, including:

[0051] The controller obtains a first average contrast, a first average brightness, and a first average color temperature corresponding to the entire frame of the nth frame detection image;

[0052] The controller obtains the second average contrast, the second average brightness and the second average color temperature corresponding to the entire frame of the n-th frame target image;

[0053] The controller determines a contrast parameter based on a difference between the first average contrast and the second average contrast, a difference between the first average brightness and the second average brightness, and a difference between the first average color temperature and the second average color temperature.

[0054] In some embodiments, the system is configured to:

[0055] The controller determines a contrast parameter based on a difference between the first average contrast and the second average contrast, a difference between the first average brightness and the second average brightness, and a difference between the first average color temperature and the second average color temperature, including:

[0056] The controller obtains comparison parameters, which meet the following requirements:

[0057] D=a|x1-x2|+b|y1-y2|+c|z1-z2|;

[0058] Among them, x1 is the first average contrast, x2 is the second average contrast, y1 is the first average brightness, y2 is the second average brightness, z1 is the first average color temperature, z2 is the second average color temperature, a, b, c are different weight parameters respectively, and D is the contrast parameter.

[0059] In some embodiments, the system is configured to:

[0060] If the comparison parameters meet the preset conditions, the controller performs image recognition on the target image of the nth frame, and confirms whether there are obstacles on the transport path based on the image recognition results, including:

[0061] The controller obtains a preset threshold value and compares the preset threshold value with a comparison parameter;

[0062] If the comparison parameter is greater than or equal to the preset threshold, the controller performs image recognition on the nth frame target image.

[0063] In some embodiments, the system is configured to:

[0064] If the comparison parameter is greater than or equal to the preset threshold, the controller performs image recognition on the target image of the nth frame, including:

[0065] The controller determines the target features and performs feature extraction on the target image of the nth frame;

[0066] The controller inputs the target features extracted from the target image of the nth frame into the classifier and obtains the output result of the classifier;

[0067] Based on the output results, the controller determines the category and location information of the obstacle in the nth frame target image.

[0068] In some embodiments, the system is configured to:

[0069] If the controller determines that there is an obstacle, it controls the intelligent handling robot to execute the obstacle avoidance program, including:

[0070] If the controller determines that there is an obstacle, the controller identifies an alternative path from multiple alternative paths and uses it as a new transport path.

[0071] A third aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0072] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method proposed in the first aspect of the embodiment of the present invention.

[0073] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the embodiment of the present invention.

[0074] In summary, the above method and device have the following technical effects:

[0075] The embodiment of the present application proposes an obstacle avoidance method and system for an intelligent handling robot based on a visual neural network. First, the controller acquires m frames of detection images of the intelligent handling robot in the first handling cycle T1, which are sorted according to the acquisition time, based on the image acquisition component. Then, the controller continuously acquires multiple frames of sequential target images in the second handling cycle T2 based on the image acquisition component. Then, the corresponding relationship between each frame of the target image and the detection image is determined. In the second cycle, each frame of the target image is acquired and compared with the corresponding detection image. If the comparison result shows that the image changes greatly, it proves that there may be obstacles in the image at this time. At this time, the controller can perform image recognition on the image and perform specific detection on the obstacle. The obstacle avoidance method for an intelligent handling robot based on a visual neural network proposed in the present application determines whether there are obstacles according to the images of different cycles during the periodic handling of the intelligent handling robot, without using real-time image recognition technology to identify obstacles on the path. The image is only recognized when needed, which requires less computing power and saves production and use costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A flow chart of the obstacle avoidance method for an intelligent transport robot based on a visual neural network proposed in an embodiment of the present application.

[0077] Figure 2 This is a schematic diagram of the structure of the intelligent transport robot obstacle avoidance system based on visual neural network proposed in the embodiment of the present application.

[0078] Figure 3 A schematic diagram of the structure of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] See also Figure 1 The embodiment of the present application proposes an intelligent handling robot obstacle avoidance method based on a visual neural network, which is applicable to an intelligent handling robot obstacle avoidance system based on a visual neural network. Figure 2 The intelligent handling robot obstacle avoidance system based on visual neural network includes a controller and an intelligent handling robot, the intelligent handling robot includes an image acquisition component, the method is applicable to the controller, and the method includes the following steps:

[0081] S101: The controller obtains m frames of detection images of the intelligent transport robot within the first transport cycle T1 sorted by acquisition time based on the image acquisition component, which are the 1st frame, the 2nd frame...the mth frame respectively. The detection images are images of the intelligent transport robot when it moves on the transport path.

[0082] It can be understood that in this embodiment, the image acquisition component can be a camera or other image acquisition device, which is not limited in this embodiment. In this embodiment, a camera is taken as an example. In the first handling cycle T1, the camera can continuously acquire multiple frames of images, and the multiple frames of images can form a continuous video. In this embodiment, the controller acquires m frames of detection images of the intelligent handling robot in the first handling cycle T1 based on the image acquisition component, which are sorted by the acquisition time, and are the 1st frame, the 2nd frame...the mth frame respectively. The detection image is the image of the intelligent handling robot when it moves on the handling path. For example, it can be an image in front, or it can be an image facing other directions according to the handling requirements, which is not limited in this embodiment.

[0083] It should be noted that, in the present embodiment, the controller obtains m frames of detection images of the intelligent transport robot within the first transport cycle T1 sorted by acquisition time based on the image acquisition component, which are the 1st frame, the 2nd frame...the mth frame respectively. The detection image can be each frame in a continuous image, or multiple frames with the same interval in a video, which is not limited here.

[0084] In this embodiment, the first handling cycle T1 is used as the initial cycle, and the integrity of the route should be ensured as much as possible, and there should be no obstacles. For example, the first handling cycle T1 can be a cycle formed by merging the back and forth processes of the intelligent handling robot during the handling process. The intelligent handling robot also moves in the same way in the subsequent cycles. In this way, when the image obtained in the first handling cycle T1 is compared with the subsequent images, the comparison result is more obvious when the picture changes.

[0085] S102: The controller continuously acquires a plurality of frames of sequential target images in the second transport period T2 based on the image acquisition component, and determines the correspondence between each frame of the target image and the detection image.

[0086] It can be understood that in this embodiment, the second handling cycle T2 is the operation cycle when the obstacle avoidance program needs to be executed. During the second cycle operation, the controller can also obtain multiple frames of target images based on the image acquisition component, and these target images have a one-to-one correspondence with the multiple frames of detection images obtained previously.

[0087] It can be understood that, during the second transport cycle T2, if the intelligent transport robot can complete a complete cycle without executing the obstacle avoidance program, the controller acquires a total of m frames of target images during the second transport cycle T2.

[0088] Exemplarily, in this embodiment, as an implementation mode, each frame of the detection image corresponds to each frame of the target image according to the acquired position information, that is, the corresponding images are acquired at the same position. In other embodiments, each frame of the detection image corresponds to each frame of the target image according to the time sequence of acquisition, that is, as an example, when the nth frame of the detection image is acquired at the jth moment in the first cycle, the nth frame of the target image is acquired at the jth moment in the second cycle.

[0089] Specifically, as an implementation method, in this embodiment, these target images have a one-to-one correspondence with the previously acquired multiple frames of detection images, which is a corresponding positional relationship. When the images acquired at the same position do not exist, the images should remain basically consistent.

[0090] It can be understood that in this embodiment, since these target images have a one-to-one correspondence with the previously acquired multiple frames of detection images as a corresponding positional relationship, in this embodiment, the intelligent handling robot obstacle avoidance system based on the visual neural network also includes a positioning component, which is used to obtain position information. Specifically, the controller obtains m frames of detection images within the first handling cycle T1, and obtains the corresponding position information of each frame of detection image based on the positioning component. Then, the controller determines the acquisition position of each frame of target image based on the corresponding position information of each frame of detection image. Exemplarily, in the first cycle, the nth frame of detection image is acquired at point a, and in the second cycle, the nth frame of detection image is also acquired at point a. Images at the same position can ensure the consistency of the image in the absence of obstacles.

[0091] S103: When the controller obtains the nth frame target image, the controller compares the nth frame detection image with the nth frame target image and obtains comparison parameters, where m and n are both natural numbers greater than zero and m is greater than n.

[0092] It can be understood that in this embodiment, since each frame of the detection image has a one-to-one correspondence with each frame of the target image. During the periodic movement of the handling robot, when in the same position, the picture acquired by the image acquisition component should remain basically consistent, and when an obstacle appears in the image acquisition component, the picture of the image will inevitably change. Therefore, in this embodiment, the controller can use this change to achieve preliminary detection of obstacles. In this way, only when the image changes, the image recognition method is used for artificial intelligence, which saves the computing power of the controller or the image recognition module, reduces production costs, and also increases the use time.

[0093] Specifically, as an implementation, step S103 may include the following steps:

[0094] S1031: The controller obtains the first average contrast, the first average brightness, and the first average color temperature corresponding to the entire frame of the n-th frame detection image.

[0095] S1032: The controller obtains the second average contrast, the second average brightness, and the second average color temperature corresponding to the entire frame of the n-th frame target image.

[0096] It is understandable that in this embodiment, after the controller obtains the nth frame of the target image, when it compares it with the corresponding detection image, it can compare the three values ​​of average contrast, average brightness and average color temperature at the same time. For the processor or controller, directly obtaining these parameters from the image will not consume a large amount of computing power. Exemplarily, when obtaining the average contrast, it is achieved by calculating the difference between each pixel in the image and its neighboring pixels. Common local contrast calculation methods include Laplace operator, Sobel operator, etc. These methods extract edge information in the image by performing convolution operations on the image. Of course, the global contrast of the sub-image can also be obtained by calculating the difference between the maximum and minimum pixel values ​​of the image. Combining the local average contrast with the global contrast can more comprehensively describe the contrast characteristics of the image. The corresponding methods can also be used for the average brightness and average color temperature, which will not be repeated in this embodiment.

[0097] S1033: The controller determines a contrast parameter based on a difference between the first average contrast and the second average contrast, a difference between the first average brightness and the second average brightness, and a difference between the first average color temperature and the second average color temperature.

[0098] It is understandable that in order to describe the entire picture with one number, that is, the contrast parameter, in this embodiment, the following method can be used. Exemplarily, in this embodiment, certain weights can be given to the three different indexes of contrast, color temperature and brightness, and then the actual data and the product of the corresponding weights can be integrated to achieve it.

[0099] Specifically, in this embodiment, the controller obtains comparison parameters, and the comparison parameters satisfy:

[0100] D=a|x1-x2|+b|y1-y2|+c|z1-z2|;

[0101] Among them, x1 is the first average contrast, x2 is the second average contrast, y1 is the first average brightness, y2 is the second average brightness, z1 is the first average color temperature, z2 is the second average color temperature, a, b, c are different weight parameters, and D is the contrast parameter. It can be understood that the contrast parameter obtained by the above method can be used to measure the degree of change of the overall image. When the overall image changes greatly, such as when obstacles appear, the corresponding contrast parameter is also larger.

[0102] S104: If the comparison parameters meet the preset conditions, the controller performs image recognition on the nth frame target image, and confirms whether there are obstacles on the transport path according to the result of the image recognition.

[0103] It can be understood that in the above steps, after obtaining the contrast parameter, the larger the contrast parameter, the greater the image change. Therefore, a preset threshold can be set in advance, and then the preset threshold is compared with the contrast parameter. If the contrast parameter is greater than or equal to the preset threshold, the controller performs image recognition on the target image of the nth frame, which proves that the image change has reached a certain extent and there may be an obstacle. At this time, image recognition technology can be enabled to identify the obstacle.

[0104] It can be understood that in this embodiment, after the nth frame image is determined to have a large change compared with the previous cycle, the frame image can be subjected to image recognition based on a neural network, which can make the result more accurate and further confirm whether there is an obstacle image in the image. Exemplarily, in this embodiment, step S105 may include the following steps:

[0105] S1051: The controller determines the target features and performs feature extraction on the nth frame target image;

[0106] S1052: The controller inputs the target features extracted from the nth frame target image into the classifier and obtains the output result of the classifier.

[0107] S1053: Based on the output result, the controller determines the category and location information of the obstacle in the nth frame target image.

[0108] As can be understood, the controller will first identify and determine the target features. Next, it will perform detailed feature extraction operations on the target image in the nth frame to better understand and analyze the image content. After completing the feature extraction, the controller will input the target features extracted from the target image in the nth frame into the classifier. The classifier will then analyze and process these features and finally output a result, which the controller will obtain and use for the next step of processing. Based on the output results provided by the classifier, the controller will further determine the specific category and location information of the obstacle in the nth frame target image. This process involves detailed analysis and recognition of the image to ensure that the type of obstacle and its specific location in the image can be accurately identified, thereby providing reliable data support for subsequent processing and decision-making.

[0109] In some other embodiments, other image recognition methods may be used, which are not limited in this embodiment.

[0110] S105: If the controller determines that there is an obstacle, it controls the intelligent transport robot to execute an obstacle avoidance program.

[0111] It is understandable that after determining that there is an obstacle in the image, the controller can automatically plan the route according to the shape and size of the obstacle, and also identify an alternative path from multiple alternative paths as a new transport path. How to plan or select a route has been disclosed in the relevant technology and is not limited in this embodiment.

[0112] The embodiment of the present application proposes an obstacle avoidance method for an intelligent handling robot based on a visual neural network. First, the controller acquires m frames of detection images of the intelligent handling robot in the first handling cycle T1 sorted by the acquisition time based on the image acquisition component. Then, the controller continuously acquires multiple frames of sequential target images based on the image acquisition component in the second handling cycle T2. Then, the corresponding relationship between each frame of the target image and the detection image is determined. In the second cycle, each frame of the target image is acquired and compared with the corresponding detection image. If the comparison result shows that the image changes greatly, it proves that there may be obstacles in the image at this time. At this time, the controller can perform image recognition on the image and perform specific detection on the obstacle. The obstacle avoidance method for an intelligent handling robot based on a visual neural network proposed in the present application determines whether there are obstacles according to the images of different cycles during the periodic handling of the intelligent handling robot, without using real-time image recognition technology to identify obstacles on the path. The image is only recognized when needed, which requires less computing power and saves production and use costs.

[0113] Based on the same inventive concept, the embodiment of the present application also proposes an intelligent handling robot obstacle avoidance system based on a visual neural network. The intelligent handling robot obstacle avoidance system based on a visual neural network includes a controller and an intelligent handling robot. The intelligent handling robot includes an image acquisition component. The system is configured as follows:

[0114] The controller acquires m frames of detection images of the intelligent transport robot in the first transport cycle T1, which are sorted by acquisition time based on the image acquisition component, and are the first frame, the second frame...the mth frame, respectively. The detection images are images of the intelligent transport robot when it moves on the transport path;

[0115] The controller continuously acquires a plurality of frames of sequential target images based on the image acquisition component in the second handling period T2, and determines the corresponding relationship between each frame of the target image and the detection image;

[0116] When the controller acquires the nth frame of the target image, the controller compares the nth frame of the detection image with the nth frame of the target image, and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n;

[0117] If the comparison parameters meet the preset conditions, the controller performs image recognition on the target image of the nth frame, and confirms whether there are obstacles on the transport path according to the result of image recognition;

[0118] If the controller determines that there is an obstacle, it controls the intelligent transport robot to execute an obstacle avoidance program.

[0119] In some embodiments, the system is configured to:

[0120] The controller continuously acquires multiple frames of sequential target images based on the image acquisition component in the second handling period T2, and determines the corresponding relationship between each frame of the target image and the detection image, including:

[0121] The controller acquires a total of m frames of target images in the second transport cycle T2, and each frame of the detection image corresponds to each frame of the target image in a one-to-one correspondence according to the time sequence of acquisition.

[0122] In some embodiments, the system is configured to:

[0123] The intelligent handling robot obstacle avoidance system based on visual neural network also includes a positioning component. The controller continuously acquires multiple frames of sequential target images in the second handling cycle based on the image acquisition component, and determines the corresponding relationship between each frame of the target image and the detection image, including:

[0124] The controller acquires m frames of detection images within the first handling cycle T1, and acquires corresponding position information of each frame of detection image based on the positioning component;

[0125] The controller determines the acquisition position of each frame of the target image based on the corresponding position information of each frame of the detection image.

[0126] In some embodiments, the system is configured to:

[0127] When the controller obtains the nth frame target image, the controller compares the nth frame detection image with the nth frame target image and obtains the comparison parameter, wherein, m and n are both natural numbers greater than zero and m is greater than n, including:

[0128] The controller obtains a first average contrast, a first average brightness, and a first average color temperature corresponding to the entire frame of the nth frame detection image;

[0129] The controller obtains the second average contrast, the second average brightness and the second average color temperature corresponding to the entire frame of the n-th frame target image;

[0130] The controller determines a contrast parameter based on a difference between the first average contrast and the second average contrast, a difference between the first average brightness and the second average brightness, and a difference between the first average color temperature and the second average color temperature.

[0131] In some embodiments, the system is configured to:

[0132] The controller determines a contrast parameter based on a difference between the first average contrast and the second average contrast, a difference between the first average brightness and the second average brightness, and a difference between the first average color temperature and the second average color temperature, including:

[0133] The controller obtains comparison parameters, which meet the following requirements:

[0134] D=a|x1-x2|+b|y1-y2|+c|z1-z2|;

[0135] Among them, x1 is the first average contrast, x2 is the second average contrast, y1 is the first average brightness, y2 is the second average brightness, z1 is the first average color temperature, z2 is the second average color temperature, a, b, c are different weight parameters respectively, and D is the contrast parameter.

[0136] In some embodiments, the system is configured to:

[0137] If the comparison parameters meet the preset conditions, the controller performs image recognition on the target image of the nth frame, and confirms whether there are obstacles on the transport path based on the image recognition results, including:

[0138] The controller obtains a preset threshold value and compares the preset threshold value with a comparison parameter;

[0139] If the comparison parameter is greater than or equal to the preset threshold, the controller performs image recognition on the nth frame target image.

[0140] In some embodiments, the system is configured to:

[0141] If the comparison parameter is greater than or equal to the preset threshold, the controller performs image recognition on the target image of the nth frame, including:

[0142] The controller determines the target features and performs feature extraction on the target image of the nth frame;

[0143] The controller inputs the target features extracted from the target image of the nth frame into the classifier and obtains the output result of the classifier;

[0144] Based on the output results, the controller determines the category and location information of the obstacle in the nth frame target image.

[0145] In some embodiments, the system is configured to:

[0146] If the controller determines that there is an obstacle, it controls the intelligent handling robot to execute the obstacle avoidance program, including:

[0147] If the controller determines that there is an obstacle, the controller identifies an alternative path from multiple alternative paths and uses it as a new transport path.

[0148] The embodiment of the present application proposes an intelligent handling robot obstacle avoidance system based on a visual neural network. First, the controller acquires m frames of detection images of the intelligent handling robot in the first handling cycle T1 sorted by the acquisition time based on the image acquisition component. Then, the controller continuously acquires multiple frames of sequential target images in the second handling cycle T2 based on the image acquisition component. Then, the corresponding relationship between each frame of the target image and the detection image is determined. In the second cycle, each frame of the target image is acquired and compared with the corresponding detection image. If the comparison result shows that the image changes greatly, it proves that there may be obstacles in the image at this time. At this time, the controller can perform image recognition on the image and perform specific detection on the obstacle. The intelligent handling robot obstacle avoidance system based on a visual neural network proposed in the present application, in the process of periodic handling of the intelligent handling robot, judges whether there are obstacles according to the images of different cycles, without using real-time image recognition technology to identify obstacles on the path. The image is only recognized when needed, which requires less computing power and saves production and use costs.

[0149] Based on the same inventive concept, the embodiment of the present application also proposes an electronic device, see Figure 3 , electronic equipment includes:

[0150] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the intelligent handling robot obstacle avoidance method based on a visual neural network according to an embodiment of the present application.

[0151] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the intelligent transport robot obstacle avoidance method based on a visual neural network in the embodiment of the present application is implemented.

[0152] The following is a detailed introduction to the various components of electronic equipment:

[0153] The processor is the control center of the electronic device, which may be a processor or a general term for multiple processing elements. For example, the processor is one or more central processing units (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA).

[0154] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0155] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0156] Optionally, the memory may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be integrated with the processor, or may exist independently and be coupled to the processor through an interface circuit of the electronic device, which is not specifically limited in the embodiments of the present invention.

[0157] A transceiver is used to communicate with a network device or a terminal device.

[0158] Optionally, the transceiver may include a receiver and a transmitter, wherein the receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0159] Optionally, the transceiver may be integrated with the processor, or may exist independently and be coupled to the processor via an interface circuit of the router, which is not specifically limited in the embodiment of the present invention.

[0160] In addition, the technical effects of the electronic device can refer to the technical effects of the data transmission method in the above method embodiment, which will not be repeated here.

[0161] It should be understood that the processor in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0162] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0163] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0164] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0165] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0166] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0167] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

Claims

1. An intelligent handling robot obstacle avoidance method based on visual neural network, characterized in that: The invention is applicable to an intelligent handling robot obstacle avoidance system based on a visual neural network, wherein the intelligent handling robot obstacle avoidance system based on a visual neural network comprises a controller and an intelligent handling robot, wherein the intelligent handling robot comprises an image acquisition component, and the method is applicable to the controller, comprising: The controller acquires, based on the image acquisition component, m frames of detection images of the intelligent transport robot in the first transport cycle T1, which are sorted by acquisition time, namely the 1st frame, the 2nd frame...the mth frame, and the detection images are images of the intelligent transport robot when it moves on the transport path; The controller continuously acquires a plurality of frames of sequential target images based on the image acquisition component in the second handling period T2, and determines the corresponding relationship between each frame of the target image and the detection image; When the controller acquires the target image of the nth frame, the controller compares the detection image of the nth frame with the target image of the nth frame, and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n; If the comparison parameter meets the preset condition, the controller performs image recognition on the target image of the nth frame, and confirms whether there is an obstacle on the transport path according to the result of the image recognition; If the controller determines that there is an obstacle, the controller controls the intelligent transport robot to execute an obstacle avoidance program.

2. According to the obstacle avoidance method of an intelligent handling robot based on a visual neural network as described in claim 1, it is characterized in that: The controller continuously acquires a plurality of frames of sequential target images based on the image acquisition component in the second handling period T2, and determines the corresponding relationship between each frame of the target image and the detection image, including: The controller acquires m frames of the target image in the second transport period T2, and each frame of the detection image corresponds to each frame of the target image in a one-to-one manner according to the time sequence of acquisition.

3. The obstacle avoidance method of an intelligent handling robot based on a visual neural network according to claim 1, characterized in that: The intelligent handling robot obstacle avoidance system based on visual neural network also includes a positioning component, and the controller continuously acquires multiple frames of sequential target images in the second handling cycle based on the image acquisition component, and determines the corresponding relationship between each frame of the target image and the detection image, including: The controller acquires m frames of the detection image within the first transport cycle T1, and acquires corresponding position information of each frame of the detection image based on the positioning component; The controller determines the acquisition position of each frame of the target image based on the corresponding position information of each frame of the detection image.

4. The obstacle avoidance method of an intelligent handling robot based on a visual neural network according to claim 3 is characterized in that: When the controller acquires the target image of the nth frame, the controller compares the detection image of the nth frame with the target image of the nth frame and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n, including: The controller obtains a first average contrast, a first average brightness, and a first average color temperature corresponding to the entire frame of the detection image of the nth frame; The controller obtains a second average contrast, a second average brightness, and a second average color temperature corresponding to the entire frame of the target image of the nth frame; The controller determines the contrast parameter based on a difference between the first average contrast and the second average contrast, a difference between the first average brightness and the second average brightness, and a difference between the first average color temperature and the second average color temperature.

5. The obstacle avoidance method of an intelligent handling robot based on a visual neural network according to claim 4 is characterized in that: The controller determines the contrast parameter based on a difference between the first average contrast and the second average contrast, a difference between the first average brightness and the second average brightness, and a difference between the first average color temperature and the second average color temperature, including: The controller obtains the comparison parameter, and the comparison parameter satisfies: D=a|x1-x2|+b|y1-y2|+c|z1-z2|; Among them, x1 is the first average contrast, x2 is the second average contrast, y1 is the first average brightness, y2 is the second average brightness, z1 is the first average color temperature, z2 is the second average color temperature, a, b, c are different weight parameters respectively, and D is the contrast parameter.

6. The obstacle avoidance method of an intelligent handling robot based on a visual neural network according to claim 5, characterized in that: If the comparison parameter meets the preset condition, the controller performs image recognition on the target image of the nth frame, and confirms whether there is an obstacle on the transport path according to the result of the image recognition, including: The controller obtains a preset threshold value, and compares the preset threshold value with the comparison parameter; If the comparison parameter is greater than or equal to the preset threshold, the controller performs image recognition on the target image of the nth frame.

7. The obstacle avoidance method of an intelligent handling robot based on a visual neural network according to claim 6, characterized in that: If the comparison parameter is greater than or equal to the preset threshold, the controller performs image recognition on the target image of the nth frame, including: The controller determines the target features and performs feature extraction on the target image of the nth frame; The controller inputs the target features extracted from the target image of the nth frame into the classifier, and obtains the output result of the classifier; The controller determines the category and location information of the obstacle in the target image of the nth frame based on the output result.

8. The obstacle avoidance method of an intelligent handling robot based on a visual neural network according to claim 7, characterized in that: If the controller determines that there is an obstacle, the intelligent transport robot is controlled to execute an obstacle avoidance program, including: If the controller determines that there is an obstacle, the controller identifies one of the candidate paths from a plurality of candidate paths and uses it as the new transport path.

9. An intelligent handling robot obstacle avoidance system based on visual neural network, characterized in that: The intelligent handling robot obstacle avoidance system based on visual neural network includes a controller and an intelligent handling robot, wherein the intelligent handling robot includes an image acquisition component, and the system is configured as follows: The controller acquires, based on the image acquisition component, m frames of detection images of the intelligent transport robot in the first transport cycle T1, which are sorted by acquisition time, namely the 1st frame, the 2nd frame...the mth frame, and the detection images are images of the intelligent transport robot when it moves on the transport path; The controller continuously acquires a plurality of frames of sequential target images based on the image acquisition component in the second handling period T2, and determines the corresponding relationship between each frame of the target image and the detection image; When the controller acquires the target image of the nth frame, the controller compares the detection image of the nth frame with the target image of the nth frame, and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n; If the comparison parameter meets the preset condition, the controller performs image recognition on the target image of the nth frame, and confirms whether there is an obstacle on the transport path according to the result of the image recognition; If the controller determines that there is an obstacle, the controller controls the intelligent transport robot to execute an obstacle avoidance program.

10. An electronic device, comprising: at least one processor; And, a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by at least one of the processors, and the instructions are executed by at least one of the processors so that at least one of the processors can execute the method proposed in any one of claims 1-8.

Images