Obstacle avoidance method and system for intelligent carrying robot based on visual neural network

Abstract

The application provides an obstacle avoidance method and system for an intelligent carrying robot based on a visual neural network, and belongs to the technical field of image sensing. If the comparison result shows that the image changes greatly, it proves that there may be an obstacle in the image picture at this time, and the controller can perform image recognition on the image and specifically detect the obstacle. The application provides an obstacle avoidance method for an intelligent carrying robot based on a visual neural network. In the process of periodic carrying of the intelligent carrying robot, whether there is an obstacle is judged according to the pictures of images of different periods, and the image recognition technology is not used in real time to recognize the obstacles on the path. The image is recognized only when needed, the demand for computing power is small, and the production and use costs are saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image sensing, in particular to a vision neural network-based intelligent carrying robot obstacle avoidance method and system. BACKGROUND

[0002] In modern industry and logistics fields, intelligent carrying robots play an increasingly important role. In order to ensure that these robots operate safely and efficiently in complex environments, obstacle avoidance technology is particularly important. The vision neural network-based intelligent carrying robot obstacle avoidance system can identify and process various obstacles in the surrounding environment in real time by simulating human visual perception mechanisms, thereby achieving precise obstacle avoidance.

[0003] The core of the system is the vision neural network, which captures image information of the surrounding environment through a camera installed on the robot. These image data are transmitted to the neural network for processing. The neural network analyzes and identifies the images through deep learning algorithms and extracts the features of the obstacles. Subsequently, the system calculates the position, size and motion trajectory of the obstacles based on these features, and generates the optimal obstacle avoidance strategy in combination with the current state of the robot and the predetermined path.

[0004] The vision neural network has the advantages of powerful image processing capability and adaptive learning ability. Compared with traditional obstacle avoidance systems, the vision neural network-based system can better handle complex and variable environments, identify obstacles of various shapes and colors, and adapt to environmental changes to continuously optimize obstacle avoidance effects. In addition, the system has the characteristics of high real-time performance and high accuracy, which can effectively reduce accidents and losses caused by improper obstacle avoidance.

[0005] However, for periodically operating intelligent carrying robots, the vision neural network is used to identify the surrounding images at all times, which requires a large amount of computing power, and the intelligent carrying robot requires a high-spec processing device, resulting in high cost. SUMMARY

[0006] The vision neural network-based intelligent carrying robot obstacle avoidance method and system provided by the embodiments of the present application can improve the above problems.

[0007] To achieve the above purpose, the technical solutions adopted by the present application are as follows:

[0008] In a first aspect, the embodiments of the present application propose a vision neural network-based intelligent carrying robot obstacle avoidance method, which is applicable to a vision neural network-based intelligent carrying robot obstacle avoidance system. The vision neural network-based intelligent carrying robot obstacle avoidance system includes a controller and an intelligent carrying robot. The intelligent carrying robot includes an image acquisition component. The method is applicable to the controller and includes:

[0009] The controller obtains m frames of detection images in the first carrying period T1 in time sequence based on the image acquisition component, which are the 1st frame, the 2nd frame,..., and the mth frame, and the detection images are images of the intelligent carrying robot moving on the carrying path;

[0010] The controller continuously obtains multiple target images in sequence in the second carrying period T2 based on the image acquisition component, and determines the correspondence between each target image and the detection image;

[0011] When the controller obtains the nth target image, the controller compares the nth detection image with the nth target image and obtains a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n;

[0012] If the comparison parameter meets a preset condition, the controller performs image recognition on the nth target image, and determines whether there is an obstacle on the carrying path according to the result of the image recognition;

[0013] If the controller determines that there is an obstacle, the controller controls the intelligent carrying robot to execute an obstacle avoidance program.

[0014] In some embodiments of the first aspect, the controller continuously obtains multiple target images in sequence in the second carrying period T2 based on the image acquisition component, and determines the correspondence between each target image and the detection image, including:

[0015] The controller obtains m target images in the second carrying period T2, and each detection image corresponds to each target image in time sequence.

[0016] In some embodiments of the first aspect, the intelligent carrying robot obstacle avoidance system based on the visual neural network further comprises a positioning component, and the controller continuously obtains multiple target images in sequence in the second carrying period based on the image acquisition component, and determines the correspondence between each target image and the detection image, including:

[0017] The controller obtains m detection images in the first carrying period T1, and obtains corresponding position information of each detection image based on the positioning component;

[0018] The controller determines the acquisition position of each target image based on the corresponding position information of each detection image.

[0019] In some embodiments of the first aspect, when the controller obtains the nth target image, the controller compares the nth detection image with the nth target image and obtains a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n, including:

[0020] The controller obtains a first average contrast, a first average brightness and a first average color temperature corresponding to the entire frame of the nth frame of detection image;

[0021] The controller obtains a second average contrast, a second average brightness and a second average color temperature corresponding to the entire frame of the nth frame of target image;

[0022] The controller determines a contrast parameter based on the difference between the first average contrast and the second average contrast, the difference between the first average brightness and the second average brightness, and the difference between the first average color temperature and the second average color temperature.

[0023] In combination with the first aspect, in some embodiments, the controller determines the contrast parameter based on the difference between the first average contrast and the second average contrast, the difference between the first average brightness and the second average brightness, and the difference between the first average color temperature and the second average color temperature, including:

[0024] The controller obtains the contrast parameter, and the contrast parameter satisfies:

[0025] D=a|x1-x2|+b|y1-y2|+c|z1-z2|;

[0026] wherein x1 is the first average contrast, x2 is the second average contrast, y1 is the first average brightness, y2 is the second average brightness, z1 is the first average color temperature, z2 is the second average color temperature, a, b and c are different weight parameters, and D is the contrast parameter.

[0027] In combination with the first aspect, in some embodiments, if the contrast parameter satisfies a preset condition, the controller performs image recognition on the nth frame of target image, and determines whether there is an obstacle on the conveying path according to the result of the image recognition, including:

[0028] The controller obtains a preset threshold and compares the preset threshold with the contrast parameter;

[0029] If the contrast parameter is greater than or equal to the preset threshold, the controller performs image recognition on the nth frame of target image.

[0030] In combination with the first aspect, in some embodiments, if the contrast parameter is greater than or equal to the preset threshold, the controller performs image recognition on the nth frame of target image, including:

[0031] The controller determines a target feature and performs feature extraction on the nth frame of target image;

[0032] The controller inputs the target feature extracted from the nth frame of target image into a classifier and obtains an output result of the classifier;

[0033] The controller determines the category and position information of the obstacle in the nth frame of target image based on the output result.

[0034] In combination with the first aspect, in some embodiments, if the controller determines that there is an obstacle, the controller controls the intelligent carrying robot to perform an obstacle avoidance program, including:

[0035] If the controller determines that there is an obstacle, the controller confirms one of the multiple alternative paths as a new carrying path.

[0036] In a second aspect, the embodiments of the present application also provide an intelligent carrying robot obstacle avoidance system based on a visual neural network, which comprises a controller and an intelligent carrying robot, and the intelligent carrying robot comprises an image acquisition component, and the system is configured to:

[0037] The controller acquires, based on the image acquisition component, m frames of detection images in a first carrying period T1 in the order of acquisition time, which are the 1st frame, the 2nd frame,..., and the mth frame, and the detection images are images of the intelligent carrying robot when moving on the carrying path;

[0038] The controller continuously acquires, based on the image acquisition component, multiple target images with sequence in a second carrying period T2, and determines the correspondence between each target image and the detection image;

[0039] When the controller acquires the nth target image, the controller compares the nth detection image with the nth target image, and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n;

[0040] If the comparison parameter meets a preset condition, the controller performs image recognition on the nth target image, and according to the result of the image recognition, confirms whether there is an obstacle on the carrying path;

[0041] If the controller determines that there is an obstacle, the controller controls the intelligent carrying robot to perform an obstacle avoidance program.

[0042] In some embodiments, the system is configured to:

[0043] The controller continuously acquires, based on the image acquisition component, multiple target images with sequence in a second carrying period T2, and determines the correspondence between each target image and the detection image, including:

[0044] The controller acquires m target images in the second carrying period T2, and each detection image corresponds to each target image in the order of acquisition time.

[0045] In some embodiments, the system is configured to:

[0046] The obstacle avoidance system of the intelligent carrying robot based on the visual neural network further comprises a positioning component. The controller continuously acquires a plurality of target images with sequences in the second carrying period based on the image acquisition component, and determines the corresponding relationship between each target image and the detection image, including:

[0047] The controller acquires m frames of detection images in the first carrying period T1, and acquires corresponding position information of each frame of detection images based on the positioning component;

[0048] The controller determines the acquisition position of each frame of target images based on the corresponding position information of each frame of detection images.

[0049] In some embodiments, the system is configured to:

[0050] When the controller acquires the nth frame of target images, the controller compares the nth frame of detection images with the nth frame of target images, and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n, including:

[0051] The controller acquires a first average contrast, a first average brightness, and a first average color temperature corresponding to the entire frame of the nth frame of detection images;

[0052] The controller acquires a second average contrast, a second average brightness, and a second average color temperature corresponding to the entire frame of the nth frame of target images;

[0053] The controller determines the comparison parameter based on the difference between the first average contrast and the second average contrast, the difference between the first average brightness and the second average brightness, and the difference between the first average color temperature and the second average color temperature.

[0054] In some embodiments, the system is configured to:

[0055] The controller determines the comparison parameter based on the difference between the first average contrast and the second average contrast, the difference between the first average brightness and the second average brightness, and the difference between the first average color temperature and the second average color temperature, including:

[0056] The controller acquires the comparison parameter, and the comparison parameter satisfies:

[0057] D = a|x1-x2|+b|y1-y2|+c|z1-z2|;

[0058] Wherein x1 is the first average contrast, x2 is the second average contrast, y1 is the first average brightness, y2 is the second average brightness, z1 is the first average color temperature, z2 is the second average color temperature, a, b, and c are different weight parameters, and D is the comparison parameter.

[0059] In some embodiments, the system is configured to:

[0060] If the comparison parameter meets the preset condition, the controller performs image recognition on the nth target image, and according to the result of the image recognition, confirms whether there is an obstacle on the carrying path, including:

[0061] The controller acquires a preset threshold, and compares the preset threshold with the comparison parameter;

[0062] If the comparison parameter is greater than or equal to the preset threshold, the controller performs image recognition on the nth target image.

[0063] In some embodiments, the system is configured to:

[0064] If the comparison parameter is greater than or equal to the preset threshold, the controller performs image recognition on the nth target image, including:

[0065] The controller determines a target feature, and performs feature extraction on the nth target image;

[0066] The controller inputs the target feature extracted from the nth target image into a classifier, and acquires an output result of the classifier;

[0067] The controller determines the category and location information of the obstacle in the nth target image based on the output result.

[0068] In some embodiments, the system is configured to:

[0069] If the controller determines that there is an obstacle, the controller controls the intelligent carrying robot to perform an obstacle avoidance program, including:

[0070] If the controller determines that there is an obstacle, the controller confirms an alternative path from a plurality of alternative paths as a new carrying path.

[0071] The third aspect of the embodiment of the present application provides an electronic device, which comprises:

[0072] At least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method provided in the first aspect of the embodiment of the present application.

[0073] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method provided in the first aspect of the embodiment of the present application.

[0074] In summary, the above method and device have the following technical effects:

[0075] The embodiment of the application provides an obstacle avoidance method and system of an intelligent carrying robot based on a visual neural network. First, a controller obtains m frames of detection images in a first carrying period T1 based on an image acquisition component, and then the controller continuously obtains multiple frames of target images with sequence in a second carrying period T2 based on the image acquisition component. Then, the corresponding relationship between each frame of target image and detection image is determined, and in the second period, the target image is compared with the corresponding detection image every time a frame of target image is obtained. If the comparison result shows that the image changes greatly, it is proved that there may be an obstacle in the image picture at this time. At this time, the controller can perform image recognition on the image and specifically detect the obstacle. The obstacle avoidance method of the intelligent carrying robot based on the visual neural network provided by the application can judge whether there is an obstacle according to the pictures of images in different periods in the process of periodic carrying of the intelligent carrying robot, without using real-time image recognition technology to recognize the obstacles on the path, and only the image is recognized when needed. The demand for computing power is small, and the production and use cost is saved. BRIEF DESCRIPTION OF DRAWINGS

[0076] Figure 1 The flowchart of the obstacle avoidance method of the intelligent carrying robot based on the visual neural network provided by the embodiment of the application.

[0077] Figure 2 The structural diagram of the obstacle avoidance system of the intelligent carrying robot based on the visual neural network provided by the embodiment of the application.

[0078] Figure 3 The structural diagram of the electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0080] Please refer to Figure 1 The embodiment of the application provides an obstacle avoidance method of an intelligent carrying robot based on a visual neural network, which is suitable for an obstacle avoidance system of an intelligent carrying robot based on a visual neural network. Please refer to Figure 2 The obstacle avoidance system of the intelligent carrying robot based on the visual neural network includes a controller and an intelligent carrying robot. The intelligent carrying robot includes an image acquisition component. The method is suitable for the controller, and includes the following steps.

[0081] S101: The controller obtains m frames of detection images in the first carrying period T1 according to the acquisition time based on the image acquisition component, which are respectively the first frame, the second frame, and the mth frame. The detection images are images of the intelligent carrying robot moving on the carrying path.

[0082] It can be understood that in the embodiment, the image acquisition component can be a camera or other image acquisition device, which is not limited in the embodiment. In the embodiment, the camera is taken as an example. In the first carrying period T1, the camera can continuously acquire multiple frames of images, which can constitute a continuous video. In the embodiment, the controller obtains m frames of detection images in the first carrying period T1 according to the acquisition time based on the image acquisition component, which are respectively the first frame, the second frame, and the mth frame. The detection images are images of the intelligent carrying robot moving on the carrying path. For example, the images can be front images or images in other directions according to the carrying demand, which is not limited in the embodiment.

[0083] It should be noted that in the embodiment, the controller obtains m frames of detection images in the first carrying period T1 according to the acquisition time based on the image acquisition component, which are respectively the first frame, the second frame, and the mth frame. The detection images can be each frame of picture in the continuous images or multiple frames of pictures with the same interval in a video, which is not limited herein.

[0084] In the embodiment, the first carrying period T1 is an initial period, which should be as complete as possible and without obstacles. For example, the first carrying period T1 can be a period obtained by combining the forward and backward processes of the intelligent carrying robot in the carrying process. The intelligent carrying robot also moves in the same manner in the subsequent periods. In this way, when the images obtained in the first carrying period T1 are compared with the subsequent images, the comparison result is more obvious when the picture changes.

[0085] S102: The controller continuously obtains multiple frames of target images with sequence based on the image acquisition component in the second carrying period T2, and determines the corresponding relationship between each frame of target image and detection image.

[0086] It can be understood that in the embodiment, the second carrying period T2 is a running period when the obstacle avoidance program needs to be executed. In the second period running process, the controller can also obtain multiple frames of target images based on the image acquisition component. These target images have a one-to-one corresponding relationship with the previously obtained multiple frames of detection images.

[0087] It can be understood that, in the second carrying period T2, if the intelligent carrying robot can run a complete cycle without executing the obstacle avoidance program, the controller obtains m frames of target images in the second carrying period T2.

[0088] For example, in this embodiment, as an implementation, each frame of detection image and each frame of target image correspond to each other according to the position information obtained, that is, the corresponding images are obtained at the same position. In other embodiments, each frame of detection image and each frame of target image correspond to each other according to the time sequence of the obtained, that is, as an example, the nth frame of detection image is obtained at the jth moment of the first period, and the nth frame of target image is obtained at the jth moment of the second period.

[0089] Specifically, as an implementation, in this embodiment, the one-to-one correspondence between the target images and the previously obtained multiple frames of detection images is a corresponding position relationship, that is, the images obtained at the same position should be basically consistent in the absence of obstacles.

[0090] It can be understood that, in this embodiment, since the one-to-one correspondence between the target images and the previously obtained multiple frames of detection images is a corresponding position relationship, in this embodiment, the intelligent carrying robot obstacle avoidance system based on the visual neural network further comprises a positioning component for obtaining position information. Specifically, the controller obtains m frames of detection images within the first carrying period T1, and obtains the corresponding position information of each frame of detection image based on the positioning component, and then determines the acquisition position of each frame of target image based on the corresponding position information of each frame of detection image. For example, in the first period, the nth frame of detection image is obtained at point a, and in the second period, the nth frame of detection image is also obtained at point a. The images at the same position can ensure the consistency of the images in the absence of obstacles.

[0091] S103: When the controller obtains the nth frame of target image, the controller compares the nth frame of detection image with the nth frame of target image, and obtains a comparison parameter, wherein, m and n are both natural numbers greater than zero and m is greater than n.

[0092] It can be understood that, in the embodiment, since each frame of the detection image has a one-to-one corresponding relationship with each frame of the target image. In the process of periodic movement of the carrying robot, the image acquisition assembly should obtain a picture that is basically consistent at the same position, and when an obstacle appears in the image acquisition assembly, the picture of the image will inevitably change. Therefore, in the embodiment, the controller can use the change to realize preliminary detection of the obstacle. In this way, image recognition is only performed in the image change by using artificial intelligence, which saves the computing power of the controller or the image recognition module, reduces the production cost, and also improves the use time.

[0093] Specifically, as an implementation manner, the step S103 can include the following steps.

[0094] S1031: The controller obtains a first average contrast, a first average brightness, and a first average color temperature corresponding to the entire frame of the nth frame of detection image.

[0095] S1032: The controller obtains a second average contrast, a second average brightness, and a second average color temperature corresponding to the entire frame of the nth frame of target image.

[0096] It can be understood that, in the embodiment, when the controller compares the nth frame of target image with the corresponding detection image, the average contrast, the average brightness, and the average color temperature can be compared at the same time. For the processor or the controller, it will not consume a large amount of computing power to directly obtain these parameters from the image. For example, when obtaining the average contrast, it can be obtained by calculating the difference between each pixel and its neighborhood pixels in the image. Common local contrast calculation methods include Laplace operator, Sobe l operator, etc. These methods extract edge information in the image by convolution operation on the image. Of course, the global contrast of the sub-image can be obtained by calculating the difference between the maximum and minimum pixel values of the image. The local average contrast and the global contrast can be combined to more comprehensively describe the contrast characteristics of the image. The average brightness and the average color temperature can also be obtained by using the corresponding method, which is not described in detail in the embodiment.

[0097] S1033: The controller determines the contrast parameter based on the difference between the first average contrast and the second average contrast, the difference between the first average brightness and the second average brightness, and the difference between the first average color temperature and the second average color temperature.

[0098] It can be understood that, in order to use a number to represent the entire picture, that is, the contrast parameter, in the embodiment, the following method can be used. For example, in the embodiment, certain weights can be assigned to the three different indices of contrast, color temperature, and brightness, and then the actual data and the product of the corresponding weights are integrated to realize it.

[0099] Specifically, in the embodiment, the controller acquires a contrast parameter, and the contrast parameter satisfies:

[0100] D = a |x1-x2| + b |y1-y2| + c |z1-z2|;

[0101] wherein x1 is the first average contrast, x2 is the second average contrast, y1 is the first average brightness, y2 is the second average brightness, z1 is the first average color temperature, z2 is the second average color temperature, a, b, c are different weight parameters respectively, and D is the contrast parameter. It can be understood that the contrast parameter acquired in the above manner can be used to measure the degree of change of the whole image. When the whole image changes greatly, for example, an obstacle appears, the corresponding contrast parameter is also greater.

[0102] S104: If the contrast parameter satisfies the preset condition, the controller performs image recognition on the nth target image, and confirms whether there is an obstacle on the conveying path according to the result of the image recognition.

[0103] It can be understood that in the above step, after the contrast parameter is acquired, the greater the contrast parameter is, the greater the image change is. Therefore, a preset threshold can be set in advance, and then the preset threshold is compared with the contrast parameter. If the contrast parameter is greater than or equal to the preset threshold, the controller performs image recognition on the nth target image, that is, it is proved that the image change is great to a certain extent, and there may be an obstacle. At this time, the image recognition technology can be enabled to identify the obstacle.

[0104] It can be understood that in the embodiment, after it is determined that the nth image has a great change compared with the previous period, the neural network-based image recognition can be performed on the image at this time, so that the result is more accurate, and whether the image has an obstacle image is further confirmed. Illustratively, in the embodiment, step S105 can include the following steps:

[0105] S1051: The controller determines a target feature and performs feature extraction on the nth target image.

[0106] S1052: The controller inputs the target feature extracted from the nth target image into a classifier, and acquires an output result of the classifier.

[0107] S1053: The controller determines the category and position information of the obstacle in the nth target image based on the output result.

[0108] It can be understood that the controller first identifies and determines the target feature. Then, it performs detailed feature extraction on the target image in the nth frame to better understand and analyze the image content. After completing the feature extraction, the controller inputs the target features extracted from the nth frame target image into the classifier. The classifier then analyzes and processes these features, and finally outputs a result that the controller will obtain and use for the next step of processing. Based on the output result provided by the classifier, the controller further determines the specific category and location information of the obstacle in the nth frame target image. This process involves detailed analysis and recognition of the image, ensuring accurate identification of the type and specific location of the obstacle in the image, thereby providing reliable data support for subsequent processing and decision-making.

[0109] In some other embodiments, other image recognition methods can also be used, which are not limited in this embodiment.

[0110] S105: If the controller determines that there is an obstacle, the intelligent carrying robot is controlled to execute an obstacle avoidance program.

[0111] It can be understood that after determining that there is an obstacle in the image, the controller can automatically plan a route according to the shape, size, etc. of the obstacle, and also confirm a candidate path from multiple candidate paths as a new carrying path. How to plan a route or select a route is disclosed in the related art, which is not limited in this embodiment.

[0112] The embodiment of the application proposes an intelligent carrying robot obstacle avoidance method based on a visual neural network. First, the controller obtains m frames of detection images in the first carrying period T1 according to the acquisition time based on the image acquisition component of the intelligent carrying robot. Then, the controller continuously acquires multiple frames of target images with sequence based on the image acquisition component in the second carrying period T2. Then, the corresponding relationship between each frame of target image and detection image is determined, and in the second period, each time a frame of target image is acquired, it is compared with the corresponding detection image. If the comparison result shows that the image has changed greatly, it proves that there may be an obstacle in the image at this time. At this time, the controller can perform image recognition on the image and specifically detect the obstacle. The intelligent carrying robot obstacle avoidance method based on the visual neural network proposed in the application judges whether there is an obstacle according to the image of different periods during the periodic carrying of the intelligent carrying robot, without using real-time image recognition technology to recognize the obstacles on the path. The image is only recognized when needed, which has a smaller demand for computing power and saves production and use costs.

[0113] Based on the same inventive concept, the embodiment of the present application also proposes an intelligent carrying robot obstacle avoidance system based on a visual neural network. The intelligent carrying robot obstacle avoidance system based on the visual neural network comprises a controller and an intelligent carrying robot. The intelligent carrying robot comprises an image acquisition component. The system is configured to:

[0114] The controller acquires m frames of detection images in the first carrying period T1 in the order of acquisition time based on the image acquisition component, which are the 1st frame, the 2nd frame,..., and the mth frame. The detection images are images of the intelligent carrying robot moving on the carrying path.

[0115] The controller continuously acquires multiple frames of target images with sequence in the second carrying period T2 based on the image acquisition component, and determines the corresponding relationship between each frame of target images and the detection images.

[0116] When the controller acquires the nth frame of target images, the controller compares the nth frame of detection images with the nth frame of target images, and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n.

[0117] If the comparison parameter meets a preset condition, the controller performs image recognition on the nth frame of target images, and confirms whether there is an obstacle on the carrying path according to the result of image recognition.

[0118] If the controller determines that there is an obstacle, the controller controls the intelligent carrying robot to perform an obstacle avoidance program.

[0119] In some embodiments, the system is configured to:

[0120] The controller continuously acquires multiple frames of target images with sequence in the second carrying period T2 based on the image acquisition component, and determines the corresponding relationship between each frame of target images and the detection images, comprising:

[0121] The controller acquires m frames of target images in the second carrying period T2. Each frame of detection images corresponds to each frame of target images in the order of acquisition time.

[0122] In some embodiments, the system is configured to:

[0123] The intelligent carrying robot obstacle avoidance system based on the visual neural network further comprises a positioning component. The controller continuously acquires multiple frames of target images with sequence in the second carrying period based on the image acquisition component, and determines the corresponding relationship between each frame of target images and the detection images, comprising:

[0124] The controller acquires m frames of detection images in the first carrying period T1, and acquires corresponding position information of each frame of detection images based on the positioning component.

[0125] The controller determines the acquisition position of each target image based on corresponding position information of each detection image.

[0126] In some embodiments, the system is configured to:

[0127] When the controller acquires the nth target image, the controller compares the nth detection image with the nth target image and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n, including:

[0128] The controller acquires a first average contrast, a first average brightness, and a first average color temperature corresponding to the entire frame of the nth detection image.

[0129] The controller acquires a second average contrast, a second average brightness, and a second average color temperature corresponding to the entire frame of the nth target image.

[0130] The controller determines the comparison parameter based on the difference between the first average contrast and the second average contrast, the difference between the first average brightness and the second average brightness, and the difference between the first average color temperature and the second average color temperature.

[0131] In some embodiments, the system is configured to:

[0132] The controller determines the comparison parameter based on the difference between the first average contrast and the second average contrast, the difference between the first average brightness and the second average brightness, and the difference between the first average color temperature and the second average color temperature, including:

[0133] The controller acquires the comparison parameter, which satisfies:

[0134] D = a |x1-x2| + b |y1-y2| + c |z1-z2|;

[0135] wherein x1 is the first average contrast, x2 is the second average contrast, y1 is the first average brightness, y2 is the second average brightness, z1 is the first average color temperature, z2 is the second average color temperature, a, b, and c are different weight parameters, and D is the comparison parameter.

[0136] In some embodiments, the system is configured to:

[0137] If the comparison parameter satisfies a preset condition, the controller performs image recognition on the nth target image, and determines whether there is an obstacle on the conveying path according to the result of the image recognition, including:

[0138] The controller acquires a preset threshold and compares the preset threshold with the comparison parameter.

[0139] If the comparison parameter is greater than or equal to the preset threshold, the controller performs image recognition on the nth target image.

[0140] In some embodiments, the system is configured to:

[0141] If the comparison parameter is greater than or equal to a preset threshold, the controller performs image recognition on the nth target image, including:

[0142] The controller determines the target feature and performs feature extraction on the nth target image;

[0143] The controller inputs the target feature extracted from the nth target image into the classifier and obtains the output result of the classifier;

[0144] The controller determines the category and location information of the obstacle in the nth target image based on the output result.

[0145] In some embodiments, the system is configured to:

[0146] If the controller determines that there is an obstacle, the controller controls the intelligent carrying robot to perform an obstacle avoidance program, including:

[0147] If the controller determines that there is an obstacle, the controller confirms an alternative path from a plurality of alternative paths as a new carrying path.

[0148] The embodiment of the present application proposes an intelligent carrying robot obstacle avoidance system based on a visual neural network. First, the controller obtains m frames of detection images in the first carrying period T1 according to the acquisition time based on the image acquisition component of the intelligent carrying robot. Then, the controller continuously acquires a plurality of target images with sequence based on the image acquisition component in the second carrying period T2. Then, the corresponding relationship between each target image and the detection image is determined, and in the second period, each time a target image is acquired, it is compared with the corresponding detection image. If the comparison result shows that the image changes greatly, it proves that there may be an obstacle in the image at this time. At this time, the controller can perform image recognition on the image and specifically detect the obstacle. The intelligent carrying robot obstacle avoidance system based on the visual neural network proposed in the present application can determine whether there is an obstacle according to the image of different periods in the process of periodic carrying of the intelligent carrying robot, without using real-time image recognition technology to recognize the obstacle on the path. Only when necessary, the image is recognized, the demand for computing power is small, and the production and use cost is saved.

[0149] Based on the same inventive concept, the embodiments of the present application also propose an electronic device, please refer to Figure 3 The electronic device includes:

[0150] At least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for obstacle avoidance of the intelligent carrying robot based on the visual neural network.

[0151] In addition, to achieve the above object, the embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method for obstacle avoidance of the intelligent carrying robot based on the visual neural network.

[0152] The various components of the electronic device will be specifically introduced as follows:

[0153] The processor is the control center of the electronic device, and can be one processor or a plurality of processing elements. For example, the processor is one or more central processing units (CPU), application specific integrated circuits (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, such as one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA).

[0154] Optionally, the processor can execute various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0155] The memory is used to store software programs for implementing the embodiments of the present application, and is controlled by the processor to execute. The specific implementation manner can refer to the above method embodiments, and will not be described here.

[0156] Optionally, the memory can be read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage, optical disk storage (including compact disks and digital versatile disks), magnetic disk storage or other magnetic storage devices, or any other media that can be used to carry or store desired program codes in the form of instructions or data structures and that can be accessed by the computer, but not limited to. The memory can be integrated with the processor or exist independently and be coupled to the processor through the interface circuit of the electronic device, and the embodiments of the present application do not make specific limitations thereto.

[0157] The transceiver is configured to communicate with the network device or the terminal device.

[0158] Optionally, the transceiver can include a receiver and a transmitter. The receiver is configured to implement the receiving function, and the transmitter is configured to implement the transmitting function.

[0159] Optionally, the transceiver can be integrated with the processor or exist independently and be coupled to the processor through the interface circuit of the router, and the embodiments of the present application do not make specific limitations thereto.

[0160] In addition, the technical effects of the electronic device can refer to the technical effects of the data transmission method of the above-mentioned method embodiments, which will not be repeated here.

[0161] It should be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0162] It should also be understood that the memory in the embodiments of the present application can be volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. Where nonvolatile memory is utilized, it can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), used as external cache memory. By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0163] The above-described embodiments can be implemented in part or in whole through software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When loaded and executed by a computer, the computer instructions or computer programs can generate the flow or function according to the embodiments of the present application in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, such as from one website site, computer, server, or data center to another website site, computer, server, or data center through a wired (e.g., infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing a set of one or more available media. The available media can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0164] It should be understood that the term "and / or" used herein is merely an association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after it, but it can also represent an "and / or" relationship, which can be understood in the context before and after it.

[0165] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0166] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0167] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on specific applications and design constraints. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

Claims

1. A visual neural network-based intelligent carrying robot obstacle avoidance method, characterized in that, The application is suitable for a visual neural network-based intelligent carrying robot obstacle avoidance system, which comprises a controller and an intelligent carrying robot, the intelligent carrying robot comprises an image acquisition component, the method is suitable for the controller, and comprises the following steps: The controller obtains m detection images of the intelligent carrying robot in a first carrying cycle The detection images are images of the intelligent carrying robot moving on a carrying path. The controller obtains a second carrying cycle based on the image acquisition component A plurality of frames of target images with sequence are continuously acquired, and a corresponding relationship between each frame of the target images and the detection image is determined. When the controller acquires the nth frame of the target image, the controller compares the nth frame of the detection image with the nth frame of the target image and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n; If the comparison parameter meets a preset condition, the controller performs image recognition on the nth frame of the target image, and determines whether there is an obstacle on the carrying path according to the result of image recognition; If the controller determines that there is an obstacle, the controller controls the intelligent carrying robot to execute an obstacle avoidance program.

2. The visual neural network-based intelligent carrying robot obstacle avoidance method according to claim 1, characterized in that, The controller obtains a second carrying cycle based on the image acquisition component Continuously acquiring a plurality of target images with sequence, and determining the corresponding relationship between each target image and the detection image, comprising: The controller acquires m frames of the target image in the second carrying cycle Each frame of the detection image corresponds to each frame of the target image in the time order of acquisition. 3.The visual neural network-based intelligent carrying robot obstacle avoidance method of claim 1, wherein, The vision neural network-based intelligent carrying robot obstacle avoidance system further comprises a positioning component, and the controller determines the corresponding relationship between each frame of the target image and the detection image based on the image acquisition component in the second carrying cycle A plurality of frames of target images with sequences are continuously acquired, and a corresponding relationship between each frame of the target image and the detection image is determined, including: The controller acquires m frames of the detection image within the first carrying cycle and obtains corresponding position information of each frame of the detection image based on the positioning component; The controller determines the acquisition position of each frame of the target image based on the corresponding position information of each frame of the detection image.

4. The visual neural network-based intelligent carrying robot obstacle avoidance method according to claim 3, characterized in that, When the controller acquires the nth frame of the target image, the controller compares the nth frame of the detection image with the nth frame of the target image and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n, comprising the following steps: The controller acquires a first average contrast, a first average brightness and a first average color temperature corresponding to the entire frame of the nth frame of the detection image; The controller acquires a second average contrast, a second average brightness and a second average color temperature corresponding to the entire frame of the nth frame of the target image; The controller determines the comparison parameter based on the difference between the first average contrast and the second average contrast, the difference between the first average brightness and the second average brightness, and the difference between the first average color temperature and the second average color temperature.

5. The visual neural network-based intelligent carrying robot obstacle avoidance method according to claim 4, characterized in that, The controller determines the comparison parameter based on the difference between the first average contrast and the second average contrast, the difference between the first average brightness and the second average brightness, and the difference between the first average color temperature and the second average color temperature, comprising the following steps: The controller acquires the comparison parameter, and the comparison parameter meets the following conditions: D= ; wherein, is the first average contrast, is the second average contrast, is the first average brightness, is the second average brightness, is the first average color temperature, is the second average color temperature, , , are different weight parameters, respectively, and D is the contrast parameter.

6. The visual neural network-based intelligent carrying robot obstacle avoidance method according to claim 5, characterized in that, If the comparison parameter meets a preset condition, the controller performs image recognition on the nth frame of the target image, and determines whether there is an obstacle on the carrying path according to the result of image recognition, comprising the following steps: The controller acquires a preset threshold and compares the preset threshold with the comparison parameter; If the comparison parameter is greater than or equal to the preset threshold, the controller performs image recognition on the nth frame of the target image.

7. The visual neural network-based intelligent carrying robot obstacle avoidance method according to claim 6, characterized in that, If the comparison parameter is greater than or equal to the preset threshold, the controller performs image recognition on the nth frame of the target image, comprising the following steps: The controller determines a target feature and performs feature extraction on the nth frame of the target image; The controller inputs the target feature extracted from the nth frame of the target image into a classifier and acquires an output result of the classifier; The controller determines the category and position information of the obstacle in the nth frame of the target image based on the output result. 8.The method of claim 7, wherein if the controller determines that there is an obstacle, the controller controls the intelligent carrying robot to execute an obstacle avoidance procedure, including: determining whether the carrying path is blocked by the obstacle; if the carrying path is blocked by the obstacle, the controller controls the intelligent carrying robot to execute a path switching procedure. If the controller determines that there is an obstacle, the controller confirms one of the alternative paths as the new carrying path. The intelligent carrying robot obstacle avoidance system based on a visual neural network includes a controller and an intelligent carrying robot, the intelligent carrying robot includes an image acquisition component, and the system is configured to: acquire a target image of the carrying path by the image acquisition component; 9. A visual neural network-based intelligent carrying robot obstacle avoidance system, characterized in that, When the controller acquires the nth frame of the target image, the controller compares the mth frame of the detection image with the nth frame of the target image and acquires a comparison parameter, wherein m and n are both natural numbers greater than zero and m is greater than n; The controller obtains detection images of m frames in which the intelligent carrying robot is in the first carrying cycle The detection images are images of the intelligent carrying robot moving on the carrying path. The controller obtains a second carrying cycle based on the image acquisition component A plurality of frames of target images with sequence are continuously acquired, and a corresponding relationship between each frame of the target images and the detection image is determined. If the comparison parameter meets a preset condition, the controller performs image recognition on the nth frame of the target image, and according to the result of the image recognition, confirms whether there is an obstacle on the carrying path; If the controller determines that there is an obstacle, the controller controls the intelligent carrying robot to execute an obstacle avoidance procedure. 10.An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8. ​

Images