Obstacle avoidance method and device based on target detection
By acquiring color and depth images, and using the YOLO model trained with data augmentation for object recognition and point cloud fusion, the problem of determining obstacle categories and spatial locations in existing technologies is solved, enabling precise obstacle avoidance for intelligent forklifts.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 海南 ROBOT (ZHEJIANG) CO LTD
- Filing Date
- 2022-09-22
- Publication Date
- 2026-06-26
AI Technical Summary
Existing technologies cannot determine the type and spatial location of obstacles, nor can they obtain the category attributes of obstacles, resulting in insufficient accuracy in obstacle avoidance by intelligent forklifts.
An object detection-based approach is adopted. By acquiring color and depth images of a preset region, a YOLO model trained with data augmentation is used for object recognition. The object is then fused with point cloud information to obtain the category and spatial location information of obstacles for obstacle avoidance.
It enables accurate obstacle identification and avoidance, improving the obstacle avoidance accuracy and safety of intelligent forklifts.
Smart Images
Figure CN115601727B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection technology, and in particular to an obstacle avoidance method and apparatus based on target detection. Background Technology
[0002] With the advent of Industry 4.0 and intelligent manufacturing, the industrial sector is continuously developing from traditional manufacturing towards digitalization, intelligence, and unmanned operation. As customer needs become more personalized and customized, and with the increase in the smallest inventory unit and the continuous changes in logistics operation scenarios, intelligent driving is playing an increasingly important role in the forklift field. Obstacle avoidance is a key aspect of achieving intelligent driving forklifts.
[0003] Currently, existing intelligent obstacle avoidance systems typically use laser calibration to determine the distance between obstacles and intelligent forklifts. However, existing technologies can only determine the straight-line distance between obstacles and intelligent forklifts, and cannot determine the type and spatial location of obstacles, nor can they obtain the category attributes of obstacles.
[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention
[0005] The main objective of this invention is to provide an obstacle avoidance method and apparatus based on target detection, which aims to solve the technical problems that the prior art cannot determine the category and spatial location of obstacles, and cannot obtain the category attributes of obstacles.
[0006] To achieve the above objectives, the present invention provides an obstacle avoidance method based on target detection, the method comprising the following steps:
[0007] Acquire the color and depth images of the preset region;
[0008] Based on the color image, object recognition is performed using a preset recognition model to obtain category information. The preset recognition model is a YOLO model trained with data augmentation.
[0009] The depth image is preprocessed to obtain the target bounding box point cloud information;
[0010] The category information is fused with the target bounding box point cloud information to obtain the vehicle body position information of the target bounding box;
[0011] Obstacle detection is performed based on the vehicle position information of the target box to obtain obstacle information;
[0012] The corresponding obstacle avoidance process is performed based on the obstacle information.
[0013] Optionally, before the step of obtaining category information by performing object recognition based on the color image using a preset recognition model, the following steps are included:
[0014] Construct the initial YOLO model;
[0015] The initial YOLO model is trained using the Bag-of-Freebies object detection training method and sample images to obtain the data-augmented YOLO model.
[0016] Optionally, training the initial YOLO model using the Bag-of-Freebies object detection training method and sample images to obtain the data-augmented YOLO model includes:
[0017] The sample images are linearly mixed using Mixup according to a preset ratio determined by the beta distribution to obtain the mixed sample images;
[0018] The one-hot labels corresponding to the sample images are linearly mixed using Mixup according to a preset ratio determined by the beta distribution to obtain the mixed one-hot labels;
[0019] The initial YOLO model is trained based on the mixed sample images and the mixed one-hot labels to obtain the data-augmented YOLO model.
[0020] Optionally, the Bag-of-Freebies object detection training method and sample images are used to train the initial YOLO model to obtain the data-augmented YOLO model, which further includes:
[0021] Label smoothing is performed on the one-hot labels corresponding to the sample images;
[0022] The initial YOLO model is trained based on the one-hot labels corresponding to the processed sample images to obtain the data-augmented YOLO model.
[0023] Optionally, the Bag-of-Freebies object detection training method and sample images are used to train the initial YOLO model to obtain the data-augmented YOLO model, which further includes:
[0024] The sample images are subjected to random transformation processing;
[0025] The initial YOLO model is trained using the processed sample images to obtain the data-augmented YOLO model.
[0026] Optionally, the Bag-of-Freebies object detection training method and sample images are used to train the initial YOLO model to obtain the data-augmented YOLO model, which further includes:
[0027] The learning rate of the initial YOLO model is adjusted according to cosine learning decay, so that the initial YOLO model is trained on the sample images according to the adjusted learning rate, thereby obtaining the data-augmented YOLO model.
[0028] Optionally, the Bag-of-Freebies object detection training method and sample images are used to train the initial YOLO model to obtain the data-augmented YOLO model, which further includes:
[0029] The step size corresponding to the preset YOLO model is determined according to the type of the preset YOLO model;
[0030] In batch processing, the width and height of the sample images are set to any integer multiple of the step size corresponding to the preset YOLO model;
[0031] The initial YOLO model is trained based on the sample image with determined width and height to obtain the data-augmented YOLO model.
[0032] Optionally, the object detection-based obstacle avoidance method is applied to an object detection-based obstacle avoidance device, which includes a monocular camera and a binocular depth camera; acquiring the color image and depth image of the preset area includes:
[0033] The monocular camera acquires a color image of the preset area, and the binocular depth camera acquires a depth image of the preset area.
[0034] Optionally, before the steps of acquiring a color image of a preset area using the monocular camera and acquiring a depth image of the preset area using the binocular depth camera, the following steps are included:
[0035] Color images and depth images are acquired using the monocular camera and the binocular depth camera, respectively.
[0036] The color image and the depth image are matched based on the checkerboard matching algorithm;
[0037] The monocular camera and the binocular depth camera are calibrated based on the matching results.
[0038] In addition, to achieve the above objectives, the present invention also proposes an obstacle avoidance device based on target detection, the device comprising: an image acquisition module, an image recognition module, a point cloud information acquisition module, a fusion module, an obstacle avoidance module based on target detection, and an obstacle avoidance module;
[0039] The image acquisition module is used to acquire the color image and depth image of a preset area;
[0040] The image recognition module is used to perform object recognition based on the color image using a preset recognition model to obtain category information. The preset recognition model is a YOLO model trained with data augmentation.
[0041] The point cloud information acquisition module is used to preprocess the depth image to obtain point cloud information;
[0042] The fusion module is used to fuse the category information with the target box point cloud information to obtain the vehicle body position information of the target box;
[0043] The obstacle avoidance module based on target detection is used to detect obstacles based on the vehicle body position information of the target box and obtain obstacle information;
[0044] The obstacle avoidance module is used to perform corresponding obstacle avoidance processing based on the obstacle information.
[0045] This invention primarily relates to the field of target detection technology, specifically a target detection-based obstacle avoidance method and device. The method includes: acquiring a color image and a depth image of a preset region; performing object recognition based on the color image using a preset recognition model (a YOLO model trained with data augmentation) to obtain category information; preprocessing the depth image to obtain point cloud information; fusing the category information with the point cloud information to obtain fused information; and performing obstacle avoidance based on the fused information to obtain obstacle information. This invention acquires a color image and a depth image of a preset region, inputs the color image into a data augmentation-trained YOLO model to obtain category information, preprocesses the depth image to obtain point cloud information, and determines obstacles based on the fused information obtained from the fusion of category information and point cloud information, thereby obtaining the category and spatial location of obstacles in the preset region. Attached Figure Description
[0046] Figure 1 This is a schematic diagram of the structure of an obstacle avoidance device based on target detection in the hardware operating environment involved in the embodiments of the present invention;
[0047] Figure 2 This is a flowchart illustrating the first embodiment of the obstacle avoidance method based on target detection of the present invention;
[0048] Figure 3 This is a schematic diagram of the training step size adjustment in the first embodiment of the obstacle avoidance method based on target detection of the present invention;
[0049] Figure 4 This is a schematic flowchart of the third embodiment of the obstacle avoidance method based on target detection of the present invention;
[0050] Figure 5 This is a structural block diagram of the first embodiment of the obstacle avoidance device based on target detection of the present invention.
[0051] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0052] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.
[0053] Reference Figure 1 , Figure 1 This is a schematic diagram of the obstacle avoidance device structure based on target detection in the hardware operating environment involved in the embodiments of the present invention.
[0054] like Figure 1 As shown, the obstacle avoidance device based on target detection may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005 (GPU or NPU). The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.
[0055] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the obstacle avoidance device based on target detection, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0056] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and an obstacle avoidance program based on target detection.
[0057] exist Figure 1 In the obstacle avoidance device based on target detection shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the obstacle avoidance device based on target detection of the present invention can be set in the obstacle avoidance device based on target detection. The obstacle avoidance device based on target detection calls the obstacle avoidance program based on target detection stored in the memory 1005 through the processor 1001 and executes the obstacle avoidance method based on target detection provided in the embodiment of the present invention.
[0058] This invention provides an obstacle avoidance method based on target detection, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the obstacle avoidance method based on target detection of the present invention.
[0059] In this embodiment, the obstacle avoidance method based on target detection includes the following steps:
[0060] Step S10: Obtain the color image and depth image of the preset area;
[0061] It should be understood that the executing entity of this embodiment can be an electronic device with data processing, network communication, and program execution functions, such as a mobile phone, tablet computer, personal computer, etc., or other electronic devices that can achieve the same or similar functions, such as the above-mentioned obstacle avoidance device based on target detection. This embodiment does not limit this. The following uses the above-mentioned obstacle avoidance device based on target detection as an example to specifically describe this embodiment and the following embodiments.
[0062] It should be understood that the aforementioned preset area is the target area for obstacle avoidance based on target detection. In specific implementation, the aforementioned preset area is the target area for the planned travel route. By detecting obstacles within the aforementioned preset area, the aforementioned obstacle avoidance device based on target detection can plan a path according to the obstacle detection results.
[0063] It should be understood that the color image mentioned above is an RGB image; the depth image mentioned above is an image that records the depth information of objects within the preset area, and the depth information includes the distance between the objects within the preset area and the depth image acquisition point mentioned above;
[0064] It should be understood that the above-mentioned obstacle avoidance device based on target detection can be used for the automatic driving of vehicles. This embodiment does not limit this. This embodiment is illustrated using an intelligent forklift as an example.
[0065] Furthermore, the object detection-based obstacle avoidance method is applied to an object detection-based obstacle avoidance device, which includes a monocular camera and a binocular depth camera; step S10: acquiring the color image and depth image of a preset area includes:
[0066] Step S11: Obtain a color image of the preset area using the monocular camera, and obtain a depth image of the preset area using the binocular depth camera.
[0067] Step S20: Based on the color image, perform object recognition using a preset recognition model to obtain category information. The preset recognition model is a YOLO model trained with data augmentation.
[0068] It should be understood that the preset recognition model is a target detection model. The preset recognition model can perform target recognition on the preset area based on the input color image to obtain the category information of each target in the preset area. Specifically, the preset recognition model can be a YOLO model. In this invention, the YOLOv4-tiny model is used as an example for explanation.
[0069] It should be noted that the category information includes the shape category of the target in the color image, etc., and the host computer can determine the category of each target in the above-mentioned preset area based on the category information;
[0070] It should be understood that the YOLO model trained with the aforementioned data augmentation is the model obtained by training the aforementioned YOLOv4-tiny using bag-of-freebies.
[0071] Step S30: Preprocess the depth image to obtain point cloud information;
[0072] It should be understood that the preprocessing includes downsampling and filtering. Through downsampling and filtering, each point in the acquired depth image is processed to obtain the point cloud information. The point cloud information includes: a set of point data of each target surface in the preset area. The set of point data includes the three-dimensional information of each point. The three-dimensional information can be used to identify the target distance detection position.
[0073] Step S40: Fuse the category information with the point cloud information to obtain fused information;
[0074] It should be understood that fusing the point cloud information and category information means matching the shape category of the target obtained through target recognition in the preset area with the point cloud information one by one. The fused information includes the category information and position information of each target in the preset area, thereby determining the shape and position of each target in the preset area.
[0075] Step S50: Perform obstacle avoidance based on target detection based on the fused information to obtain obstacle information;
[0076] It should be understood that the above obstacle information is the information used by the obstacle avoidance device based on target detection to determine whether each target in the preset area is a target that affects the route planning, based on the category information and location information contained in the above fused information.
[0077] It should be understood that the preset area contains several detected targets. The obstacle avoidance device based on target detection judges the several detected targets in the preset area according to the fused information to determine whether the targets have the ability to affect the driving of the intelligent forklift.
[0078] Step S60: Perform corresponding obstacle avoidance processing based on the obstacle information.
[0079] It should be understood that the above obstacle avoidance processing includes adjusting the obstacle avoidance strategy of the intelligent forklift, such as adjusting the deceleration distance and stopping distance.
[0080] It should be understood that there are various situations regarding the obstacles detected, including whether the obstacle is completely detected and whether the obstacle can be ignored. The obstacle avoidance handling of the intelligent forklift also includes multiple handling methods.
[0081] In practical implementation, when planning the optimal travel route based on the fusion information of obstacles in the aforementioned preset area, firstly, during actual operation, when performing obstacle avoidance based on target detection, since the category and location information of targets in the preset area may not be fully collected in one go, each target in the aforementioned preset area may not appear completely in the aforementioned preset area due to occlusion or framing selection. At this time, it is necessary to mark the targets connected to the boundary of the aforementioned preset area, and to perform corresponding image expansion acquisition according to the boundary type between the target and the aforementioned preset area to obtain the complete color image and depth image of the target. Specifically, if a target is partially outside the aforementioned preset area, then the edge of the preset area connected to the target is selected, and the image is acquired from the outside of that edge until the entire target is collected.
[0082] Secondly, it is necessary to determine whether an obstacle can be ignored based on the obstacle category information. The obstacle avoidance device based on target detection marks the obstacle, that is, it determines whether the obstacle can directly affect the movement. When the obstacle avoidance device based on target detection identifies a certain immovable and indestructible target in the preset area, it treats the target as an obstacle and obtains the obstacle information. For example, if the target is a few steel bones or a solid iron ball, which cannot be ignored and is difficult to move by its own strength, the target is marked as an obstacle, and the path planning avoids such an obstacle that cannot be ignored. When it is determined that an obstacle in the preset area does not affect the movement of the intelligent forklift, it is marked as a non-obstacle or a low-risk obstacle. That is, the fusion information of the detected targets in the preset area shows that the target can be moved or changed. For example, when the target is detected as a piece of foam or an inflated empty garbage bag, the obstacle avoidance device based on target detection judges it as a non-obstacle, and ignores the target or reduces the weight of the target in the path planning.
[0083] Finally, during path planning, if obstacles within the aforementioned preset area cannot be avoided, that is, if a reasonable path cannot be planned to avoid the obstacles based on the location information of the obstacles, then the preset area needs to be selected again. For example, with the current color image and depth image acquisition points as the center, the acquisition device is rotated 30 degrees clockwise to acquire the preset area again, and the obstacle avoidance based on target detection is performed based on the newly acquired preset area to obtain a reasonable path plan.
[0084] This embodiment mainly relates to the field of target detection technology, specifically an obstacle avoidance method based on target detection. The method includes: Step S10: acquiring a color image and a depth image of a preset region; Step S20: performing object recognition based on the color image using a preset recognition model to obtain category information, wherein the preset recognition model is a YOLO model trained with data augmentation; Step S30: preprocessing the depth image to obtain point cloud information; Step S40: fusing the category information with the point cloud information to obtain fused information; Step S50: performing obstacle avoidance based on target detection based on the fused information to obtain obstacle information. This invention acquires a color image and a depth image of a preset region, inputs the color image into a YOLO model trained with data augmentation to obtain category information, preprocesses the depth image to obtain point cloud information, and determines obstacles based on the fused information obtained from the fusion of category information and point cloud information, thereby obtaining the category and spatial location of obstacles in the preset region.
[0085] Based on the first embodiment described above, in this embodiment, before step S20, the method further includes:
[0086] Step S21: Construct the initial YOLO model;
[0087] It should be understood that the initial YOLO model mentioned above can be the YOLOv4-tiny model.
[0088] Step S22: Train the initial YOLO model using the Bag-of-Freebies object detection training method and sample images to obtain the data-augmented YOLO model.
[0089] Furthermore, step S22 may specifically include the following steps:
[0090] Step S2211: The sample image is linearly mixed using Mixup according to a preset ratio determined by the beta distribution to obtain the mixed sample image;
[0091] It is important to understand that the Mixup refers to linearly blending one image with another at a certain ratio, while simultaneously blending the one-hot labels of the two images at the same ratio.
[0092] It is important to understand that the beta distribution contains two parameters, α and β. The values of these two parameters directly affect the effect of the Mixup on the training of the YOLO model. For example, when the α value of the beta distribution is between 0.2 and 2, the impact on Mixup is relatively small. However, if the α value is too small, the impact of the Mixup processing on the training of the YOLO model will be insufficient. If the α value is too large, it will fail to reflect the diversity of the input image samples, thus causing the model's learning progress to be slow. Therefore, in practice, the value of α is set to 1.5, and the corresponding β value is also set to 1.5.
[0093] Step S2212: The one-hot labels corresponding to the sample images are linearly mixed using Mixup according to the preset ratio determined by the beta distribution to obtain the mixed one-hot labels;
[0094] It should be understood that the above one-hot labels are linearly mixed according to the preset ratio determined by the beta distribution, and the values of α and β of the beta distribution are the same as the values of the beta distribution in step S2211 above. In this invention, the values of α and β are both 1.5.
[0095] Step S2213: Train the initial YOLO model based on the mixed sample image and the mixed one-hot label to obtain the data-augmented YOLO model.
[0096] Step S2221: Perform label smoothing processing on the one-hot labels corresponding to the sample images; it should be understood that the above label smoothing processing formula is:
[0097]
[0098] Where, q i Let K be the one-hot label of the sample, K be the number of classes, and ε be the smoothing factor. Taking ε as 0.09 as an example, after label smoothing, the one-hot label of "7" becomes:
[0099] [0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.91 0.01 0.01], it can be seen that by using label smoothing, the trained model can be prevented from being overconfident during training, thereby improving the model's generalization ability and reducing the risk of overfitting.
[0100] Step S2222: Train the initial YOLO model based on the one-hot labels corresponding to the processed sample images to obtain the data-augmented YOLO model.
[0101] Step S2231: Randomly transform the sample image;
[0102] It should be understood that the above-mentioned random transformation processing includes random transformation of the shape of the acquired image, such as random flipping, random rotation and random cropping of the sample image. The above-mentioned random transformation processing also includes random adjustment of the brightness, color, contrast and saturation of the sample image, and random addition of noise to the sample image.
[0103] Step S2232: Train the initial YOLO model using the processed sample images to obtain the data-augmented YOLO model.
[0104] Step S2241: Adjust the learning rate of the initial YOLO model according to the cosine learning decay, so that the initial YOLO model is trained on the sample images according to the adjusted learning rate, and obtain the data-augmented YOLO model.
[0105] refer to Figure 3 , Figure 3 This is a schematic diagram of training step size adjustment in an embodiment of the present invention. It should be understood that the above-mentioned adjustment of the learning rate of the initial YOLO model according to cosine learning decay can specifically be to slowly reduce the large learning rate in the early stage of model training, then rapidly reduce the learning rate in the middle, and finally slowly reduce the small learning rate until the learning rate is equal to 0.
[0106] Step S2251: Determine the step size corresponding to the preset YOLO model according to the type of the preset YOLO model;
[0107] It should be understood that, in this invention, the YOLOv4-tiny model is used as an example, and the step size corresponding to the above YOLOv4-tiny model is 32.
[0108] Step S2252: Set the width and height of the sample images in the batch processing to any integer multiple of the step size corresponding to the preset YOLO model;
[0109] It should be understood that the stride size corresponding to the YOLOv4-tiny model of this invention is 32, so the height and width of the sample images in the batch processing can be set to the same value in the following array:
[0110] {320, 352, 384, 416, 448, 480, 512, 544, 576, 608}
[0111] The above arrays are all multiples of 32 corresponding to the step size of the YOLOv4-tiny model of the present invention.
[0112] Step S2253: Train the initial YOLO model based on the sample image after determining the width and height to obtain the data-augmented YOLO model.
[0113] This embodiment trains the initial model using bag-of-freebies to optimize the YOLOv4-tiny model in multiple aspects, further enhancing its accuracy and precision. By using the improved YOLOv4-tiny model to detect color images, the accuracy of the object detection-based obstacle avoidance method is further improved.
[0114] refer to Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the obstacle avoidance method based on target detection of the present invention. Based on the above embodiments, to obtain more stable obstacle avoidance results based on target detection, a third embodiment of the obstacle avoidance method based on target detection of the present invention is proposed. Before step S11, the method further includes:
[0115] Step S01: Acquire color images and depth images using the monocular camera and the binocular depth camera respectively;
[0116] Step S02: Match the color image with the depth image based on the checkerboard matching algorithm;
[0117] It should be understood that the above checkerboard matching algorithm is a method that divides both the color image and the depth image into several image blocks according to a certain ratio, and then matches the image blocks on the segmented color image and the segmented depth image one by one.
[0118] Step S03: Calibrate the monocular camera and the binocular depth camera according to the matching results.
[0119] It should be understood that after matching the color image and the depth image using the checkerboard matching algorithm described above, the monocular camera and the binocular depth camera are calibrated based on the matching results.
[0120] In a specific implementation, the color image and depth image used for calibration of the monocular camera and the binocular depth camera are pre-set in the obstacle avoidance device based on target detection. Before each image acquisition by the monocular camera and the binocular depth camera, the monocular camera and the binocular depth camera need to be calibrated according to the color image and the depth image.
[0121] In this embodiment, before image acquisition by the monocular camera and the binocular depth camera, a checkerboard matching algorithm is used to match the color image and the depth image. Based on the matching results, the monocular camera and the binocular depth camera are calibrated to obtain more stable obstacle avoidance results based on target detection.
[0122] refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of the first embodiment of the obstacle avoidance device based on target detection of the present invention.
[0123] like Figure 5 As shown, the obstacle avoidance device based on target detection proposed in this embodiment of the invention includes:
[0124] Image acquisition module 501, image recognition module 502, point cloud information acquisition module 503, fusion module 504, and obstacle avoidance module 505 based on target detection;
[0125] The image acquisition module 501 is used to acquire the color image and depth image of a preset area;
[0126] The image recognition module 502 is used to perform object recognition based on the color image using a preset recognition model to obtain category information. The preset recognition model is a YOLO model trained with data augmentation.
[0127] The point cloud information acquisition module 503 is used to preprocess the depth image to obtain point cloud information;
[0128] The fusion module 504 is used to fuse the category information with the point cloud information to obtain fused information;
[0129] The obstacle avoidance module 505 based on target detection is used to perform obstacle avoidance based on target detection based on the fused information to obtain obstacle information;
[0130] The obstacle avoidance module 506 is used to perform corresponding obstacle avoidance processing based on the obstacle information.
[0131] Other embodiments or specific implementations of the obstacle avoidance device based on target detection of the present invention can be referred to the above-described method embodiments, and will not be repeated here.
[0132] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0133] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0135] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. An obstacle avoidance method based on target detection, characterized in that, The obstacle avoidance method based on target detection includes: Acquire the color and depth images of the preset region; Construct the initial YOLO model; The initial YOLO model was trained using the Bag-of-Freebies object detection training method and sample images to obtain a data-augmented YOLO model. Based on the color image, object recognition is performed using a preset recognition model to obtain category information. The preset recognition model is a YOLO model trained with data augmentation. The depth image is preprocessed to obtain the target bounding box point cloud information; The category information is fused with the target bounding box point cloud information to obtain the vehicle body position information of the target bounding box; Obstacle detection is performed based on the vehicle position information of the target box to obtain obstacle information; Perform corresponding obstacle avoidance processing based on the obstacle information; The step of training the initial YOLO model using the Bag-of-Freebies object detection training method and sample images to obtain the data-augmented YOLO model includes: The sample images are linearly mixed using Mixup according to a preset ratio determined by the beta distribution to obtain the mixed sample images; The one-hot labels corresponding to the sample images are linearly mixed using Mixup according to a preset ratio determined by the beta distribution to obtain the mixed one-hot labels; The initial YOLO model is trained based on the mixed sample images and the mixed one-hot labels to obtain the data-augmented YOLO model; The Bag-of-Freebies object detection training method and sample images are used to train the initial YOLO model to obtain the data-augmented YOLO model, which further includes: Label smoothing is performed on the one-hot labels corresponding to the sample images; The initial YOLO model is trained based on the one-hot labels corresponding to the processed sample images to obtain the data-augmented YOLO model; The Bag-of-Freebies object detection training method and sample images are used to train the initial YOLO model to obtain the data-augmented YOLO model, which further includes: The learning rate of the initial YOLO model is adjusted according to cosine learning decay, so that the initial YOLO model is trained on the sample images according to the adjusted learning rate, thereby obtaining the data-augmented YOLO model.
2. The obstacle avoidance method based on target detection as described in claim 1, characterized in that, The Bag-of-Freebies object detection training method and sample images are used to train the initial YOLO model to obtain the data-augmented YOLO model, which further includes: The sample images are subjected to random transformation processing; The initial YOLO model is trained using the processed sample images to obtain the data-augmented YOLO model.
3. The obstacle avoidance method based on target detection as described in claim 1, characterized in that, The Bag-of-Freebies object detection training method and sample images are used to train the initial YOLO model to obtain the data-augmented YOLO model, which further includes: The step size corresponding to the preset YOLO model is determined according to the type of the preset YOLO model; In batch processing, the width and height of the sample images are set to any integer multiple of the step size corresponding to the preset YOLO model; The initial YOLO model is trained based on the sample image with determined width and height to obtain the data-augmented YOLO model.
4. The obstacle avoidance method based on target detection as described in claim 1, characterized in that, The object detection-based obstacle avoidance method is applied to an object detection-based obstacle avoidance device, which includes a monocular camera and a binocular depth camera; acquiring the color image and depth image of a preset area includes: The monocular camera acquires a color image of the preset area, and the binocular depth camera acquires a depth image of the preset area.
5. The obstacle avoidance method based on target detection as described in claim 4, characterized in that, Before the steps of acquiring a color image of a preset area using the monocular camera and acquiring a depth image of the preset area using the binocular depth camera, the following steps are included: Color images and depth images are acquired using the monocular camera and the binocular depth camera, respectively. The color image and the depth image are matched based on the checkerboard matching algorithm; The monocular camera and the binocular depth camera are calibrated based on the matching results.
6. An obstacle avoidance device based on target detection, characterized in that, The device includes: an image acquisition module, an image recognition module, a point cloud information acquisition module, a fusion module, an obstacle avoidance module based on target detection, and an obstacle avoidance module; The image acquisition module is used to acquire the color image and depth image of a preset area; The image acquisition module is also used to construct an initial YOLO model; the initial YOLO model is trained using the Bag-of-Freebies object detection training method and sample images to obtain a data-augmented YOLO model; The image recognition module is used to perform object recognition based on the color image using a preset recognition model to obtain category information. The preset recognition model is a YOLO model trained with data augmentation. The point cloud information acquisition module is used to preprocess the depth image to obtain point cloud information; The fusion module is used to fuse the category information with the target box point cloud information to obtain the vehicle body position information of the target box; The obstacle avoidance module based on target detection is used to detect obstacles based on the vehicle body position information of the target box and obtain obstacle information; The obstacle avoidance module is used to perform corresponding obstacle avoidance processing based on the obstacle information; The image acquisition module is further configured to linearly mix the sample image according to a preset ratio determined by the beta distribution using Mixup to obtain a mixed sample image; linearly mix the one-hot labels corresponding to the sample image according to the preset ratio determined by the beta distribution using Mixup to obtain mixed one-hot labels; and train the initial YOLO model based on the mixed sample image and the mixed one-hot labels to obtain the data-augmented YOLO model. The image acquisition module is further configured to perform label smoothing on the one-hot labels corresponding to the sample images; and to train the initial YOLO model based on the one-hot labels corresponding to the processed sample images to obtain the data-augmented YOLO model. The image acquisition module is further configured to adjust the learning rate of the initial YOLO model according to cosine learning decay, so that the initial YOLO model is trained on the sample images according to the adjusted learning rate, thereby obtaining the data-augmented YOLO model.