A forklift blind area visual perception and safety warning method based on edge computing

Abstract

The application discloses a forklift blind area visual perception and safety early warning method based on edge computing and relates to the technical field of industrial vehicle factory internal traffic safety monitoring. In the no-pedestrian target detection state of the forklift, inherent components on the forklift body are used as fixed features to obtain a spatial position mapping relationship between a direct imaging area of the fixed features and a mirror imaging area formed by reflection on the surface of a goods shelf, and a visual echo path mapping model is generated based on the spatial position mapping relationship. In the subsequent forklift operation process, candidate pedestrian targets detected are analyzed, reference pedestrian targets are screened, the visual echo path mapping model is used to predict corresponding mirror prediction areas according to the positions of the reference pedestrian targets, and then mirror pedestrian targets formed by reflection on the goods shelf are recognized and removed, so that the real pedestrian targets and the false targets formed by reflection are effectively distinguished.

Description

Technical Field

[0001] This invention relates to the field of traffic safety monitoring technology within industrial vehicle factory areas, and in particular to a method for visual perception and safety early warning of forklift blind spots based on edge computing. Background Technology

[0002] To improve the operational safety of forklifts in complex environments such as warehousing and logistics, machine vision-based safety monitoring methods are commonly used. This involves installing cameras on the forklift to capture images of the surrounding environment, and then analyzing these images using target detection algorithms to identify pedestrian targets around the forklift. Furthermore, based on the positional relationships and movement status of the detected pedestrian targets, the system determines whether there is a collision risk and outputs safety warning messages when warning conditions are met. This type of method can, to a certain extent, achieve personnel awareness within the forklift's operating area, thereby improving the safety of forklift operation.

[0003] Currently, in actual warehousing environments, forklifts typically operate in areas with reflective properties such as metal shelves and partitions. Images captured by onboard cameras often show mirrored pedestrian targets reflected from the shelf surfaces. Because these mirrored targets have similar appearance features to real pedestrians and their images change accordingly with forklift movement, existing visual perception methods based on target detection often struggle to accurately distinguish between real pedestrian targets and false targets created by shelf reflections, leading to false alarms and reduced accuracy in safety warnings. Existing methods typically focus on improving target detection accuracy or using motion information to filter targets, but lack modeling and analysis of shelf reflection paths. They cannot proactively predict and eliminate potential mirrored areas based on the location of real pedestrian targets, thus failing to effectively solve the mirrored interference problem in forklift visual perception under metal shelf environments. Summary of the Invention

[0004] To address the technical problems existing in the background art, this invention proposes a method for visual perception and safety warning of forklift blind spots based on edge computing.

[0005] This invention proposes a forklift blind spot visual perception and safety warning method based on edge computing, comprising the following steps: S1. During forklift operation, acquire an image sequence corresponding to a collection cycle; the collection cycle includes multiple collection moments, and the image sequence includes multiple images, with each collection moment corresponding to one image. As an explanation: Multiple images are arranged in chronological order according to the time of acquisition to form an image sequence; As an explanation, images can be acquired through the vehicle's onboard camera during forklift operation.

[0006] Pedestrian target detection is performed on each frame of the image sequence. When no pedestrian target is detected in any frame of the image sequence, the shelf mirror reflection feature analysis of the image sequence is triggered to determine whether to generate a visual echo path mapping model. As an explanation, when performing pedestrian target detection on each frame of the image, the image object detection method is used to perform pedestrian target detection on each frame of the image. If a pedestrian target exists, the bounding box corresponding to the pedestrian target is output.

[0007] S2. After generating the visual echo path mapping model in any acquisition cycle, the acquisition cycle following that acquisition cycle is taken as the subsequent acquisition cycle; obtain the image sequence corresponding to the subsequent acquisition cycle, and generate a candidate pedestrian target set based on the image sequence corresponding to the subsequent acquisition cycle; generate a reference pedestrian target set based on the candidate pedestrian target set. S3. Generate a set of mirror prediction regions based on the reference pedestrian target set; S4. Generate a set of real pedestrian targets based on the candidate pedestrian target set and the mirror prediction region set; S5. Obtain the blind spot warning range of the forklift; determine whether to generate a blind spot safety warning signal for real pedestrian targets in the real pedestrian target set based on the blind spot warning range of the forklift.

[0008] Preferably, in S1, the shelf specular reflection feature analysis of the image sequence is triggered to determine whether a visual echo path mapping model should be generated, as follows: The analysis of the mirror reflection characteristics of the shelf includes: A fixed feature of the forklift itself is a component with edges or corners on its body. For illustration, inherent components include: forklift fork arms, overhead guard support struts, steering wheel outer edge, and vehicle warning light outer contour; Obtain the geometric contour of the forklift's own fixed features; In the image sequence, one frame is acquired sequentially according to the order of acquisition time as the current frame image, and the following operations are performed on each current frame image: An image segmentation method is used to segment the target region from the current frame image that matches the geometric contour of the forklift's own fixed features; If the target region is segmented, the target region is taken as the direct imaging region corresponding to the fixed features of the forklift itself, and the subsequent frames in the image sequence are traversed. If the target region is not segmented in any of the images in the image sequence, it is determined that there are no fixed features of the forklift itself in the current image sequence, and the analysis of the mirror reflection features of the shelf is terminated. Once the target region is segmented, multiple images in the image sequence are traversed to obtain all directly imaged regions corresponding to the fixed features of the forklift itself. For any directly imaged region, the image containing that region is selected, and the selected image is horizontally flipped to obtain the flipped image. An image segmentation method is used to segment out the region from the flipped image that matches the geometric contour of the forklift's own fixed features, which is then used as the mirror flipped region. The mirrored area is horizontally flipped to obtain the mirrored imaging area corresponding to the fixed features of the forklift itself. As an explanation, the mirrored imaging area is the mirrored imaging area formed by the reflection of the fixed features of the forklift itself onto the surface of the surrounding shelving.

[0009] In the selected image, obtain the center pixel coordinates of the directly imaged area and the center pixel coordinates of the mirrored imaged area; The center pixel coordinates of the direct imaging region and the center pixel coordinates of the mirror imaging region are used as a set of spatial position mapping relationships; By traversing all directly imaged areas, multiple sets of spatial location mapping relationships are obtained; Based on multiple sets of spatial location mapping relationships, a visual echo path mapping model is calibrated and generated.

[0010] Preferably, in S1, a visual echo path mapping model is calibrated and generated based on multiple sets of spatial location mapping relationships, as follows: Obtain the camera coordinate system; Based on multiple sets of spatial location mapping relationships, the plane equation of the shelf surface in the camera coordinate system is obtained by using the direct linear transformation method; Using the law of specular reflection, with the plane equation as the reflecting surface, the center pixel coordinates of the directly imaged area in multiple sets of spatial position mapping relationships as the source point coordinates, and the center pixel coordinates of the corresponding mirrored imaged area as the mirror projection point coordinates, a mapping function from the source point coordinates to the mirror projection point coordinates is generated by the parameter fitting method, and this mapping function is used as the visual echo path mapping model.

[0011] Preferably, in S2, a set of candidate pedestrian targets is generated based on the image sequence corresponding to subsequent acquisition cycles, as follows: The image sequences corresponding to subsequent acquisition cycles are used as warning image sequences; multiple bounding box sequences are obtained based on the warning image sequences; each bounding box sequence is used as a candidate pedestrian target, and all candidate pedestrian targets are obtained to form a candidate pedestrian target set; This process involves obtaining multiple bounding box sequences based on the warning image sequence; each bounding box sequence is treated as a candidate pedestrian target, and all candidate pedestrian targets are collected to form a candidate pedestrian target set, as follows: Traverse multiple images in the early warning image sequence and acquire one frame at a time in the order of acquisition time as the current image; The image object detection method is used to detect pedestrian targets in the current image. If a pedestrian target is detected in the current image, the bounding box corresponding to each pedestrian target is obtained and the acquisition time of each bounding box is recorded. If no pedestrian target is detected in the current image, the next frame of the warning image sequence is traversed. After the early warning image sequence is traversed, all bounding boxes corresponding to all acquisition times in the early warning image sequence are taken as bounding boxes to be associated. As an explanation, if no bounding box corresponding to a pedestrian target is found after the traversal of the warning image sequence is completed, the traversal of the image sequence corresponding to the next acquisition cycle will continue. A multi-target tracking algorithm is used to perform cross-frame association of bounding boxes to be associated, thereby obtaining the cross-frame association relationship between each bounding box to be associated. Based on the cross-frame association relationship between each bounding box to be associated, several bounding boxes with cross-frame association relationship are arranged in chronological order of acquisition time to form a bounding box sequence, resulting in multiple bounding box sequences. Each bounding box sequence is treated as a candidate pedestrian target, and all candidate pedestrian targets are obtained to form a candidate pedestrian target set; For clarification, a bounding box sequence refers to several bounding boxes to be associated that belong to the same pedestrian at multiple acquisition times.

[0012] Preferably, in S2, a reference pedestrian target set is generated based on the candidate pedestrian target set, as follows: For any candidate pedestrian target, the center pixel coordinates of each bounding box to be associated in the candidate pedestrian target are obtained as trajectory coordinates; Arrange all trajectory coordinates in chronological order of acquisition time to form a sequence of target position changes corresponding to the candidate pedestrian target; Optical flow estimation method is used to calculate the optical flow of multiple images in the early warning image sequence to obtain the optical flow field corresponding to each image; Based on the optical flow field corresponding to each image, the optical flow information corresponding to the background region in the optical flow field is extracted to obtain the image background flow field corresponding to each image. For clarification, this warning image sequence refers to a warning image sequence in which there are bounding boxes corresponding to pedestrian targets.

[0013] Obtain the acquisition time corresponding to each trajectory coordinate in the target position change sequence; For any trajectory coordinate, obtain the background flow field of the image at the same acquisition time as the trajectory coordinate; locate the pixel position at the same position as the trajectory coordinate in the background flow field of the image, and use it as the selected pixel position; extract the background optical flow vector at the selected pixel position from the background flow field of the image, and use it as the background optical flow vector corresponding to the trajectory coordinate; Obtain the background optical flow vector corresponding to the trajectory coordinates in the target position change sequence; For any trajectory coordinate, subtract the background optical flow vector corresponding to the trajectory coordinate from the trajectory coordinate to obtain the compensated trajectory coordinate corresponding to the trajectory coordinate; Obtain the compensated trajectory coordinates corresponding to each trajectory coordinate, and arrange the compensated trajectory coordinates according to the time sequence of the acquisition time to obtain the sequence of position changes after compensation; Based on the compensated position change sequence, determine whether the candidate pedestrian target has continuous position changes. If there are continuous position changes, mark the candidate pedestrian target as a reference pedestrian target. Obtain all reference pedestrian targets to form a reference pedestrian target set; As an explanation, if the reference pedestrian target set is empty, the process continues to iterate through the image sequence corresponding to the next acquisition cycle. In other words, if the reference pedestrian target set is empty, it is determined that there are no pedestrian targets satisfying the actual motion conditions within the current acquisition cycle, the processing flow for the current acquisition cycle ends, and the candidate pedestrian target detection and reference pedestrian target recognition process is re-executed for the image sequence corresponding to the next acquisition cycle. This situation arises because all candidate pedestrian targets detected in the image are mirrored targets formed by reflections from the shelf, or there are falsely detected targets in the image, but no real pedestrians.

[0014] Preferably, in S2, based on the compensated position change sequence, it is determined whether the candidate pedestrian target has continuous position changes. If continuous position changes exist, the candidate pedestrian target is marked as a reference pedestrian target, as follows: In the compensated position change sequence, the Euclidean distance between all adjacent compensated trajectory coordinates is obtained to form an Euclidean distance sequence; in the Euclidean distance sequence, if M consecutive Euclidean distances are greater than the image positioning error, it is determined that the candidate pedestrian target has continuous position changes; M is a positive integer greater than or equal to 2; Image positioning error is generated in the following way: Under the condition that the forklift is stationary and the acquisition environment remains unchanged, acquire static scene images corresponding to multiple consecutive acquisition moments; An object detection algorithm is used to detect objects in each static scene image to obtain multiple bounding boxes corresponding to the same fixed object. Obtain the center pixel coordinates of each bounding box corresponding to the same fixed target, and use them as the detection center coordinates; The abscissa components of multiple detection center coordinates are summed and divided by the number of detection center coordinates to obtain the average abscissa. The average ordinate is obtained by summing the ordinate components of multiple detection center coordinates and dividing by the number of detection center coordinates. The average x-coordinate and average y-coordinate are combined to obtain the average center pixel coordinates; Obtain the Euclidean distance between the coordinates of each detection center and the average center pixel coordinates; take the maximum value of the Euclidean distance between the coordinates of each detection center and the average center pixel coordinates as the image localization error.

[0015] Preferably, in S3, a set of mirror prediction regions is generated based on the reference pedestrian target set, as follows: In the set of reference pedestrian targets, for any reference pedestrian target, obtain the last bounding box to be associated in that reference pedestrian target and use it as the target bounding box; Obtain the center pixel coordinates of the target bounding box, and use them as the target pixel coordinates; Input the target pixel coordinates into the visual echo path mapping model to obtain the mirror projection point coordinates corresponding to the reference pedestrian target, and use the mirror projection point coordinates corresponding to the reference pedestrian target as the mirror projection position. Obtain the size of the target bounding box, and use the mirror projection position as the center point to generate a region with the same size as the target bounding box. Use this region as the mirror prediction region. Obtain all mirror prediction regions and form a mirror prediction region set.

[0016] Preferably, in S4, a set of real pedestrian targets is generated based on the candidate pedestrian target set and the mirror prediction region set, as follows: From the set of candidate pedestrian targets, the last bounding box to be associated in each candidate pedestrian target is obtained as the target tracking bounding box; Obtain the center pixel coordinates of each target tracking bounding box; for any target tracking bounding box, if the center pixel coordinates of the target tracking bounding box are located within any mirror prediction region in the mirror prediction region set, then mark the candidate pedestrian target corresponding to the center pixel coordinates of the target tracking bounding box as a mirror pedestrian target; In the candidate pedestrian target set, all marked mirrored pedestrian targets are deleted, and the remaining candidate pedestrian targets are taken as real pedestrian targets to obtain the real pedestrian target set.

[0017] Preferably, in S5, it is determined whether to generate a blind spot safety warning signal for real pedestrian targets in the real pedestrian target set based on the blind spot warning range of the forklift, as follows: In the set of real pedestrian targets, obtain the last bounding box to be associated in each real pedestrian target and use it as the real pedestrian tracking bounding box; Obtain the center pixel coordinates of the bounding boxes for each real pedestrian tracking object; For any real pedestrian tracking bounding box, if the center pixel coordinates of the real pedestrian tracking bounding box are within the forklift blind spot warning range, a blind spot safety warning signal is generated.

[0018] The forklift blind spot visual perception and safety warning method based on edge computing proposed in this invention has the following beneficial technical effects: This application utilizes inherent components on the forklift body as fixed features in a forklift without pedestrian detection to obtain the spatial position mapping relationship between the direct imaging area of ​​the fixed features and the mirror imaging area formed by reflection from the shelf surface. Based on the spatial position mapping relationship, a visual echo path mapping model is generated. During subsequent forklift operations, candidate pedestrian targets are analyzed and reference pedestrian targets are selected. The visual echo path mapping model is then used to predict the corresponding mirror prediction area based on the position of the reference pedestrian target, thereby identifying and eliminating mirror pedestrian targets formed by shelf reflection. This effectively distinguishes between real pedestrian targets and false reflection targets, greatly alleviating the problem in existing technologies where images captured by vehicle-mounted cameras in metal shelf environments easily show pedestrian mirror targets formed by reflection from the shelf surface. This makes it difficult for existing visual perception methods based on target detection to accurately distinguish between real pedestrian targets and false targets formed by shelf reflection, easily leading to false alarms and reducing the accuracy of safety warnings. Attached Figure Description

[0019] Figure 1 This is a flowchart of a forklift blind spot visual perception and safety warning method based on edge computing according to the present invention. Detailed Implementation

[0020] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0021] like Figure 1 The method for visual perception and safety warning of forklift blind spots based on edge computing, as shown, includes the following steps: S1. During forklift operation, acquire an image sequence corresponding to one acquisition cycle. The acquisition cycle includes multiple acquisition moments, and the image sequence includes multiple images, with one image corresponding to each acquisition moment. As an explanation: Multiple images are arranged in chronological order according to the time of acquisition to form an image sequence; As an explanation, images can be acquired through the vehicle's onboard camera during forklift operation.

[0022] Pedestrian target detection is performed on each frame of the image sequence. When no pedestrian target is detected in any frame of the image sequence, the shelf mirror reflection feature analysis of the image sequence is triggered to determine whether to generate a visual echo path mapping model. In an optional embodiment, when performing pedestrian target detection on each frame of the image, the image object detection method is used to perform pedestrian target detection on each frame of the image. If a pedestrian target exists, the bounding box corresponding to the pedestrian target is output.

[0023] In an optional embodiment, in S1, the shelf specular reflection feature analysis of the image sequence is triggered to determine whether to generate a visual echo path mapping model, as follows: The analysis of the mirror reflection characteristics of the shelf includes: A fixed feature of the forklift itself is a component with edges or corners on its body. For illustration, inherent components include: forklift fork arms, overhead guard support struts, steering wheel outer edge, and vehicle warning light outer contour; Obtain the geometric contour of the forklift's own fixed features; In the image sequence, one frame is acquired sequentially according to the order of acquisition time as the current frame image, and the following operations are performed on each current frame image: An image segmentation method is used to segment the target region from the current frame image that matches the geometric contour of the forklift's own fixed features; If the target region is segmented, the target region is taken as the direct imaging region corresponding to the fixed features of the forklift itself, and the subsequent frames in the image sequence are traversed. If the target region is not segmented in any of the images in the image sequence, it is determined that there are no fixed features of the forklift itself in the current image sequence, and the analysis of the mirror reflection features of the shelf is terminated. Once the target region is segmented, multiple images in the image sequence are traversed to obtain all directly imaged regions corresponding to the fixed features of the forklift itself. For any directly imaged region, the image containing that region is selected, and the selected image is horizontally flipped to obtain the flipped image. An image segmentation method is used to segment out the region from the flipped image that matches the geometric contour of the forklift's own fixed features, which is then used as the mirror flipped region. The mirrored area is horizontally flipped to obtain the mirrored imaging area corresponding to the fixed features of the forklift itself. As an explanation, the mirrored imaging area is the mirrored imaging area formed by the reflection of the fixed features of the forklift itself onto the surface of the surrounding shelving.

[0024] In the selected image, obtain the center pixel coordinates of the directly imaged area and the center pixel coordinates of the mirrored imaged area; The center pixel coordinates of the direct imaging region and the center pixel coordinates of the mirror imaging region are used as a set of spatial position mapping relationships; By traversing all directly imaged areas, multiple sets of spatial location mapping relationships are obtained; Based on multiple sets of spatial location mapping relationships, a visual echo path mapping model is calibrated and generated. In an optional embodiment, in S1, a visual echo path mapping model is calibrated and generated based on multiple sets of spatial location mapping relationships, as follows: Obtain the camera coordinate system; Based on multiple sets of spatial location mapping relationships, the plane equation of the shelf surface in the camera coordinate system is obtained by using the direct linear transformation method; Using the law of specular reflection, with the plane equation as the reflecting surface, the center pixel coordinates of the directly imaged area in multiple sets of spatial position mapping relationships as the source point coordinates, and the center pixel coordinates of the corresponding mirrored imaged area as the mirror projection point coordinates, a mapping function from the source point coordinates to the mirror projection point coordinates is generated by the parameter fitting method, and this mapping function is used as the visual echo path mapping model.

[0025] S2. After generating the visual echo path mapping model in any acquisition cycle, the acquisition cycle following that acquisition cycle will be taken as the subsequent acquisition cycle. Obtain the image sequences corresponding to subsequent acquisition cycles, generate a candidate pedestrian target set based on the image sequences corresponding to subsequent acquisition cycles, and generate a reference pedestrian target set based on the candidate pedestrian target set. In an optional embodiment, in S2, a set of candidate pedestrian targets is generated based on the image sequence corresponding to subsequent acquisition cycles, as follows: The image sequences corresponding to subsequent acquisition cycles are used as warning image sequences; multiple bounding box sequences are obtained based on the warning image sequences; each bounding box sequence is used as a candidate pedestrian target, and all candidate pedestrian targets are obtained to form a candidate pedestrian target set; This process involves obtaining multiple bounding box sequences based on the warning image sequence; each bounding box sequence is treated as a candidate pedestrian target, and all candidate pedestrian targets are collected to form a candidate pedestrian target set, as follows: Traverse multiple images in the early warning image sequence and acquire one frame at a time in the order of acquisition time as the current image; The image object detection method is used to detect pedestrian targets in the current image. If a pedestrian target is detected in the current image, the bounding box corresponding to each pedestrian target is obtained and the acquisition time of each bounding box is recorded. If no pedestrian target is detected in the current image, the next frame of the warning image sequence is traversed. After the early warning image sequence is traversed, all bounding boxes corresponding to all acquisition times in the early warning image sequence are taken as bounding boxes to be associated. As an explanation, if no bounding box corresponding to a pedestrian target is found after the traversal of the warning image sequence is completed, the traversal of the image sequence corresponding to the next acquisition cycle will continue. A multi-target tracking algorithm is used to perform cross-frame association of bounding boxes to be associated, thereby obtaining the cross-frame association relationship between each bounding box to be associated. Based on the cross-frame association relationship between each bounding box to be associated, several bounding boxes with cross-frame association relationship are arranged in chronological order of acquisition time to form a bounding box sequence, resulting in multiple bounding box sequences. Each bounding box sequence is treated as a candidate pedestrian target, and all candidate pedestrian targets are obtained to form a candidate pedestrian target set; For clarification, a bounding box sequence refers to several bounding boxes to be associated that belong to the same pedestrian at multiple acquisition times. In an optional embodiment, in S2, a reference pedestrian target set is generated based on the candidate pedestrian target set, as follows: For any candidate pedestrian target, the center pixel coordinates of each bounding box to be associated in the candidate pedestrian target are obtained as trajectory coordinates; Arrange all trajectory coordinates in chronological order of acquisition time to form a sequence of target position changes corresponding to the candidate pedestrian target; Optical flow estimation method is used to calculate the optical flow of multiple images in the early warning image sequence to obtain the optical flow field corresponding to each image; Based on the optical flow field corresponding to each image, the optical flow information corresponding to the background region in the optical flow field is extracted to obtain the image background flow field corresponding to each image. For clarification, this warning image sequence refers to a warning image sequence in which there are bounding boxes corresponding to pedestrian targets.

[0026] Obtain the acquisition time corresponding to each trajectory coordinate in the target position change sequence; For any trajectory coordinate, obtain the background flow field of the image at the same acquisition time as the trajectory coordinate; locate the pixel position at the same position as the trajectory coordinate in the background flow field of the image, and use it as the selected pixel position; extract the background optical flow vector at the selected pixel position from the background flow field of the image, and use it as the background optical flow vector corresponding to the trajectory coordinate; Obtain the background optical flow vector corresponding to the trajectory coordinates in the target position change sequence; For any trajectory coordinate, subtract the background optical flow vector corresponding to the trajectory coordinate from the trajectory coordinate to obtain the compensated trajectory coordinate corresponding to the trajectory coordinate; Obtain the compensated trajectory coordinates corresponding to each trajectory coordinate, and arrange the compensated trajectory coordinates according to the time sequence of the acquisition time to obtain the sequence of position changes after compensation; Based on the compensated position change sequence, determine whether the candidate pedestrian target has continuous position changes. If there are continuous position changes, mark the candidate pedestrian target as a reference pedestrian target. Obtain all reference pedestrian targets to form a reference pedestrian target set; As an explanation, if the reference pedestrian target set is empty, the image sequence corresponding to the next acquisition cycle continues to be traversed; that is, if the reference pedestrian target set is empty, it is determined that there is no pedestrian target that meets the real motion conditions in the current acquisition cycle, the processing flow of the current acquisition cycle ends, and the candidate pedestrian target detection and reference pedestrian target recognition process is re-executed for the image sequence corresponding to the next acquisition cycle. This situation occurs because all candidate pedestrian targets detected in the image are mirror targets formed by reflections from the shelf, or there are falsely detected targets in the image, but no real pedestrians. In an optional embodiment, in S2, based on the compensated position change sequence, it is determined whether the candidate pedestrian target has continuous position changes. If continuous position changes exist, the candidate pedestrian target is marked as a reference pedestrian target, as follows: In the compensated position change sequence, the Euclidean distance between all adjacent compensated trajectory coordinates is obtained to form an Euclidean distance sequence; in the Euclidean distance sequence, if M consecutive Euclidean distances are greater than the image positioning error, it is determined that the candidate pedestrian target has continuous position changes; M is a positive integer greater than or equal to 2; Image positioning error is generated in the following way: Under the condition that the forklift is stationary and the acquisition environment remains unchanged, acquire static scene images corresponding to multiple consecutive acquisition moments; An object detection algorithm is used to detect objects in each static scene image to obtain multiple bounding boxes corresponding to the same fixed object. Obtain the center pixel coordinates of each bounding box corresponding to the same fixed target, and use them as the detection center coordinates; The abscissa components of multiple detection center coordinates are summed and divided by the number of detection center coordinates to obtain the average abscissa. The average ordinate is obtained by summing the ordinate components of multiple detection center coordinates and dividing by the number of detection center coordinates. The average x-coordinate and average y-coordinate are combined to obtain the average center pixel coordinates; Obtain the Euclidean distance between the coordinates of each detection center and the average center pixel coordinates; take the maximum value of the Euclidean distance between the coordinates of each detection center and the average center pixel coordinates as the image localization error; S3. Generate a set of mirror prediction regions based on the reference pedestrian target set; In an optional embodiment, in S3, a set of mirror prediction regions is generated based on the reference pedestrian target set, as follows: In the set of reference pedestrian targets, for any reference pedestrian target, obtain the last bounding box to be associated in that reference pedestrian target and use it as the target bounding box; Obtain the center pixel coordinates of the target bounding box, and use them as the target pixel coordinates; Input the target pixel coordinates into the visual echo path mapping model to obtain the mirror projection point coordinates corresponding to the reference pedestrian target, and use the mirror projection point coordinates corresponding to the reference pedestrian target as the mirror projection position. Obtain the size of the target bounding box, and use the mirror projection position as the center point to generate a region with the same size as the target bounding box. Use this region as the mirror prediction region. Obtain all mirror prediction regions and form a mirror prediction region set; S4. Generate a set of real pedestrian targets based on the candidate pedestrian target set and the mirror prediction region set; In an optional embodiment, in S4, a set of real pedestrian targets is generated based on the candidate pedestrian target set and the mirror prediction region set, as follows: From the set of candidate pedestrian targets, the last bounding box to be associated in each candidate pedestrian target is obtained as the target tracking bounding box; Obtain the center pixel coordinates of each target tracking bounding box; for any target tracking bounding box, if the center pixel coordinates of the target tracking bounding box are located within any mirror prediction region in the mirror prediction region set, then mark the candidate pedestrian target corresponding to the center pixel coordinates of the target tracking bounding box as a mirror pedestrian target; In the candidate pedestrian target set, all marked mirrored pedestrian targets are deleted, and the remaining candidate pedestrian targets are taken as real pedestrian targets to obtain the real pedestrian target set; S5. Obtain the blind spot warning range of the forklift; determine whether to generate a blind spot safety warning signal for real pedestrian targets in the real pedestrian target set based on the blind spot warning range of the forklift; In an optional embodiment, in S5, it is determined whether to generate a blind spot safety warning signal for real pedestrian targets in the set of real pedestrian targets based on the blind spot warning range of the forklift, as follows: In the set of real pedestrian targets, obtain the last bounding box to be associated in each real pedestrian target and use it as the real pedestrian tracking bounding box; Obtain the center pixel coordinates of the bounding boxes for each real pedestrian tracking object; For any real pedestrian tracking bounding box, if the center pixel coordinates of the real pedestrian tracking bounding box are within the forklift blind spot warning range, a blind spot safety warning signal is generated. This application utilizes inherent components on the forklift body as fixed features in a forklift without pedestrian detection to obtain the spatial position mapping relationship between the direct imaging area of ​​the fixed features and the mirror imaging area formed by reflection from the shelf surface. Based on the spatial position mapping relationship, a visual echo path mapping model is generated. During subsequent forklift operations, candidate pedestrian targets are analyzed and reference pedestrian targets are selected. The visual echo path mapping model is then used to predict the corresponding mirror prediction area based on the position of the reference pedestrian target, thereby identifying and eliminating mirror pedestrian targets formed by shelf reflection. This effectively distinguishes between real pedestrian targets and false reflection targets, greatly alleviating the problem in existing technologies where images captured by vehicle-mounted cameras in metal shelf environments easily show pedestrian mirror targets formed by reflection from the shelf surface. This makes it difficult for existing visual perception methods based on target detection to accurately distinguish between real pedestrian targets and false targets formed by shelf reflection, easily leading to false alarms and reducing the accuracy of safety warnings.

[0027] For clarification, "acquisition" in this application refers to obtaining the required content or data using existing technical means.

[0028] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0029] In the embodiments provided by this invention, it should be understood that the disclosed system or method can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative; for instance, the division of modules is only a logical functional division, and there may be other division methods in actual implementation.

[0030] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0031] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in the form of hardware plus software functional modules.

[0032] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the basic characteristics of the present invention.

[0033] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for visual perception and safety warning of forklift blind spots based on edge computing, characterized in that, Includes the following steps: S1. During forklift operation, acquire an image sequence corresponding to a collection cycle; the collection cycle includes multiple collection times, the image sequence includes multiple images, and each collection time corresponds to one image; perform pedestrian target detection on each frame of the image sequence; when no pedestrian target is detected in each frame of the image sequence, trigger the analysis of the shelf mirror reflection features of the image sequence to determine whether to generate a visual echo path mapping model. S2. After generating the visual echo path mapping model in any acquisition cycle, the acquisition cycle following that acquisition cycle is taken as the subsequent acquisition cycle; obtain the image sequence corresponding to the subsequent acquisition cycle, and generate a candidate pedestrian target set based on the image sequence corresponding to the subsequent acquisition cycle. Generate a reference pedestrian target set based on the candidate pedestrian target set; S3. Generate a set of mirror prediction regions based on the reference pedestrian target set; S4. Generate a set of real pedestrian targets based on the candidate pedestrian target set and the mirror prediction region set; S5. Obtain the blind spot warning range for forklifts; Determine whether to generate a blind spot safety warning signal for real pedestrian targets in the real pedestrian target set based on the blind spot warning range of the forklift; In S1, the analysis of the shelf mirror reflection features of the image sequence is triggered to determine whether to generate a visual echo path mapping model, as follows: The analysis of the mirror reflection characteristics of the shelf includes: A fixed feature of the forklift itself is a component with edges or corners on its body. Obtain the geometric contour of the forklift's own fixed features; In the image sequence, one frame is acquired sequentially according to the order of acquisition time as the current frame image, and the following operations are performed on each current frame image: An image segmentation method is used to segment the target region from the current frame image that matches the geometric contour of the forklift's own fixed features; If the target region is segmented, the target region is taken as the direct imaging region corresponding to the fixed features of the forklift itself, and the subsequent frames in the image sequence are traversed. If the target region is not segmented in any of the images in the image sequence, it is determined that there are no fixed features of the forklift itself in the current image sequence, and the analysis of the mirror reflection features of the shelf is terminated. Once the target region is segmented, multiple images in the image sequence are traversed to obtain all directly imaged regions corresponding to the fixed features of the forklift itself. For any directly imaged region, the image containing that region is selected, and the selected image is horizontally flipped to obtain the flipped image. An image segmentation method is used to segment out the region from the flipped image that matches the geometric contour of the forklift's own fixed features, which is then used as the mirror flipped region. The mirrored area is horizontally flipped to obtain the mirrored imaging area corresponding to the fixed features of the forklift itself. In the selected image, obtain the center pixel coordinates of the directly imaged area and the center pixel coordinates of the mirrored imaged area; The center pixel coordinates of the direct imaging region and the center pixel coordinates of the mirror imaging region are used as a set of spatial position mapping relationships; By traversing all directly imaged areas, multiple sets of spatial location mapping relationships are obtained; Based on multiple sets of spatial location mapping relationships, a visual echo path mapping model is calibrated and generated. In S1, a visual echo path mapping model is calibrated and generated based on multiple sets of spatial location mapping relationships, as follows: Obtain the camera coordinate system; Based on multiple sets of spatial location mapping relationships, the plane equation of the shelf surface in the camera coordinate system is obtained by using the direct linear transformation method; Using the law of specular reflection, with the plane equation as the reflecting surface, the center pixel coordinates of the directly imaged area in multiple sets of spatial position mapping relationships as the source point coordinates, and the center pixel coordinates of the corresponding mirrored imaged area as the mirror projection point coordinates, a mapping function from the source point coordinates to the mirror projection point coordinates is generated by the parameter fitting method, and this mapping function is used as the visual echo path mapping model. In S2, a set of candidate pedestrian targets is generated based on the image sequences corresponding to subsequent acquisition cycles, as follows: The image sequences corresponding to subsequent acquisition cycles are used as warning image sequences; multiple bounding box sequences are obtained based on the warning image sequences; each bounding box sequence is used as a candidate pedestrian target, and all candidate pedestrian targets are obtained to form a candidate pedestrian target set; This process involves obtaining multiple bounding box sequences based on the warning image sequence; each bounding box sequence is treated as a candidate pedestrian target, and all candidate pedestrian targets are collected to form a candidate pedestrian target set, as follows: Traverse multiple images in the early warning image sequence and acquire one frame at a time in the order of acquisition time as the current image; The image object detection method is used to detect pedestrian targets in the current image. If a pedestrian target is detected in the current image, the bounding box corresponding to each pedestrian target is obtained and the acquisition time of each bounding box is recorded. If no pedestrian target is detected in the current image, the next frame of the warning image sequence is traversed. After the early warning image sequence is traversed, all bounding boxes corresponding to all acquisition times in the early warning image sequence are taken as bounding boxes to be associated. A multi-target tracking algorithm is used to perform cross-frame association of bounding boxes to be associated, thereby obtaining the cross-frame association relationship between each bounding box to be associated. Based on the cross-frame association relationship between each bounding box to be associated, several bounding boxes with cross-frame association relationship are arranged in chronological order of acquisition time to form a bounding box sequence, resulting in multiple bounding box sequences. Each bounding box sequence is treated as a candidate pedestrian target, and all candidate pedestrian targets are obtained to form a candidate pedestrian target set; In S2, a reference pedestrian target set is generated based on the candidate pedestrian target set, as follows: For any candidate pedestrian target, the center pixel coordinates of each bounding box to be associated in the candidate pedestrian target are obtained as trajectory coordinates; Arrange all trajectory coordinates in chronological order of acquisition time to form a sequence of target position changes corresponding to the candidate pedestrian target; Optical flow estimation method is used to calculate the optical flow of multiple images in the early warning image sequence to obtain the optical flow field corresponding to each image; Based on the optical flow field corresponding to each image, the optical flow information corresponding to the background region in the optical flow field is extracted to obtain the image background flow field corresponding to each image; Obtain the acquisition time corresponding to each trajectory coordinate in the target position change sequence; For any trajectory coordinate, obtain the background flow field of the image at the same acquisition time as the trajectory coordinate; locate the pixel position at the same position as the trajectory coordinate in the background flow field of the image, and use it as the selected pixel position; extract the background optical flow vector at the selected pixel position from the background flow field of the image, and use it as the background optical flow vector corresponding to the trajectory coordinate; Obtain the background optical flow vector corresponding to the trajectory coordinates in the target position change sequence; For any trajectory coordinate, subtract the background optical flow vector corresponding to the trajectory coordinate from the trajectory coordinate to obtain the compensated trajectory coordinate corresponding to the trajectory coordinate; Obtain the compensated trajectory coordinates corresponding to each trajectory coordinate, and arrange the compensated trajectory coordinates according to the time sequence of the acquisition time to obtain the sequence of position changes after compensation; Based on the compensated position change sequence, determine whether the candidate pedestrian target has continuous position changes. If there are continuous position changes, mark the candidate pedestrian target as a reference pedestrian target. Obtain all reference pedestrian targets to form a reference pedestrian target set; In S2, based on the compensated position change sequence, it is determined whether the candidate pedestrian target has continuous position changes. If continuous position changes exist, the candidate pedestrian target is marked as a reference pedestrian target, as follows: In the compensated position change sequence, the Euclidean distance between all adjacent compensated trajectory coordinates is obtained to form an Euclidean distance sequence; in the Euclidean distance sequence, if M consecutive Euclidean distances are greater than the image positioning error, it is determined that the candidate pedestrian target has continuous position changes; M is a positive integer greater than or equal to 2; Image positioning error is generated in the following way: Under the condition that the forklift is stationary and the acquisition environment remains unchanged, acquire static scene images corresponding to multiple consecutive acquisition moments; An object detection algorithm is used to detect objects in each static scene image to obtain multiple bounding boxes corresponding to the same fixed object. Obtain the center pixel coordinates of each bounding box corresponding to the same fixed target, and use them as the detection center coordinates; The abscissa components of multiple detection center coordinates are summed and divided by the number of detection center coordinates to obtain the average abscissa. The average ordinate is obtained by summing the ordinate components of multiple detection center coordinates and dividing by the number of detection center coordinates. The average x-coordinate and average y-coordinate are combined to obtain the average center pixel coordinates; Obtain the Euclidean distance between the coordinates of each detection center and the average center pixel coordinates; take the maximum value of the Euclidean distance between the coordinates of each detection center and the average center pixel coordinates as the image localization error; In S3, a set of mirror prediction regions is generated based on the reference pedestrian target set, as follows: In the set of reference pedestrian targets, for any reference pedestrian target, obtain the last bounding box to be associated in that reference pedestrian target and use it as the target bounding box; Obtain the center pixel coordinates of the target bounding box, and use them as the target pixel coordinates; Input the target pixel coordinates into the visual echo path mapping model to obtain the mirror projection point coordinates corresponding to the reference pedestrian target, and use the mirror projection point coordinates corresponding to the reference pedestrian target as the mirror projection position. Obtain the size of the target bounding box, and use the mirror projection position as the center point to generate a region with the same size as the target bounding box. Use this region as the mirror prediction region. Obtain all mirror prediction regions and form a mirror prediction region set; In S4, the set of real pedestrian targets is generated based on the candidate pedestrian target set and the mirror prediction region set, as follows: From the set of candidate pedestrian targets, the last bounding box to be associated in each candidate pedestrian target is obtained as the target tracking bounding box; Obtain the center pixel coordinates of each target tracking bounding box; for any target tracking bounding box, if the center pixel coordinates of the target tracking bounding box are located within any mirror prediction region in the mirror prediction region set, then mark the candidate pedestrian target corresponding to the center pixel coordinates of the target tracking bounding box as a mirror pedestrian target; In the candidate pedestrian target set, all marked mirrored pedestrian targets are deleted, and the remaining candidate pedestrian targets are taken as real pedestrian targets to obtain the real pedestrian target set.

2. The forklift blind spot visual perception and safety warning method based on edge computing according to claim 1, characterized in that, In S5, based on the forklift blind spot warning range, it is determined whether to generate a blind spot safety warning signal for real pedestrian targets in the real pedestrian target set, as follows: In the set of real pedestrian targets, obtain the last bounding box to be associated in each real pedestrian target and use it as the real pedestrian tracking bounding box; Obtain the center pixel coordinates of the bounding boxes for each real pedestrian tracking object; For any real pedestrian tracking bounding box, if the center pixel coordinates of the real pedestrian tracking bounding box are within the forklift blind spot warning range, a blind spot safety warning signal is generated.

Images