Projection marking method based on visual sensing

Abstract

The invention relates to the technical field of projection marking, and discloses a projection marking method based on visual sensing, and the method comprises the steps: obtaining a visible light image and a near-infrared image of a working environment of a forklift, carrying out the posture solving of an arrow pattern from the visible light image based on an arrow contour and a phase field, and obtaining a direction vector and a position coordinate of the arrow pattern; performing shielding judgment on projection pixels of the arrow pattern according to the direction vector and the position coordinates to obtain shielding pixel points; determining the visible pixel number of the visible pixels according to the shielding pixel points, and calculating the shielding rate of the projection pixels based on the visible pixel number and the total area of the projection template; screening out an optimal visible plane of the plane set by using a preset scoring function; and performing re-projection processing on the arrow template by using the optimal visible plane to obtain a bispectrum arrow map. According to the invention, the navigation continuity and safety in the dispatching process of the unmanned forklift are ensured.

Description

Technical Field

[0001] This invention relates to the field of projection signage technology, and in particular to a projection signage method based on visual sensing. Background Technology

[0002] In modern automated warehouses, numerous unmanned forklifts and shuttle pallets operate in parallel, with narrow material transfer aisles and dynamically changing routes. To avoid installing displays or high-power laser projectors on the vehicles, the industry is increasingly adopting top-mounted projection and ground-based arrow markings: the system uses a visible-near-infrared dual-spectrum projector to precisely project dynamic guide arrows onto the ground or the top of the pallet; a suspended dual-spectrum camera works synchronously with a TOF / LiDAR sensor to collect scene images, depth point clouds, and forklift attitude information in real time, and continuously corrects the projection homography matrix through a vision-depth fusion algorithm to ensure the arrows are always in the forklift's direction of travel. This solution combines low power consumption, ease of maintenance, and human-machine interface characteristics, providing intuitive guidance in day / night cycles, highly reflective metal shelves, or dusty environments, and is therefore becoming the mainstream navigation aid in automated warehousing.

[0003] However, in multi-vehicle collaborative operation scenarios, forklift lifting forks, pallet handling, or obstacle avoidance often results in large-scale occlusion of arrow projections, leading to navigation ambiguity. Most existing methods rely on single-spectrum visible light contours or fixed depth thresholds to determine occlusion. When the arrow contour is instantaneously covered by ≥30% of the fork arm or pallet, the system cannot accurately predict the occlusion area, select a new visible plane, and reproject within a single projection frame period. If the reprojection delay exceeds 200ms, the forklift scheduling algorithm has already executed based on expired instructions, exacerbating the risk of conflicts between vehicles and between vehicles and goods. Especially when fork arm pitch, extension, and vehicle vibration are superimposed, relying solely on RGB contours or single-plane depth judgment will produce angle-depth coupling errors, leading to occlusion judgment distortion, repeated arrow flashing, or prolonged absence, failing to meet the real-time and reliability requirements of "multi-target-obstacle avoidance self-updating" in dense warehousing. Therefore, there is an urgent need for a projection labeling method that can quickly fuse arrow bispectral features, forklift kinematic parameters, and multi-source depth data under complex occlusion dynamics, output the visible plane in real time, and complete reprojection, so as to ensure the navigation continuity and safety during the unmanned forklift scheduling process. Summary of the Invention

[0004] This invention provides a projection marking method based on visual sensing, the main purpose of which is to solve the problem that existing technologies cannot guarantee the continuity and safety of navigation during the scheduling of unmanned forklifts.

[0005] To achieve the above objectives, the present invention provides a projection marking method based on visual sensing, comprising: S1. Obtain visible light and near-infrared images of the forklift's working environment, extract the arrow outline of the projection device from the visible light image, and calculate the phase field of the frequency domain complex matrix in the near-infrared image. S2. Solve the attitude of the arrow pattern based on the arrow profile and phase field to obtain the direction vector and position coordinates of the arrow pattern; S3. Based on the direction vector and position coordinates, determine the occlusion of the projected pixels of the arrow pattern to obtain the occluded pixels. S4. Determine the number of visible pixels of the visible pixels based on the occluded pixels, and calculate the occlusion rate of the projected pixels based on the number of visible pixels and the total area of ​​the projection template. S5. Use a preset scoring function to filter out the optimal visible plane of the plane set; S6. Reproject the arrow template using the optimal visible plane to obtain the bispectral arrow pattern.

[0006] In a preferred embodiment, acquiring visible light and near-infrared images of the forklift's working environment, and extracting the arrow outline of the projection device from the visible light image, includes: The system uses cameras to capture near-infrared and visible light images of the forklift's working environment, covering the forklift, arrows, and the ground environment. The visible light image is converted to grayscale, and the highlighted outline edge of the arrow is extracted using the Canny edge detection algorithm; Morphological closing operations are used to filter the edges of the bright contour, fill the small holes in the edges of the bright contour, and remove discrete noise points to obtain the arrow contour.

[0007] In a preferred embodiment, calculating the phase field of the frequency domain complex matrix in the near-infrared image includes: Denoising preprocessing of near-infrared images; A two-dimensional Fourier transform is performed on the preprocessed near-infrared image to obtain a complex matrix in the frequency domain; The phase value of each element in the frequency domain complex matrix is ​​calculated using the arctangent function, and a phase field is generated based on all the phase values.

[0008] In a preferred embodiment, the arrow pattern is attitude-determined based on the arrow profile and phase field to obtain the direction vector and position coordinates of the arrow pattern, including: Image analysis is performed on the arrow contour to obtain the contour vertex coordinates, contour side length, and contour angle; Phase calculation is performed on the phase field to obtain a set of depth compensation values ​​and a set of angle compensation coefficients; Convert the contour vertex coordinates to the initial 3D vertex coordinates in the camera coordinate system; Based on the set of depth compensation values ​​and the set of angle compensation coefficients, and combined with the correction formula, the initial three-dimensional vertex coordinates are corrected to obtain the corrected three-dimensional vertex coordinates; The direction vector of the arrow pattern is calculated based on the starting and ending vertices of the corrected 3D vertex coordinates, combined with the vector calculation formula. Calculate the average vertex coordinates of the corrected 3D vertex coordinates, substitute the average vertex coordinates into the spatial coordinate mapping formula, and obtain the position coordinates of the arrow pattern.

[0009] In a preferred embodiment, occlusion determination is performed on the projected pixels of the arrow pattern based on the direction vector and position coordinates to obtain occluded pixel points, including: Based on the forklift motion parameters and forklift joint parameters, and combined with the forklift kinematic model, calculate the real-time forklift plane equation and forklift plane; Determine the plane normal vector of the fork arm based on the real-time plane equation of the fork arm; By traversing each projected pixel of the arrow pattern based on the direction vector and position coordinates, the direction of the light ray corresponding to the projected pixel is determined. Calculate the angle between the plane normal of the fork arm and the direction of the ray; Calculate the distance between the projected pixels and the plane of the fork arm; When the included angle is acute and the distance is less than a preset threshold, the projected pixel is determined as an occluded pixel.

[0010] In a preferred embodiment, the number of visible pixels is determined based on the occluded pixels, and the occlusion rate of the projected pixels is calculated based on the number of visible pixels and the total area of ​​the projection template, including: Iterate through all pixels of the projection template, count the number of pixels that are not marked as occluded pixels, and record them as the number of visible pixels; Substituting the number of visible pixels and the total area of ​​the projection template into the formula for calculating the occlusion rate yields the occlusion rate.

[0011] In a preferred embodiment, the optimal visible plane of the plane set is selected using a preset scoring function, including: Based on TOF point cloud data and LiDAR point cloud data, the set of planes in which visible pixels are not obscured by the forklift is fitted using the RANSAC plane fitting algorithm. Calculate the occlusion ratio of each plane in the plane set; A preset scoring function is generated based on the occlusion ratio and the direction vector of the arrow pattern; The plane set is scored using a preset scoring function to obtain the maximum score; The plane corresponding to the highest score is selected as the optimal visible plane of the plane set.

[0012] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention first acquires visible light and near-infrared bispectral images simultaneously within the same sampling period: the visible light path refines the arrow's highlighted outline through grayscale conversion, Canny, and morphological closing operations to ensure complete geometric shape before obstruction by forks or pallets; the near-infrared path performs a two-dimensional Fourier transform and phase field reconstruction on the same scene, continuously outputting high signal-to-noise depth-phase information even when the outline is momentarily obscured. The system fuses the two data streams, corrects the arrow's attitude in real time based on phase compensation, and embeds a depth-grayscale adaptive threshold into the occlusion prediction loop, achieving continuous tracking with visible light distortion and near-infrared fallback. This bispectral-phase collaborative mechanism significantly suppresses navigation interruptions caused by ≥30% outline loss, ensuring the recognizability and occlusion robustness of arrow commands in complex lighting and dusty environments.

[0013] 2. After obtaining the forklift plane equation in real time based on forklift kinematics, the system calculates the angle and distance of each pixel ray of the arrow for dual-condition occlusion discrimination and accurately labels the occlusion mask. Then, the mask result is back-matched to the registered and fused TOF and LiDAR point clouds, and a candidate unoccluded plane set is obtained by using multiple RANSAC fittings. For each plane, the occlusion ratio and the combined score of the normal and the angle of the arrow direction are calculated. The scoring function naturally encapsulates the angle-depth coupling error as a gradient constraint, which enables the system to automatically select the optimal visible plane in a single frame without manually setting a fixed threshold. It can stably complete the plane switching and projection plane update in the scene of forklift pitch, extension and retraction and vehicle vibration superimposed, and avoid the scheduling algorithm from accidentally triggering the collision threshold due to expired projection.

[0014] 3. On the selected optimal plane, this invention constructs a spatial mapping relationship between the arrow template and the plane, performs reprojection at the pixel level, and performs edge anti-aliasing, brightness, and contrast adaptive enhancement in real time; by embedding near-infrared phase features into the projection texture through bispectral frequency domain analysis, the newly generated arrows have high contrast and uniform geometric distortion control in both visible and near-infrared channels, and can continuously output clear and flicker-free navigation arrows under conditions of multiple vehicles crossing, rapid fork lifting and lowering, and ground vibration, thereby significantly reducing the probability of conflicts between unmanned forklifts and between forklifts and goods, and improving the overall scheduling efficiency and safety redundancy of dense warehousing environments. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a projection marking method based on visual sensing, provided in an embodiment of the present invention. The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0016] Reference Figure 1The diagram shown is a flowchart illustrating a projection marking method based on visual sensing according to an embodiment of the present invention. In this embodiment, the projection marking method based on visual sensing includes: S1. Obtain visible light and near-infrared images of the forklift's working environment, extract the arrow outline of the projection device from the visible light image, and calculate the phase field of the frequency domain complex matrix in the near-infrared image. In embodiments of the present invention, the following are included: The system uses cameras to capture near-infrared and visible light images of the forklift's working environment, covering the forklift, arrows, and the ground environment. Specifically, in a forklift operation scenario, at least one set of near-infrared and visible light cameras are deployed simultaneously. The cameras are installed in appropriate positions within the forklift's operating area to ensure that their shooting angles cover key areas such as the forklift's body, forks, the arrow pattern projected by the projection device, and the ground environment where the arrow is located. The cameras are then activated, and the near-infrared and visible light cameras are controlled to simultaneously capture images according to a preset image acquisition frequency. This ensures that each captured near-infrared and visible light image fully includes the forklift's appearance, the arrow's projected outline, and environmental information such as the texture and markings on the ground where the arrow is located. This process obtains near-infrared and visible light images that meet subsequent processing requirements and cover the forklift, the arrow, and the ground environment.

[0017] The visible light image is converted to grayscale, and the highlighted outline edge of the arrow is extracted using the Canny edge detection algorithm; Specifically, the acquired visible light image is first converted into a grayscale image. The red, green, and blue channel values ​​of each pixel are then calculated using a specific weighted average algorithm: grayscale value equals 0.299 multiplied by the red channel value, plus 0.587 multiplied by the green channel value, plus 0.114 multiplied by the blue channel value, resulting in a single-channel grayscale image containing only brightness information. Next, Gaussian filtering is applied to this grayscale image to reduce noise interference. A Gaussian kernel function is constructed, and a weighted average convolution operation is performed on the image pixels to smooth the image. Finally, the gradient magnitude and direction of the filtered grayscale image are calculated using edge detection operators such as the Sobel operator, calculating the gradient in both the horizontal and vertical directions. The gradient of each pixel is calculated, and then the gradient magnitude and arctangent are used to obtain the gradient magnitude and direction. Next, non-maximum suppression is applied to the gradient magnitude. Each pixel is iterated over, and its gradient magnitude is compared with the gradient magnitudes of its two adjacent pixels along the gradient direction. If the gradient magnitude of a pixel is not a local maximum, it is set to 0 to refine the edge. Finally, two thresholds are set: a high threshold and a low threshold. Pixels with gradient magnitudes greater than the high threshold are identified as strong edge points, pixels between the high and low thresholds and connected to strong edge points are identified as weak edge points, and the remaining pixels are identified as non-edge points. By connecting strong edge points and the weak edge points that meet the conditions, the highlighted outline edge of the arrow is extracted.

[0018] Morphological closing operations are used to filter the edges of the bright contour, fill the small holes in the edges of the bright contour, and remove discrete noise points to obtain the arrow contour.

[0019] Specifically, first, the structuring element used for the morphological closing operation is determined, and a suitable size and shape (such as circle or rectangle) are selected. The size of the structuring element is set according to the characteristics of small holes and discrete noise points in the highlighted contour edge. Then, the extracted arrow highlighted contour edge image is first subjected to a dilation operation. The selected structuring element is used to traverse the image, and the maximum pixel value in the area covered by the structuring element is assigned to the center pixel, causing the highlighted contour edge to expand outward, filling the small holes in the contour edge, and connecting adjacent contour parts. Next, the dilated image is subjected to an erosion operation. Similarly, the same structuring element is used to traverse the image, and the minimum pixel value in the area covered by the structuring element is assigned to the center pixel, restoring the highlighted contour edge to a width close to the original, removing redundant connections introduced by the dilation, and eliminating discrete noise points. Through the above morphological closing operation of dilation followed by erosion, the small holes in the highlighted contour edge are filled, and discrete noise points are removed, finally obtaining a complete, clear, and continuous arrow contour for subsequent pose calculation and other processes.

[0020] In this embodiment of the invention, calculating the phase field of the complex matrix in the frequency domain of a near-infrared image includes: Denoising preprocessing of near-infrared images; A two-dimensional Fourier transform is performed on the preprocessed near-infrared image to obtain a complex matrix in the frequency domain; The phase value of each element in the frequency domain complex matrix is ​​calculated using the arctangent function, and a phase field is generated based on all the phase values.

[0021] Specifically, to calculate the phase field of the frequency domain complex matrix in a near-infrared image, the acquired near-infrared image is first subjected to denoising preprocessing, such as Gaussian filtering. By constructing a suitable filter kernel, the image pixels are weighted and smoothed to effectively suppress noise interference in the image. Next, a two-dimensional Fourier transform is performed on the denoised near-infrared image to convert the image from the spatial domain to the frequency domain. The frequency domain complex matrix containing amplitude and phase information is obtained by calculating using the Fourier transform algorithm. Then, for each element in the frequency domain complex matrix, the arctangent function is used to calculate its corresponding phase value based on the real and imaginary parts of the complex number. Finally, the phase values ​​of all elements are integrated and, according to the spatial correspondence of the image pixels, a phase field that reflects the phase distribution characteristics of the near-infrared image is generated, providing basic data for subsequent phase field-based processing.

[0022] S2. Solve the attitude of the arrow pattern based on the arrow profile and phase field to obtain the direction vector and position coordinates of the arrow pattern; In this embodiment of the invention, the attitude of the arrow pattern is solved based on the arrow contour and phase field to obtain the direction vector and position coordinates of the arrow pattern, including: Image analysis is performed on the arrow contour to obtain the contour vertex coordinates, contour side length, and contour angle; Specifically, firstly, subpixel-level edge detection is performed on the arrow contour. By calculating the gray-level change rate in the gradient direction, the precise position of the contour is determined, resulting in a subpixel-precision contour point set. Next, polygon approximation is performed on the contour point set. Using the Douglas-Peucker algorithm and setting an appropriate distance threshold, a continuous sequence of contour points is fitted into a polygon composed of a finite number of vertices, and the vertex coordinates of the contour are extracted. Then, based on the extracted vertex coordinates, the Euclidean distance between adjacent vertices is calculated, and the total side length of the contour is obtained by accumulating these distances. The vector angle between adjacent sides is calculated through vector cross product and dot product operations. Combined with the connection order of the vertices, the interior angles of the contour are determined, thus completing the image analysis of the arrow contour and obtaining the contour vertex coordinates, contour side length, and contour angles.

[0023] Phase calculation is performed on the phase field to obtain a set of depth compensation values ​​and a set of angle compensation coefficients; Specifically, a phase unwrapping algorithm is used to process the phase field. Because the original phase field has discontinuities caused by phase jumps, the least squares method is used to traverse the phase field pixels along a reasonable path, calculate the phase difference between adjacent pixels and accumulate them, eliminating 2π-integer multiple jumps and obtaining a continuous absolute phase distribution. Then, based on the calibration parameters of the near-infrared imaging system, including the system's baseline distance, focal length, and wavelength, a mapping model between absolute phase and actual depth is established. The continuous absolute phase values ​​are substituted into the model to calculate the depth deviation of the corresponding pixels, and the depth deviations of all pixels are organized into a set of depth compensation values. Simultaneously, based on the correlation between the phase gradient direction in the phase field and the arrow contour posture, the spatial distribution law of phase change is analyzed. Combined with information such as the arrow contour direction vector, compensation parameters used to correct the arrow posture angle are calculated. These parameters are integrated into a set of angle compensation coefficients, thus completing the phase calculation of the phase field and outputting the set of depth compensation values ​​and the set of angle compensation coefficients, providing data support for subsequent 3D vertex correction.

[0024] Convert the contour vertex coordinates to the initial 3D vertex coordinates in the camera coordinate system; Specifically, using the intrinsic parameter matrix obtained from camera calibration, which contains parameters such as the camera's focal length and principal point coordinates, the pixel coordinates of the contour vertices are transformed from the image coordinate system to the camera's normalized plane coordinate system, resulting in normalized coordinates. Next, based on the depth information obtained from the phase field calculation of the near-infrared image, a corresponding depth value is assigned to each normalized coordinate, expanding it into a 3D coordinate system. Then, considering the camera's extrinsic parameter matrix, which describes the camera's position and orientation in the world coordinate system, the 3D coordinates based on the normalized plane are transformed to the camera coordinate system through rotation and translation transformations, obtaining the initial 3D vertex coordinates. Finally, the transformed initial 3D vertex coordinates are optimized and corrected. Combining known scene prior information or reference points, the accuracy of the coordinate transformation is further improved by minimizing reprojection errors, ensuring the accuracy of the initial 3D vertex coordinates.

[0025] Based on the set of depth compensation values ​​and the set of angle compensation coefficients, and combined with the correction formula, the initial 3D vertex coordinates are corrected to obtain the corrected 3D vertex coordinates. The correction formula is as follows: In the formula, These are the initial 3D vertex coordinates. It corrects the coordinates of three-dimensional vertices. It is the first in the set of angle compensation coefficients One element, It is the first in the depth compensation value set One element; Specifically, when correcting the initial 3D vertex coordinates, the first step is to match the corresponding depth compensation value and angle compensation coefficient for each initial 3D vertex coordinate. That is, the depth compensation value corresponding to the current vertex is obtained from the depth compensation value set. Obtain the corresponding angle compensation coefficient from the set of angle compensation coefficients. Next, the transformation matrix required for correction is constructed, which consists of angle compensation coefficients. Generates according to the matrix form in the formula, while also taking into account the depth compensation value. Determine the depth scaling factor; then, initialize the 3D vertex coordinates. The transformation matrix and depth scaling factor are used for calculations. First, the transformation matrix is ​​used to... An angle-related spatial transformation is performed, and then multiplied by a depth scaling factor to complete the coordinate correction calculation. Finally, the above matching, matrix construction and operation operations are performed on all initial 3D vertex coordinates one by one to obtain the corrected 3D vertex coordinates corresponding to each vertex. These are then integrated to form a set of corrected 3D vertex coordinates for subsequent pose calculation and other processes.

[0026] Based on the corrected 3D vertex coordinates of the starting and ending vertices, and using the vector calculation formula, the direction vector of the arrow pattern is calculated as follows: In the formula, It is the direction vector of the arrow pattern. It is the starting vertex. It is the final vertex. It is the vector magnitude; Specifically, when calculating the direction vector of the arrow pattern, firstly, based on the geometric features of the arrow outline and the connection order of the vertices, the starting vertex representing the starting position of the arrow and the ending vertex representing the ending position are identified and selected from the modified three-dimensional vertex coordinate set. Next, the coordinate difference between these two vertices is calculated by subtracting the coordinates of the starting vertex from the coordinates of the ending vertex, resulting in a vector that reflects the spatial displacement relationship from the starting vertex to the ending vertex. Then, the magnitude of the difference vector is calculated by taking the square root of the sum of the squares of the vector components, which quantifies the length of the vector. Finally, the difference vector is divided by its magnitude according to the formula... The result of the calculation is the direction vector of the arrow pattern. This vector is a unit vector and can accurately represent the direction of the arrow in space.

[0027] Calculate the average vertex coordinates of the corrected 3D vertex coordinates, and substitute these average vertex coordinates into the spatial coordinate mapping formula to obtain the position coordinates of the arrow pattern. The spatial coordinate mapping formula is as follows: In the formula, These are the position coordinates of the arrow pattern. It corrects the coordinates of three-dimensional vertices. It is an identifier for correcting the coordinates of three-dimensional vertices. It is the total number of corrected 3D vertex coordinates.

[0028] Specifically, when calculating the position coordinates of the arrow pattern, firstly, the set of modified 3D vertex coordinates is traversed to obtain all modified 3D vertex coordinates, and the total number of modified 3D vertex coordinates is determined; then, the total number of modified 3D vertex coordinates is summed, and each... The corresponding coordinate components are summed to obtain the total coordinate components. Then, the total coordinate components are divided by the total number of corrected 3D vertex coordinates to calculate the average vertex coordinates. The average vertex coordinates comprehensively reflect the positional concentration trend of all corrected 3D vertex coordinates. Finally, according to the spatial coordinate mapping formula, the calculated average vertex coordinates are directly used as the position coordinates of the arrow pattern to accurately represent the position of the arrow pattern in space.

[0029] S3. Based on the direction vector and position coordinates, determine the occlusion of the projected pixels of the arrow pattern to obtain the occluded pixels. In this embodiment of the invention, occlusion determination is performed on the projected pixels of the arrow pattern based on the direction vector and position coordinates to obtain occluded pixel points, including: Based on the forklift motion parameters and forklift joint parameters, and combined with the forklift kinematic model, calculate the real-time forklift plane equation and forklift plane; Specifically, when calculating the real-time forklift plane equation and forklift plane of the forklift, the forklift motion parameters are first obtained, including the forklift's travel speed, steering angle, forklift lifting height, forklift extension length, etc., as well as the forklift joint parameters, including basic data such as the rotation angle of each joint, joint spacing, and joint connection relationship. Then, based on the pre-established forklift kinematic model, which is constructed based on the DH parameter method, the forklift motion parameters and joint parameters are substituted into the model, and the spatial pose of each link of the forklift is calculated sequentially through homogeneous coordinate transformation, matrix operations, etc., to determine the key features on the forklift. The three-dimensional coordinates of points (such as the endpoints of the fork arm, hinge points, etc.) in the world coordinate system are determined. Then, using the coordinates of at least three non-collinear key feature points on the fork arm, a plane fitting algorithm (such as least squares fitting of the plane) is used to calculate the normal vector and plane constant term of the fork arm plane, and then the concrete equation of the fork arm plane is derived. Finally, by combining the equation of the fork arm plane with the spatial distribution of the key feature points of the fork arm, the spatial position and orientation of the fork arm plane in the world coordinate system are determined, and the real-time fork arm plane is fully constructed, providing a spatial geometric basis for subsequent operations such as the occlusion judgment of the fork arm and the projected arrow.

[0030] Determine the plane normal vector of the fork arm based on the real-time plane equation of the fork arm; Specifically, after obtaining the real-time forklift plane equation, which is calculated by combining motion parameters and joint parameters from the forklift kinematic model, it is presented as a mathematical expression describing the spatial relationship in planar space. Due to the inherent properties of the plane equation, the vector formed by the coefficients of the corresponding unknowns in the equation is itself perpendicular to the plane. By directly extracting the coefficients that reflect the perpendicularity from the plane equation, the vector formed by these coefficients is the forklift plane normal vector, which can be used for subsequent analysis of the forklift plane's spatial orientation and other scenarios.

[0031] By traversing each projected pixel of the arrow pattern based on the direction vector and position coordinates, the direction of the light ray corresponding to the projected pixel is determined. Specifically, when determining the direction of the projected pixel light rays, the direction vector and position coordinates of the arrow pattern are first defined. The direction vector represents the orientation of the arrow in space, and the position coordinates determine the reference point of the arrow in space. Next, for each projected pixel of the arrow pattern, its coordinate information in the image coordinate system is obtained. Then, combining the camera's intrinsic parameters, such as focal length and principal point coordinates, with extrinsic parameters, such as the camera's position and orientation in the world coordinate system, a mapping relationship from image pixel coordinates to spatial light rays is constructed. Starting from the position coordinates, the direction of the spatial light ray corresponding to the pixel in the world coordinate system is calculated based on the projected pixel coordinates and camera parameters. This spatial light ray direction is then associated with the arrow's direction vector. Through vector operations, the light ray direction corresponding to the projected pixel is determined, ensuring that the light ray direction conforms to the geometric laws of camera imaging and matches the orientation characteristics of the arrow itself. This provides a basis for subsequent operations such as determining the occlusion relationship between the projected pixel and the fork plane.

[0032] Calculate the angle between the plane normal of the fork arm and the direction of the ray; Calculate the distance between the projected pixels and the plane of the fork arm; When the included angle is acute and the distance is less than a preset threshold, the projected pixel is determined as an occluded pixel.

[0033] Specifically, when calculating the angle between the normal vector of the fork-arm plane and the ray direction, the distance between the projected pixel and the fork-arm plane, and determining the occluded pixel, the following steps are taken: First, the calculated normal vector of the fork-arm plane and the ray direction corresponding to the projected pixel are obtained. The normal vector of the fork-arm plane is determined by the real-time equation of the fork-arm plane, and the ray direction is calculated based on the direction vector of the arrow pattern, the position coordinates, and the camera parameters. Next, the dot product formula is used to multiply the normal vector of the fork-arm plane and the ray direction vector. Then, the angle between the two vectors is calculated using inverse trigonometric functions, combined with the magnitudes of the two vectors. Finally, an empty... The method for calculating the distance from a point to a plane is based on the coordinates of the spatial point corresponding to the projected pixel and the equation of the fork plane. The distance between the projected pixel and the fork plane is calculated. Finally, the calculated included angle is compared with the acute angle condition, and the calculated distance is compared with a preset threshold. When the included angle is acute and the distance is less than the preset threshold, it is determined that the projected pixel intersects or is close to the fork plane, and it is identified as an occluded pixel. Otherwise, it is identified as a non-occluded pixel. This completes the judgment of whether all projected pixels are occluded pixels, providing data support for subsequent projection effect optimization.

[0034] S4. Determine the number of visible pixels of the visible pixels based on the occluded pixels, and calculate the occlusion rate of the projected pixels based on the number of visible pixels and the total area of ​​the projection template. In this embodiment of the invention, the number of visible pixels of visible pixels is determined based on the occluded pixels, and the occlusion rate of the projected pixels is calculated based on the number of visible pixels and the total area of ​​the projection template, including: Iterate through all pixels of the projection template, count the number of pixels that are not marked as occluded pixels, and record them as the number of visible pixels; Substituting the number of visible pixels and the total area of ​​the projected template into the formula for calculating the occlusion rate, we obtain the occlusion rate. The formula for calculating the occlusion rate is as follows: In the formula, It's the occlusion rate. It is the number of visible pixels. It is the total area of ​​the projection template.

[0035] In detail, the projection template refers to a predefined set of pixels used to describe the projection characteristics of the arrow pattern. It contains the initial layout information of all pixels of the arrow pattern projected onto the imaging plane, covering the pixel distribution corresponding to the shape and size that the arrow pattern projection should present. It is the basic reference object for subsequent statistical analysis of pixel occlusion and calculation of occlusion rate. By traversing and checking all its pixels, the number of visible pixels and the occlusion situation are determined, thereby quantifying the degree of occlusion of the projection by forks, etc.

[0036] Specifically, when calculating the occlusion rate of projected pixels, firstly, all pixels of the projection template are traversed, and each pixel is checked to see if it is marked as an occluded pixel. Pixels not marked as occluded pixels are counted, and the count result is the number of visible pixels. Next, the total area of ​​the projection template is determined, which is the total area size corresponding to the number of pixels contained in the projection template. Then, the counted number of visible pixels and the known total area of ​​the projection template are substituted into the occlusion rate calculation formula. According to the ratio of the number of visible pixels to the total area of ​​the projection template in the formula, the ratio is first calculated, and then subtracted from 1. The result is the occlusion rate. This process can accurately quantify the proportion of pixels in the projection template that are occluded, providing data support for subsequent decisions or analyses based on the occlusion rate.

[0037] S5. Use a preset scoring function to filter out the optimal visible plane of the plane set; In this embodiment of the invention, the optimal visible plane of the plane set is selected using a preset scoring function, including: Based on TOF point cloud data and LiDAR point cloud data, the set of planes in which visible pixels are not obscured by the forklift is fitted using the RANSAC plane fitting algorithm. Specifically, TOF point cloud data is a set of three-dimensional coordinate information of object surfaces in a scene, acquired based on time-of-flight technology. The distance to the target object is determined by measuring the time of flight of light pulses, thus obtaining the spatial position of each point on the object's surface. LiDAR point cloud data, also known as laser radar point cloud data, is generated by a laser radar emitting laser signals and receiving reflected signals. Through data acquisition, navigation, and point cloud calculation, a spatial point dataset containing information such as three-dimensional coordinates is calculated for environmental perception. The RANSAC plane fitting algorithm, or Random Sample Consensus Algorithm, fits a model by randomly selecting data points multiple times, judges interior points based on a preset threshold, and selects the model with the most interior points as the result to achieve robust fitting of the data. The visible set of planes where pixels are not occluded by the forklift refers to a set of spatial plane features describing the area not occluded by the forklift, determined through prior pixel occlusion analysis, and then fitted using TOF and LiDAR point cloud data with the RANSAC plane fitting algorithm. This set is used for subsequent scene analysis and projection effect verification.

[0038] Specifically, when fitting a planar set of visible pixels not obscured by the forklift, TOF point cloud data and LiDAR point cloud data are first acquired simultaneously. TOF point cloud data uses the time-of-flight principle to obtain the 3D coordinate information of object surfaces in the scene, while LiDAR point cloud data calculates the 3D point cloud of the environment by transmitting and receiving laser pulses using a LiDAR scanner. Next, the two types of point cloud data are registered and fused. Based on the spatial coordinates, timestamps, or feature point matching of the point cloud data, the TOF and LiDAR point clouds are unified into the same spatial coordinate system, forming a complete and accurate fused point cloud dataset. Then, the point cloud corresponding to the visible pixels not obscured by the forklift is selected from the fused point cloud data. Using the previously determined information on obscured pixels, the data is then used to inversely... The process begins by extracting the point cloud portion that is not marked as occluded. Then, the RANSAC plane fitting algorithm is applied, randomly selecting the minimum set of sampled points from the selected point cloud to fit an initial plane model. The distances from other points in the point cloud to the plane under this model are calculated, and the number of interior points with distances less than a set threshold is counted. This process of randomly selecting sampled points, fitting planes, and counting interior points is repeated multiple times, and the plane model with the most interior points is selected as the optimal fitting plane. Finally, a full traversal and multiple RANSAC fitting operations are performed on the point cloud in the unoccluded areas to obtain multiple fitting planes that meet the criteria. These planes together form a set of planes where visible pixels are not occluded by the forklift, providing a planar reference for subsequent analysis of scene spatial structure and verification of projection effects.

[0039] Calculate the occlusion ratio of each plane in the plane set; A preset scoring function is generated based on the occlusion ratio and the direction vector of the arrow pattern. The preset scoring function is as follows: In the formula, It is a two-dimensional rating. It is the normal vector of the plane. It is the direction vector of the arrow pattern. It's the occlusion ratio; The plane set is scored using a preset scoring function to obtain the maximum score; The plane corresponding to the highest score is selected as the optimal visible plane of the plane set.

[0040] Specifically, when selecting the optimal visible plane for a plane set, firstly, for each plane in the set, the proportion of occluded pixels to the total number of pixels on that plane is calculated. By comparing the total number of pixels on the plane with the number of unoccluded pixels (or directly counting the number of occluded pixels), the occlusion ratio of each plane is calculated. Next, the normal vector of each plane and the direction vector of the arrow pattern are obtained. The plane normal vector and the arrow direction vector are multiplied by a dot product. Simultaneously, combined with the occlusion ratio of the plane, the sum of the dot product results is added according to the calculation logic of the preset scoring function to obtain the score of each plane. Then, the scores of all planes in the set are iterated and compared, and the score with the largest value is found and recorded as the maximum score. Finally, the plane corresponding to the maximum score is selected. This plane is the optimal visible plane in the set that best meets the visibility requirements of the arrow projection, providing an ideal projection plane basis for subsequent arrow projection positioning and other operations.

[0041] S6. Reproject the arrow template using the optimal visible plane to obtain the bispectral arrow pattern.

[0042] Specifically, when performing the reprojection process on the arrow template using the optimal visible plane to obtain a bispectral arrow pattern, the spatial parameters of the determined optimal visible plane are first accurately obtained, including the plane's normal vector and its position coordinates in the world coordinate system. These parameters are calculated based on the occlusion ratio and arrow direction vector in the previous plane selection process and serve as the spatial reference for subsequent projection. Simultaneously, the original data of the arrow template is extracted, covering the pixel distribution, shape features, and color information of the arrow pattern, clarifying the various attributes of the arrow template in the initial coordinate system. Next, a projection transformation relationship is constructed from the original coordinate system of the arrow template to the corresponding spatial coordinate system of the optimal visible plane. Based on the spatial parameters of the optimal visible plane, combined with the projection parameters of the camera imaging model or projection device, the target projection coordinates of each pixel of the arrow template on the optimal visible plane are determined through coordinate transformation matrix operations. Then, pixel-by-pixel reprojection calculations are performed on the arrow template. According to the established projection transformation relationship, each pixel of the arrow template is mapped from its original position to its corresponding position on the optimal visible plane. During this process, the effects of factors such as the spatial orientation and distance of the plane on the scaling, rotation, and translation of the projected pixels are considered to ensure that the position and shape of the projected pixels are adapted to the optimal visible plane. Subsequently, image optimization processing is performed after projection. To address potential pixel distortion and edge blurring issues during reprojection, image enhancement algorithms are employed, such as anti-aliasing of projected edge pixels and adjustments to overall brightness and contrast, resulting in a clear and natural projected arrow pattern. Simultaneously, bispectral analysis is introduced. Based on the mathematical principles of bispectrum, spectral analysis and reconstruction are performed on the reprojected arrow pattern, extracting its bispectral features and integrating them into the image data. This enhances the information representation of the pattern in the frequency domain, giving the final generated image bispectral characteristics. Finally, all processed pixel information is integrated to generate a complete bispectral arrow map. This arrow map retains the original features of the arrow template, adapts to the spatial layout of the optimal visible plane, and incorporates bispectral information to meet specific technical or application requirements, providing a precise image data foundation for subsequent recognition, analysis, and other operations based on this bispectral arrow map.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method of visual-sensor-based projection marking, characterized in that The method comprises: S1, acquiring a visible light image and a near-infrared image of a forklift working environment, extracting an arrow contour of a projection device from the visible light image, and calculating a phase field of a frequency domain complex matrix in the near-infrared image; S2, solving the pose of the arrow pattern based on the arrow contour and the phase field to obtain a direction vector and a position coordinate of the arrow pattern; S3, performing occlusion judgment on the projection pixels of the arrow pattern according to the direction vector and the position coordinate to obtain occluded pixel points; S4, determining the number of visible pixels according to the occluded pixel points, and calculating the occlusion rate of the projection pixels based on the number of visible pixels and the total area of the projection template; S5, screening the optimal visible plane of the plane set by using a preset scoring function; S6, performing re-projection processing on the arrow template by using the optimal visible plane to obtain a dual-spectrum arrow diagram.

2. A method of projecting a mark based on visual sensing as claimed in claim 1, wherein, The method comprises: S1, acquiring a visible light image and a near-infrared image of a forklift working environment, extracting an arrow contour of a projection device from the visible light image, and calculating a phase field of a frequency domain complex matrix in the near-infrared image; S2, solving the pose of the arrow pattern based on the arrow contour and the phase field to obtain a direction vector and a position coordinate of the arrow pattern; S3, performing occlusion judgment on the projection pixels of the arrow pattern according to the direction vector and the position coordinate to obtain occluded pixel points; 3. The method of claim 1, wherein the method further comprises: S4, determining the number of visible pixels according to the occluded pixel points, and calculating the occlusion rate of the projection pixels based on the number of visible pixels and the total area of the projection template; S5, screening the optimal visible plane of the plane set by using a preset scoring function; S6, performing re-projection processing on the arrow template by using the optimal visible plane to obtain a dual-spectrum arrow diagram. The method comprises:

4. The method of claim 1, wherein the method further comprises: S1, acquiring a visible light image and a near-infrared image of a forklift working environment, extracting an arrow contour of a projection device from the visible light image, and calculating a phase field of a frequency domain complex matrix in the near-infrared image; S2, solving the pose of the arrow pattern based on the arrow contour and the phase field to obtain a direction vector and a position coordinate of the arrow pattern; S3, performing occlusion judgment on the projection pixels of the arrow pattern according to the direction vector and the position coordinate to obtain occluded pixel points; S4, determining the number of visible pixels according to the occluded pixel points, and calculating the occlusion rate of the projection pixels based on the number of visible pixels and the total area of the projection template; S5, screening the optimal visible plane of the plane set by using a preset scoring function; S6, performing re-projection processing on the arrow template by using the optimal visible plane to obtain a dual-spectrum arrow diagram. The method comprises:

5. The method of claim 1, wherein the method further comprises: S1, acquiring a visible light image and a near-infrared image of a forklift working environment, extracting an arrow contour of a projection device from the visible light image, and calculating a phase field of a frequency domain complex matrix in the near-infrared image; S2, solving the pose of the arrow pattern based on the arrow contour and the phase field to obtain a direction vector and a position coordinate of the arrow pattern; S3, performing occlusion judgment on the projection pixels of the arrow pattern according to the direction vector and the position coordinate to obtain occluded pixel points; S4, determining the number of visible pixels according to the occluded pixel points, and calculating the occlusion rate of the projection pixels based on the number of visible pixels and the total area of the projection template; S5, screening the optimal visible plane of the plane set by using a preset scoring function; S6, performing re-projection processing on the arrow template by using the optimal visible plane to obtain a dual-spectrum arrow diagram. When the included angle is an acute angle and the distance is less than a preset threshold, the projection pixel is determined as an occluded pixel point.

6. The method of claim 1, wherein the method further comprises: A visible pixel number of the visible pixel is determined according to the occluded pixel point, and an occlusion rate of the projection pixel is calculated based on the visible pixel number and a total area of the projection template, including: All pixels of the projection template are traversed, and a number of pixels that are not marked as the occluded pixel point is counted, and is recorded as the visible pixel number; The visible pixel number and the total area of the projection template are substituted into a calculation formula of the occlusion rate to obtain the occlusion rate.

7. The method of claim 1, wherein the method further comprises: determining a location of the object based on the image data; and displaying a visual indicator on the object based on the location of the object. 7 An optimal visible plane of the plane set is screened out by using a preset scoring function, including: Based on the TOF point cloud data and the LiDAR point cloud data, a plane set in which the visible pixel is not occluded by the forklift is fitted by combining a RANSAC plane fitting algorithm; An occlusion proportion of each plane in the plane set is calculated; A preset scoring function is generated according to the occlusion proportion and a direction vector of the arrow pattern; The plane set is scored by using the preset scoring function to obtain a maximum score; The plane corresponding to the maximum score is selected as the optimal visible plane of the plane set.

Images